FedAttr: Towards Privacy-preserving Client-Level Attribution in Federated LLM Fine-tuning
arXiv:2605.06596v1 [cs.CR] 7 May 2026
Su Zhang Department of Computer Science University of Maryland, College Park [email protected]
Junfeng Guo Department of Computer Science University of Maryland, College Park [email protected]
Heng Huang Department of Computer Science University of Maryland, College Park [email protected]
Abstract Watermark radioactivity testing type of methods can detect whether a model was trained on watermarked documents, and have become key tools for protecting data ownership in the fine-tuning of large language models (LLMs). Existing works have proved their effectiveness in centralized LLM fine-tuning. However, this type of method faces several challenges and remains underexplored in federated learning (FL), a widely-applied paradigm for fine-tuning LLMs collaboratively on private data across different users. FL mainly ensures privacy through secure aggregation (SA), which allows the server to aggregate updates while keeping clients’ updates private. This mechanism preserves privacy but makes it difficult to identify which client trained on watermarked documents. In this work, we propose FedAttr, a new client-level attribution protocol for FL. FedAttr identifies which clients trained on watermarked data via a paired-subset-difference mechanism, while preserving the privacy guarantees of SA and FL performance. FedAttr proceeds in three steps: (i) estimate each client’s update by differencing two SA queries, (ii) score the estimate with the watermark detector via differential scoring, and (iii) combine scores across rounds via Stouffer method. We theoretically show that FedAttr produces an unbiased estimator of each client’s update with bounded mutual information leakage (i.e., O(d∗ /N ) per-round update). Moreover, FedAttr empirically achieves 100% TPR and 0% FPR, outperforming all baselines by at least 44.4% in TPR or 19.1% in FPR, with only 6.3% overhead relative to FL training time. Ablation studies confirm that FedAttr is robust to protocol parameters and configurations.
1
Introduction
Large language models (LLMs) are increasingly fine-tuned on documents obtained from external sources, often under license terms that restrict data use to the licensee’s own training. Watermark radioactivity testing type of methods have emerged as a practical tool for detecting violations of such terms: the data provider embeds a watermark into the documents, and a watermark detection test can later determine whether a model was trained on watermarked documents [Sander et al., 2024, Cui et al., 2025]. This approach has been validated for centralized LLM fine-tuning. In federated learning (FL), where multiple institutions jointly fine-tune a model on their private data without sharing it [Ye et al., 2024, Fan et al., 2023], however, the watermark radioactivity test faces two challenges and remains underexplored. A global model radioactivity test can still detect that Preprint.
Data Distribution
W
Corpus Owner
FL Training
Global Radioactivity Test W
Data Use License
Server Corpus Owner
Secure Aggregation (SA)
d
Watermark Detection
FedAttr W
FedAttr
Watermarked Global Model
Secure Aggregation (SA)
Corpus Owner Secure Aggregation (SA)
Client-level Attribution
Clients
(a)
Clients
Watermarked Client
(b)
(c)
Benign Client
Watermarked Client
Benign Client
(d)
Figure 1: Overview of Data Attribution in Federated Learning. (a) The corpus owner distributes watermarked documents to clients under a data-use license. (b) Clients collaboratively fine-tune a shared model via federated learning through secure aggregation (SA). (c) A global radioactivity test can detect the watermark signal of the trained model, but cannot identify which clients are responsible without violating SA. (d) FedAttr identifies which clients use the watermark documents through SA. the trained model was influenced by watermarked documents. However, it cannot identify which client used them. This distinction matters because license terms are often granted to individual institutions: the data provider needs to know exactly which institutions violated the terms. Identifying the responsible clients is challenging for two reasons. First, FL systems widely adopt the secure aggregation (SA) mechanism to protect client privacy, which hides each client’s individual update from the server. This mechanism preserves privacy but makes it difficult to identify which client trained on watermarked documents. Second, even if individual updates were available, the global model already carries watermark signals from previous rounds, so even a benign client’s update appears watermarked when tested, giving over 57% FPR in our experiments. Existing FL forensic methods [Zhang et al., 2022, Jia et al., 2024] do not resolve these challenges: they are designed to detect adversarial poisoning attacks, in which malicious clients send manipulated updates. In our setting, all clients faithfully follow the protocol, producing none of the adversarial signals these methods rely on. Moreover, both methods require plaintext access to updates, violating SA. In this work, we propose FedAttr, a novel client-level attribution protocol for federated LLM finetuning. FedAttr preserves the standard FL training and SA protocol. It identifies which clients were trained on watermarked data via the paired-subset-difference mechanism, while preserving the privacy guarantees of SA and FL performance. FedAttr proceeds in three steps. First, to overcome SA’s restriction on observing individual updates, FedAttr estimates each client’s update by differencing two authorized SA subset queries, one that includes the target client and one that excludes it, yielding an unbiased estimator with bounded variance (proved by Theorems 1–2). Second, since the global model accumulates watermark bias across rounds, FedAttr reduces this bias by scoring each estimate relative to the current global model. Third, since the per-round watermark signal is weak, FedAttr combines per-round differential scores across T communication rounds via Stouffer’s method to identify √ which client is watermarked. The score gaps between watermarked and benign clients grow with T , driving error rates to zero exponentially (proved by Theorem 3). We summarize our contributions as follows. (1) Problem and protocol. We formalize client-level attribution problem and propose FedAttr, which combines a client-level update estimator, differential scoring, and cross-round Stouffer combination to decide which client uses the watermarked documents through the SA mechanism. (2) Theoretical guarantees. We prove the client-level update estimator is unbiased with bounded variance (Theorems 1, 2), and derive two-sided exponential error bounds for cross-round Stouffer combination that drive false negatives to zero in T (Theorem 3). (3) Privacy analysis. We bound the per-round mutual information leakage of the estimator about each client’s update by O(d∗ /N ), where d∗ is the effective subspace dimension and N is the subset size of SA queries (Theorem 4). (4) Empirical validation. In federated LoRA fine-tuning experiment, FedAttr achieves 100% TPR at 0% FPR within 5 rounds across two watermark families and two aggregation strategies, outperforming all baselines by at least 44.4% in TPR or 19.1% in FPR, with only 6.3% overhead relative to FL training time. Ablation studies confirm that FedAttr is robust to parameters and configurations. 2
2
Related Work
Federated LLM fine-tuning. Federated learning [McMahan et al., 2017] has been extended to LLM fine-tuning via parameter-efficient adapters such as LoRA [Hu et al., 2022], with aggregation strategies including FedIT [Zhang et al., 2023] and FLoRA [Wang et al., 2024]. We evaluate both in our experiments. Detailed descriptions of two aggregation strategies are in Appendix F. Training-data watermarking. Embedding detectable signals into training data so that downstream models inherit measurable traces, known as radioactivity [Sablayrolles et al., 2020], was extended to LLMs in two forms: Sander et al. [2024] generate documents with a watermarked LLM whose green-token bias transfers to models fine-tuned on them, and Cui et al. [2025] inject fabricated entity-attribute pairs that the fine-tuned model memorizes and can be detected via QA probes. These methods assume a single training party; in FL, the global model aggregates all clients’ updates, so a detection on the global model no longer identifies which client used the watermarked data. FedAttr reuses these detectors as black-box scoring functions and resolves this attribution problem. Federated forensics. FLDetector [Zhang et al., 2022] and FLForensics [Jia et al., 2024] trace malicious clients in poisoning attacks by detecting update inconsistency and misclassification influence, respectively. Both require plaintext access to the individual updates and rely on adversarial signals absent in our non-adversarial setting, where clients faithfully follow the FL protocol. Secure aggregation and privacy. SA protocols [Bonawitz et al., 2017] enable the server to compute subset sums of client updates without observing the individual updates. Elkordy et al. [2023] provide the first mutual information bound on per-round leakage under standard SA protocol. FedAttr’s privacy analysis extends this framework to the multi-query setting required by client-level attribution.
3
Problem setup
3.1
Problem Formulation
Federated fine-tuning system. FL enables K clients {1, . . . , K} to collaboratively train a shared global LLM over T communication rounds under the coordination of a central server. At each round t, the server distributes the current global model parameters wt−1 ∈ Rd to all clients; client i locally fine-tune wt−1 on its private dataset Di and returns the resulting parameter update ∆ti ∈ Rd . Then the server aggregates updates into the new global model according to a federated aggregation rule [McMahan et al., 2017]: K X wt = wt−1 + pi ∆ti , (1) i=1
PK
where aggregation weights pi ≥ 0 satisfy i=1 pi = 1, typically pi = |Di |/ all K clients participate in every communication round.
P
j |Dj |. We assume
Secure aggregation. Secure aggregation (SA) aims to preserve client updates’ privacy in FL systems [Bonawitz et al., 2017]. SA allows the server to compute the sum of client updates S t (W ) over any subset of clients W ⊆ [K] with |W | ≥ Nsa , while keeping each individual update hidden: X t S (W ) = ∆tj . (2) j∈W
The threshold Nsa prevents individual updates from being exposed.1 Client-level attribution problem. An unknown set of clients trained on watermarked documents in violation of license terms, i.e., use the watermarked documents. Given access to SA aggregations {S t (W )} over admissible subsets W and rounds t ∈ {1, . . . , T }, the client-level attribution problem is to output a per-client binary decision ri ∈ {0, 1} for each client i ∈ [K], where ri = 1 iff client i’s dataset Di contains watermarked documents. 3.2
Watermark Families
FedAttr requires only a scoring function S CORE(w; P) that returns a larger value when w has been trained on watermarked data, where w is the model under test and P is a set of evaluation prompts. 1 SA can be implemented via multi-party computation [Bonawitz et al., 2017] or homomorphic encryption [Zhang et al.,
2020]; FedAttr is agnostic to the choice of instantiation.
3
KGW watermark [Kirchenbauer et al., 2023]. Before distributing the documents, the data provider rephrases them with a watermarked LLM that partitions the vocabulary into green and red lists via a pseudorandom function and boosts green-token logits by δ during decoding. A model fine-tuned on these documents inherits the green-token bias [Sander et al., 2024]. To detect this bias, a z-test compares the observed green-token ratio against the expected null rate γ = |G|/|V|. Fictitious knowledge watermark [Cui et al., 2025]. The data provider injects fabricated entityattribute tuples (e.g., “Arlo Vance was born in 1987”) into the documents. A model fine-tuned on these documents memorizes the fictitious attributes. To detect memorization, each attribute is queried via QA, and per-attribute results are aggregated via Fisher’s method. 3.3
Threat Model and Considered Scenarios
FedAttr targets a non-adversarial license-violation setting in which all parties are honest-but-curious: they execute the protocol faithfully but may attempt to infer private information from observation. An unknown subset of clients trains on watermarked documents in violation of license terms. We consider a FL system with three parties: clients {1, . . . , K} with private datasets, a server that coordinates training via the SA interface, and a corpus owner that holds the watermark detection key. During training, the server coordinates FL training with clients and observes only authorized subset sums S t (W ) through SA. The corpus owner is not involved. After training, the server sends FedAttr estimates to the corpus owner, who applies the detection key and identifies which clients use the watermarked data. Neither party sees the other’s private inputs: the server never learns the detection key, and the corpus owner never observes individual updates.
4
FedAttr Protocol
FedAttr preserves the standard FL training and SA protocol, and enables the corpus owner to identify which clients use the watermarked documents. Specifically, FedAttr estimates the update via a paired-subset-difference mechanism motivated and supported by Theorems 1- 2. FedAttr contains three steps: (i) the server estimates each client’s update from paired SA queries and sends the estimate to the corpus owner, (ii) the corpus owner scores each estimate by the watermark detector, and (iii) the corpus owner combines per-round scores across rounds to identify the client via Stouffer method. For each stage, we perform theoretical analyses to illustrate that FedAttr can preserve the utility of client update estimates, demonstrating the effectiveness of our protocol. Algorithm 1 summarizes it. 4.1
Client-level Update Estimator
Our goal here is to construct an unbiased estimator of any single client’s update through the SA interface. The challenge is that SA hides clients’ individual updates. The first step is based on a key observation: Two subset SA queries differing only in whether they include a target client i must differ only by client i’s update in expectation. We call it the paired-subset-difference mechanism. As such, we can construct an unbiased estimator of any single client’s update via paired subset SA queries. Constructing the unbiased update estimator via paired subset SA queries: Given a target client i ∈ [K] and N ∈ [K − 1], the number of non-target clients per query.2 We define two sampling families over the non-target clients [K] \ {i}, distinguished by whether they include the target client i: UiN = U ⊆ [K] : i ∈ U, |U | = N +1 , ViN = V ⊆ [K] : i ∈ / V, |V | = N . (3) t The server draws M include-target subsets U1t , . . . , UM i.i.d. uniformly from UiN and M excludet t N target subsets V1 , . . . , VM i.i.d. uniformly from Vi , then forms the round-t update estimator M M X 1 X t b it := 1 ∆ St (Um ) − St (Vmt ). M m=1 M m=1
(4)
For convenience, we denote the paired queries at round t to target client i by Qti := t t (U1t , . . . , UM ; V1t , . . . , VM ), and all queries in round t by Qt = {Qti }K i=1 . For each non-target 2 Both subset sizes N and N + 1 must satisfy the SA protocol’s authorization threshold N sa [Bonawitz et al., 2017]; we
assume this throughout. In practice Nsa is small relative to K.
4
PM PM 1 1 t t client j ̸= i, the masking coefficient is αjt := M m=1 1{j ∈ Um } − M m=1 1{j ∈ Vm }. Note that the queries process is independent with the global model and client updates. Rejecting estimator when privacy condition fails: To ensure that the non-target updates provide sufficient masking noise for the privacy analysis (Section 5), for instance, to exclude the degenerate b t = ∆t , the case where the include and exclude subsets draw identical non-target clients, leaving ∆ i i server checks the following privacy condition before querying the SA interface: cti :=
X (αjt )2 ≥ aN,
(cti )2 t 4 ≥ aN, j̸=i (αj )
t Meff,i := P
j̸=i
and
N < K − 1,
(5)
where a := (1 − ρ)/M and ρ := N/(K − 1) < 1. If the condition fails, the server resamples the subsets. This rejection policy depends only on the subset choice n and never on client updates and the global o t t model. We define the acceptance event at round t: Ai := cti ≥ aN, Meff,i ≥ aN, N < (K − 1) . By the symmetry of the accepted sampling distribution, each non-target client’s updates cancel in expectation. We prove that FedAttr constructs an unbiased estimator (Theorem 1) with variance controlled by the non-target updates (Theorem 2). Theorem 1 (Unbiasedness under rejection sampling). Given any round t and target client i ∈ [K]. Under the subset sampling with rejection described above, for any deterministic updates ∆t1 , . . . , ∆tK , t t b i ∆1t , . . . , ∆K E ∆ , Ati = ∆it . Theorem 2 (Conditional variance under rejection sampling). Under the same setting as Theorem 1, the conditional covariance satisfies 1 2 N (K − 1 − N ) t t b it ∆1t , . . . , ∆K · · Σ−i , Cov ∆ , Ati ⪯ t · pa,i M K −2 P 1 t t ¯t ¯t ⊤ where pta,i := PrQt (Ati ) and Σt−i := K−1 j̸=i (∆j − ∆−i )(∆j − ∆−i ) .
(6)
Complete proofs are in Appendix B.1-B.2. The rejection check introduces negligible overhead: the threshold aN equals half the expected masking strength and the rejection probability decays as e−Ω(N ) nearly identical to unrestricted sampling, so the expected number of redraws 1/pta,i → 1 exponentially fast. Theorem 2 shows that increasing the query count M reduces estimator noise at the cost of additional SA queries. Detailed analysis of the acceptance rate is in the Appendix E. 4.2
Differential Scoring
In this stage, we aim to score the client’s estimate with the watermark detector. The challenge is that the global model wt−1 already carries watermark signals absorbed from previous rounds, causing the detector to assign high scores to all clients, including benign ones. Applying the detector directly to the updates causes 57% FPR (demonstrated in Table 1) even with access to plaintext updates. FedAttr addresses this via differential scoring: it evaluates the detector at both the global model wt−1 b t , and calculate the difference. For each round t and target client i: and the estimate model wt−1 + ∆ i (t)
zi
b ti ; Pt ) − S CORE(wt−1 ; Pt ), := S CORE(wt−1 + ∆
(7)
where Pt is the evaluation prompt set containing watermark pattern at round t. Motivated by analysis in Appendix D, differential scoring can effectively reduce the watermark bias in the global model, and (t) zi measures only the contribution of client i’s estimated update. As demonstrated by Theorems 1–2, b t is unbiased for ∆t , so other clients contribute only to sampling variance. ∆ i i 4.3
Cross-round Stouffer Combination
Our goal here is to combine per-round scores across rounds to identify the watermarked client. A single round yields a weak signal because the watermark signal is not completely learned after one round of local fine-tuning; the full watermark signal emerges gradually as training T increases. 5
Moreover, the estimator would introduce sampling noise in the first stage (Section 4.1). FedAttr accumulates this growing signal via cross-round Stouffer’s combination [Stouffer, 1949]: T
1 X (t) Zi = √ zi . T t=1
(8)
The corpus owner flags client i as watermarked if Zi > γ for a fixed threshold γ > 0. To state the formal guarantee, we introduce a separation condition on the per-round scores. Let Ft−1 = σ(wt−1 , Q1 , . . . , Qt−1 ) denote the global model and all queries history up to round t−1. Assumption 1 (Watermark signal separation condition). There exist constants m > 0, ϵ ∈ [0, m), (t) and ν > 0 such that for each client i ∈ [K] and round t ∈ [T ]: (i) the conditional mean µi := (t) (t) (t) E[zi | Ft−1 ] satisfies µi ≥ m if client i is watermarked and |µi | ≤ ϵ if client i is benign; (ii) the (t) (t) centered increment zi − µi is conditionally ν 2 -sub-Gaussian given Ft−1 . We verify this assumption empirically in Figure 2(b,c). Then we introduce the following theorem: √ √ Theorem 3 (Stouffer error). Under Assumption 1, for any threshold satisfying T ϵ < γ < T m, √ 2 2 exp(−(γ ) if i is benign, √− T ϵ) /2ν (9) Pr error for client i ≤ 2 exp(−( T m − γ) /2ν 2 ) if i is watermarked. The √ proof is in Appendix B.3. A fixed γ controls both errors: the false-positive bound depends on √ γ − T ϵ, while the false-negative rate decays exponentially once T m > γ.
5
Privacy Analysis
We analyze the information leakage of FedAttr’s estimation to the corpus owner with respect to clients’ updates. FedAttr operates entirely through SA, preserving the SA’s privacy guarantee for each model update. However, a residual information-leakage threat remains [Elkordy et al., 2023]: the b t of client i’s actual update ∆t . We quantify this residual leakage server obtains a noisy estimation ∆ i i using mutual information (MI), following the framework of Elkordy et al. [2023]. Our analysis extends theirs from the standard SA setting to the subset-query setting where FedAttr performs. Leakage metric. Given a target client i within round t, the per-round leakage is (t) b it Qti , Ft−1 . Ipriv := I ∆ti ; ∆
(10)
b t reveals about client i’s actual update, This quantity measures how much information the estimator ∆ i beyond what is already known from Ft−1 = σ(wt−1 , Q1 , . . . , Qt−1 ) and the round t query Qti . Inspired by Elkordy et al. [2023], we propose two assumptions on the properties of the model to shed light on the leakage of MI during the update process. Assumption 2 (Independent under whitening). Let t −1/2 t Zjt := (KG ) ξj
be the whitened update. Conditioned on Ft−1 , the coordinates of Zjt are independent, centered, t and have unit variance. For every coordinate ℓ ∈ [d∗ ], the scalar distribution Zj,ℓ has finite fourth moment and finite entropic distance to the Gaussian distribution with the same mean and variance. More explicitly, if Gℓ ∼ N (0, 1), then there exist constants M4,ℓ < ∞ and D0,ℓ < ∞, independent of j, such that t 4 E|Zj,ℓ | ≤ M4,ℓ ,
t t D(Zj,ℓ ∥Gℓ ) = h(Gℓ ) − h(Zj,ℓ ) ≤ D0,ℓ .
These are the one-dimensional regularity conditions needed to apply the Bobkov–Chistyakov–Götze entropic Berry–Esseen bound used in the independent-under-whitening case of Elkordy et al. [2023]. Assumption 3. The local datasets D1 , . . . , DK are sampled i.i.d. from a common distribution, i.e., the local dataset of client j consists of i.i.d. data samples from a distribution Pj , where Pj = P for 6
all j ∈ [K]. This implies that given round t and condition on the Ft−1 = σ(wt−1 , Q1 , . . . , Qt−1 ), each client update can decompose as ∆tj = µt + ξjt , where µt is deterministic conditioned on Ft−1 , and E[ξjt | Ft−1 ] = 0,
t Cov(ξjt | Ft−1 ) = KG .
∗ The {ξjt }K j=1 are conditionally i.i.d. on a common d -dimensional effective subspace, where t d∗ := rank(KG ).
All determinants and entropies below are taken on this effective subspace. Remark 1. Assumption 3 is the same condition as Elkordy et al. [2023, Assumption 1] and ensures the non-target updates form an i.i.d. additive mask whose entropy can be controlled via the entropic CLT. The independence-under-whitening condition (Assumption 2) is satisfied when the stochastic gradient can be approximated by a distribution with independent components or by a multivariate Gaussian [Elkordy et al., 2023, Definition 1]. Based on two assumptions, we propose our main privacy results. Theorem 4 (Release-level MI leakage). Suppose Assumptions 2 and 3 hold. If a query Qti satisfies cti ≥ aN,
t Meff,i ≥ aN,
then b ti | Qti , Ft−1 ) ≤ I(∆ti ; ∆
and
N <K −1
∗ 1 Cξ d∗ d d∗ log 1 + + =O . 2 aN aN N
The bound has two terms: the first captures leakage when the masking noise is exactly Gaussian; the second accounts for non-Gaussianity. Compared with the single-aggregate bound of Elkordy et al. [2023, Theorem 1], whose subset size N −1 is deterministic, FedAttr’s effective subset size aN arises from the random subset queries. The detailed proof is deferred to Appendix C.
6
Experiments
6.1
Experimental Setup
Federated Learning Configurations. Consistent with previous work [Ye et al., 2024, Wu et al., 2025], we fine-tune Llama-3.2-3B [Team, 2024] with LoRA on UltraChat200K [Ding et al., 2023], partitioned IID across K=10 clients for T =5 rounds. Default protocol parameters are r=3 watermarked clients, subset size N =5, query count M =5. We evaluate two aggregation strategies (FedIT [Zhang et al., 2023], FLoRA [Wang et al., 2024]) and two watermark families (KGW [Kirchenbauer et al., 2023], Fictitious Knowledge [Cui et al., 2025]). Baselines. We compare against four baselines. (i) Global model test: which applies the watermark detector to the global model but cannot attribute to clients, (ii) Direct (oracle): which applies the detector to each client’s plaintext update, violating SA, (iii) FLDetector [Zhang et al., 2022], (iv) FLForensics [Jia et al., 2024]. Notably, (ii)-(iv) require plaintext updates and violate SA. Metrics. We report TPR (fraction of watermarked clients correctly flagged) and FPR (fraction of benign clients incorrectly flagged) for each approach. FedAttr flags client i when its FedAttr Stouffer score Zi ≥ 4. We implement each baseline following its default configurations. For each approach under different settings, we report results (i.e., mean & std) calculated over three random seeds. We also report the p-value obtained by converting the Stouffer statistic Zi to a one-sided standard normal tail probability, i.e., pi = 1 − Φ(Zi ). In ablation studies, we additionally report z̄pos and z̄neg , the mean Stouffer statistics of watermarked and benign clients, to quantify signal strength beyond the binary TPR/FPR. Full hyperparameters, watermark details, and baselines are in Appendix F. 6.2
Main Results
Table 1 reports client-level attribution performance for different watermark families and FL algorithms. FedAttr achieves 100% TPR and 0% FPR in all four settings (p < 10−6 ), even completely performing 7
Table 1: Client-level watermark attribution performance (T =5 rounds, γ=4.0, mean±std over 3 seeds). FedAttr is the only method that achieves 100% TPR, 0% FPR in four settings through secure aggregation. For FLForensics, † denotes the original implementation using HDBSCAN clustering4 , and ‡ denotes our adaptation using k-means clustering, since HDBSCAN fails at small K=10. KGW FL Algorithm Baseline
SA
TPR ↑
FPR ↓
Fictitious Knowledge p-value
TPR ↑
FPR ↓
p-value
FedIT
Direct (oracle) FLDetector FLForensics† FLForensics‡ FedAttr (ours)
% 55.6±19.3 57.1±14.3 < 10−3 100.0±0.0 57.1±14.3 < 10−15 % 0.0±0.0 19.1±8.3 — 0.0±0.0 19.1±8.3 — % 0.0±0.0 0.0±0.0 — 0.0±0.0 0.0±0.0 — % 33.3±0.0 23.8±8.3 — 100.0±0.0 19.1±8.3 — ! 100.0±0.0 0.0±0.0 < 10−6 100.0±0.0 0.0±0.0 < 10−29
FLoRA
Direct (oracle) FLDetector FLForensics† FLForensics‡ FedAttr (ours)
% 100.0±0.0 71.4±14.3 < 10−6 100.0±0.0 66.7±8.3 < 10−20 % 0.0±0.0 14.3±0.0 — 0.0±0.0 23.8±8.3 — % 0.0±0.0 0.0±0.0 — 0.0±0.0 0.0±0.0 — % 55.6±19.3 23.8±8.3 — 100.0±0.0 19.1±8.3 — ! 100.0±0.0 0.0±0.0 < 10−10 100.0±0.0 0.0±0.0 < 10−50
through SA . No baseline matches this under the same privacy constraint with SA. The direct oracle has plaintext access to each client’s update but incurs FPR ≥ 57%. As the global model has already learned watermark signals from earlier rounds, the detector thus assigns high scores to all clients, including benign ones. Instead, differential scoring applied in FedAttr significantly reduces the watermark bias of the global model, accurately isolating each client’s individual effect to the watermark effectiveness. Moreover, we observe previous forensics approach Jia et al. [2024], Zhang et al. [2022] performs ineffectively in our considered scenarios as the watermark signal cannot be adapted as adversary patterns. As a result, FLDetector achieves 0% TPR, and FLForensics achieves at most 33–100% TPR with 19–24% FPR. Even more, all existing approaches require plaintext access to individual updates, violating SA, and cannot be used for watermark attribution (no p-value). 6.3
Mechanism Analysis
We further investigate the effect and soundness for each component within FedAttr. Figure 2 presents the results. Figure 2(a) compares direct and differential scoring at round t=5. Direct scoring yields 57% FPR because the global model has learned watermark signals from earlier rounds. Differential scoring subtracts the reference score, reducing the accumulated bias: only watermarked clients retain a detectable signal. Figure 2(b) shows the per-round performance of differential scores. The watermark efficacy stays above m̂=3.3 in each round, while the benign ones remain within ±ϵ̂=1.2, consistent with Assumption 1(i)(Separation). Figure 2(c) shows the empirical distribution (t) of centered residuals zi − µ̂i , closely aligns with a (sub)Gaussian distribution (ν̂=0.85), supporting Assumption 1(ii)(Sub-Gaussianity). Figure 2(d) shows the Stouffer statistic computed with varying T ′ . The watermark efficacy becomes larger than γ=4 when round T ′ ≥ 2 and reaches Zi > 8.0 at T ′ =5, yielding a margin of 4.0 above γ. Benign clients remain near zero throughout. Therefore, the Stouffer process amplifies the signal. 6.4
Ablation Studies
We study the impact of different parameters (e.g., the number of watermark clients, subset size, etc) and different configurations (e.g., LoRA rank, dataset, etc); Figures 3 (protocol parameters) and 4 (training configurations) summarize the results. Protocol parameters (Figure 3). FedAttr achieves consistently 100% TPR and 0% FPR under varying amounts of watermark clients, subsets, and watermark ratio. The watermark efficacy z̄pos exhibits the U-shaped dependence on N consistent with Theorem 2: lowest at N =5 where the variance factor N (K−1−N )/(K−2) peaks, and highest at N =1 where the estimator becomes exact (Figure 3(b)). Query count M ≥ 3 achieves 100% TPR and 0% FPR. M =2 incurs 29% FPR due to 4 https://github.com/jyqhahah/FLForensics
8
(c) Sub-Gauss. (Asm. ii)
6
0
^ = 3.3 m
2 0
0 1 2 3 4 5 6 7 8 9
Client index
²^ = 1.2
WM Benign
−2 −4
0.4
Density
(t)
5
(d) Stouffer cumulative
Empirical
4
Score zi
Score
(b) Separation (Asm. i)
Direct (WM) Diff. (WM) Direct (Benign) Diff. (Benign)
10
1
2
3
4
Round t
N(0; º^ )
0.3
º^ = 0.85
0.2 0.1 0.0
5
−2
0
WM Benign
10.0
2
Stouffer Zi
(a) Direct vs Differential
7.5 5.0 2.5 0.0
−2.5
2
1
2
3
4
5
Rounds used T 0
(t)
Residual ´i
Mean Stouffer score
Figure 2: The effect for each protocol component. (a) Differential scoring removes the accumulated bias. (b) Per-round watermarked mean stays above m̂=3.3, benign mean within ±ϵ̂=1.2, validating (t) Assumption 1(i). (c) Centered residuals zi − µ̂i match a Gaussian (ν̂=0.85), supporting Assumption 1(ii). (d) Stouffer statistic crosses γ=4 from round 2. At round 5, the statistic achieves margin Zi − γ = 4.0. (a) WM clients r 20
z¹neg
15
(c) Query count M
(b) Subset size N
z¹pos
20
20
15
15
10
10
10
5
5
5
0
0
0
1
3
WM clients r
5
1
2
4
5
6
Subset size N
8
(d) Watermark ratio 20 15
FPR=29%
10 5 0
23
5
10
Query count M
20
5
10
20
30
Watermark ratio (%)
Figure 3: Protocol parameter sensitivity. FedAttr achieves 100% TPR / 0% FPR across all tested values except M =2, which incurs 29% FPR, and shows robustness under protocol parameter selection. (a) Number of watermarked clients r. (b) Subset size N : the U-shaped curve validates Theorem 2. (c) Query count M . (d) Watermark ratio: The signal scales roughly linearly with watermark ratio.
high estimator noise (Figure 3(c)). The watermark efficacy scales roughly linearly w.r.t. watermark ratio, consistent with radioactivity theory [Sander et al., 2024]. Robustness (Figure 4). FedAttr performs robustly under different configurations of LoRA ranks, evaluated models, and datasets, consistently achieving 100% TPR and 0% FPR. Under severe non-IID heterogeneity (α=0.1), TPR degrades to 67% while FPR remains 0%; at α=0.05, FPR rises to 11%. This degradation is consistent with increased estimator variance when client updates diverge, and can be mitigated by increasing T (Table 21). The detailed analyses are included in the Appendix.
6.5
Scalability and Overhead
Scalability. We scale K from 10 to 100. Table 17 shows that watermark efficacy z̄pos decreases from 10.12 to 7.83 as K increases but remains higher above threshold γ (100% TPR, 0% FPR in all cases). FedAttr also performs effectively in a partial-participation setting (Table 20 in Appendix). Overhead. FedAttr’s overhead consists of SA queries and watermark scoring. In our main experiment, FedAttr issues 2M KT =500 SA queries, adding 5 minutes (1.0%) to the 8.5-hour FL training time; watermark scoring adds 27 minutes (5.3%), for a total overhead of 6.3%. Both costs scale linearly in K. Since all computation runs on the server, it can be overlapped with clients’ local training in the next round, effectively hiding the latency. Analysis of scalability and overhead is in Appendix H.
7
Conclusion
We introduced FedAttr, a client-level attribution protocol for federated LLM fine-tuning that identifies clients who trained on watermarked documents while preserving SA privacy. FedAttr combines unbiased update estimation from SA queries, differential scoring, and cross-round Stouffer aggregation. We provided theoretical guarantees on the estimator’s unbiasedness and variance, and bounded mutual information leakage of O(d∗ /N ) per round. Empirically, FedAttr achieves 100% TPR, 0% FPR, outperforming all baselines while being the only method that preserves SA privacy. 9
Mean Stouffer score
(a) LoRA rank 20 15
(b) Non-IID
(c) Model
20
z¹pos
TPR=67%
15
TPR=67% FPR=11%
(d) Dataset
20
20
15
15
10
10
10
10
5
5
5
5
0
0
0
z¹neg
16
64
LoRA rank
128
IID
0.5 ®=
5 0.1 0.0 ®= ®=
0
-1B
ma Lla
Data partition
-3B
ma Lla
3B
enQw
Base model
hat
raC
Ult
aca Alp
Ope
rca nO
Dataset
Figure 4: Robustness to training configurations. FedAttr achieves 100% TPR and 0% FPR across LoRA ranks, base models, and datasets. Attribution accuracy decreases moderately under severe non-IID partitions (α ≤ 0.1). (a) LoRA rank. (b) Non-IID heterogeneity. (smaller Dirichlet α means more heterogeneous) (c) Base model. (d) Training dataset.
References Sergey Bobkov, G. Chistyakov, and Friedrich Goetze. Berry-esseen bounds in the entropic central limit theorem. Probability Theory and Related Fields, 159, 05 2011. doi: 10.1007/s00440-013-0510-3. Kallista A. Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H. Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for privacy preserving machine learning. IACR Cryptol. ePrint Arch., 2017:281, 2017. URL http: //eprint.iacr.org/2017/281. W.G. Cochran. Sampling Techniques. Wiley publication in applied statistics. Wiley, 1977. ISBN 9788126515240. URL https://books.google.com/books?id=xbNn41DUrNwC. Xinyue Cui, Johnny Tian-Zheng Wei, Swabha Swayamdipta, and Robin Jia. Robust data watermarking in language models by injecting fictitious knowledge. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 - August 1, 2025, volume ACL 2025 of Findings of ACL, pages 14292–14306. Association for Computational Linguistics, 2025. URL https: //aclanthology.org/2025.findings-acl.736/. Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 3029–3051. Association for Computational Linguistics, 2023. doi: 10.18653/V1/2023. EMNLP-MAIN.183. URL https://doi.org/10.18653/v1/2023.emnlp-main.183. Ahmed Roushdy Elkordy, Jiang Zhang, Yahya H. Ezzeldin, Konstantinos Psounis, and Salman Avestimehr. How much privacy does federated learning with secure aggregation guarantee? Proc. Priv. Enhancing Technol., 2023(1):510–526, 2023. doi: 10.56553/POPETS-2023-0030. URL https://doi.org/10.56553/popets-2023-0030. Tao Fan, Yan Kang, Guoqiang Ma, Weijing Chen, Wenbin Wei, Lixin Fan, and Qiang Yang. FATE-LLM: A industrial grade federated learning framework for large language models. CoRR, abs/2310.10049, 2023. doi: 10.48550/ARXIV.2310.10049. URL https://doi.org/10.48550/ arXiv.2310.10049. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9. Yuqi Jia, Minghong Fang, Hongbin Liu, Jinghuai Zhang, and Neil Zhenqiang Gong. Tracing back the malicious clients in poisoning attacks to federated learning. arXiv preprint arXiv:2407.07221, 2024. John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A watermark for large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference 10
on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 17061–17084. PMLR, 2023. URL https: //proceedings.mlr.press/v202/kirchenbauer23a.html. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Aarti Singh and Xiaojin (Jerry) Zhu, editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS 2017, 20-22 April 2017, Fort Lauderdale, FL, USA, Proceedings of Machine Learning Research, pages 1273–1282. PMLR, 2017. URL http://proceedings.mlr.press/v54/mcmahan17a.html. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https://arxiv.org/abs/2412.15115. Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, and Hervé Jégou. Radioactive data: tracing through training. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, Proceedings of Machine Learning Research, pages 8326– 8335. PMLR, 2020. URL http://proceedings.mlr.press/v119/sablayrolles20a.html. Tom Sander, Pierre Fernandez, Alain Durmus, Matthijs Douze, and Teddy Furon. Watermarking makes language models radioactive. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024. URL http://papers.nips.cc/paper_files/paper/2024/hash/ 2567c95fd41459a98a73ba893775d22a-Abstract-Conference.html. S.A. Stouffer. The American Soldier: Adjustment During Army Life. Studies in social psychology in World War II. Princeton University Press, 1949. URL https://books.google.com/books? id=hQiBwgEACAAJ. Llama Team. The llama 3 herd of models. CoRR, abs/2407.21783, 2024. doi: 10.48550/ARXIV. 2407.21783. URL https://doi.org/10.48550/arXiv.2407.21783. Ziyao Wang, Zheyu Shen, Yexiao He, Guoheng Sun, Hongyi Wang, Lingjuan Lyu, and Ang Li. Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024. URL http://papers.nips.cc/paper_files/ paper/2024/hash/28312c9491d60ed0c77f7fff4ad86dd1-Abstract-Conference.html. Yebo Wu, Chunlin Tian, Jingguang Li, He Sun, Kahou Tam, Zhijiang Guo, Li Li, and Chengzhong Xu. A survey on federated fine-tuning of large language models. ArXiv, abs/2503.12016, 2025. URL https://api.semanticscholar.org/CorpusID:277065732. Rui Ye, Wenhao Wang, Jingyi Chai, Dihan Li, Zexi Li, Yinda Xu, Yaxin Du, Yanfeng Wang, and Siheng Chen. Openfedllm: Training large language models on decentralized private data via federated learning. In Ricardo Baeza-Yates and Francesco Bonchi, editors, Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2024, Barcelona, Spain, August 25-29, 2024, pages 6137–6147. ACM, 2024. doi: 10.1145/3637528.3671582. URL https://doi.org/10.1145/3637528.3671582. Chengliang Zhang, Suyi Li, Junzhe Xia, Wei Wang, Feng Yan, and Yang Liu. Batchcrypt: Efficient homomorphic encryption for cross-silo federated learning. In Ada Gavrilovska and Erez Zadok, editors, Proceedings of the 2020 USENIX Annual Technical Conference, USENIX ATC 2020, July 15-17, 2020, pages 493–506. USENIX Association, 2020. URL https://www.usenix.org/ conference/atc20/presentation/zhang-chengliang. 11
Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Guoyin Wang, and Yiran Chen. Towards building the federated GPT: federated instruction tuning. CoRR, abs/2305.05644, 2023. doi: 10.48550/ARXIV.2305.05644. URL https://doi.org/10.48550/arXiv.2305.05644. Zaixi Zhang, Xiaoyu Cao, Jinyuan Jia, and Neil Zhenqiang Gong. Fldetector: Defending federated learning against model poisoning attacks via detecting malicious clients. In Aidong Zhang and Huzefa Rangwala, editors, KDD ’22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, August 14 - 18, 2022, pages 2545–2555. ACM, 2022. doi: 10.1145/3534678.3539231. URL https://doi.org/10.1145/3534678.3539231.
12
A
Algorithm
Algorithm 1 presents the FedAttr protocol. At each communication round t, clients perform standard local training and submit updates through secure aggregation (Lines 1–3). For each target client i, the server repeatedly samples paired include/exclude subsets and checks the privacy condition (Eq. (5)) via rejection sampling (Lines 5–10). Upon acceptance, the server queries the SA oracle to b t (Line 11) and forwards it to the corpus owner, who computes form the unbiased update estimator ∆ i (t) the differential score zi (Lines 12–13). After all T rounds, the corpus owner aggregates per-round scores via Stouffer’s method and applies the threshold γ to produce the final attribution decision (Lines 15–21). Algorithm 1 FedAttr: Client-level Attribution through Secure Aggregation Require: Clients [K], communication rounds T , subset size N , query count M , threshold γ, SA oracle St (·), watermark detector S CORE(· ; ·), prompt sets {Pt }Tt=1 Ensure: Attribution decision for each client i ∈ [K] 1: for t = 1, . . . , T do 2: Each client i computes local update ∆ti and submits to the SA oracle PK 3: Server updates global model: wt ← wt−1 + i=1 pi ∆ti 4: for each target client i ∈ [K] do 5: repeat t i.i.d. t 6: Draw U1t , . . . , UM ∼ Unif(UiN ), Include-target, |Um | = N +1 i.i.d.
t Draw V1t , . . . , VM ∼ Unif(ViN ), Exclude-target, |Vmt | = N P P 1 1 t t 8: Compute αj ← M m 1{j ∈ Um } − M m 1{j ∈ Vmt } for all j ̸= i P P t t 4 9: Compute cti ← j̸=i (αjt )2 , Meff,i ← (cti )2 j̸=i (αj ) t 10: until cti ≥ aN and Meff,i ≥ aN and N < K−1 (Eq. (5)) PM 1 t t b t ← 1 PM St (Um )− M 11: Query SA and compute: ∆ i m=1 m=1 St (Vm ) M b t to corpus owner 12: Server sends ∆ i (t) b t ; Pt − S CORE wt−1 ; Pt 13: zi ← S CORE wt−1 + ∆ i 14: end for 15: end for 16: for each client i ∈ [K] do PT (t) 17: Zi ← √1T t=1 zi 18: if Zi > γ then 19: Flag client i as watermarked 20: else 21: Label client i as benign 22: end if 23: end for
7:
B
Proofs for Attribution Protocol
B.1
Proof of Theorem 1 (Unbiasedness under rejection sampling) P Proof. Substituting S t (W ) = j∈W ∆tj into (4) gives M X 1 b it = ∆ti + ∆ M m=1
X
∆tj −
t \{i} j∈Um
M
1 X X t ∆j . M m=1 t
t Since i ∈ Um for all m and i ∈ / Vmt for all m, this can be written as X b it = ∆ti + ∆ αjt ∆tj , j̸=i
13
j∈Vm
where αjt =
M M 1 X 1 X t 1{j ∈ Um \ {i}} − 1{j ∈ Vmt }, M m=1 M m=1
j ̸= i.
Thus it suffices to show E[αjt | Ati ] = 0
for every j ̸= i,
where Ati denotes the acceptance event t Ati = {ct ≥ aN, Meff,i ≥ aN }.
For each m, define the non-target part of the include-target query by t t Xm := Um \ {i},
and define Ymt := Vmt . t By construction, both Xm and Ymt are N -subsets of [K] \ {i}. Moreover, the proposal distribution samples t t X1t , . . . , XM , Y1t , . . . , YM
i.i.d. uniformly from the same family of N -subsets of [K] \ {i}. Let t t Qt = (X1t , . . . , XM ; Y1t , . . . , YM )
denote the non-target query design. Define the swap map T by t t T (Qt ) = (Y1t , . . . , YM ; X1t , . . . , XM ).
Equivalently, after applying T , the corresponding include-target and exclude-target subsets are reconstructed as t ′ t (Um ) = {i} ∪ Ymt , (Vmt )′ = Xm . t This map is well-defined because Xm and Ymt have the same cardinality and both lie in [K] \ {i}. It is also a bijection. Since the proposal distribution samples the X’s and Y ’s i.i.d. from the same distribution, the proposal distribution is invariant under T .
For every non-target client j ̸= i, αjt (T (Qt )) =
M M 1 X 1 X t 1{j ∈ Ymt } − 1{j ∈ Xm } = −αjt (Qt ). M m=1 M m=1
Therefore the swap map sends the coefficient vector αt = (αjt )j̸=i to −αt . The acceptance event A depends on the query design only through X ct = (αjt )2 j̸=i
and
(ct )2 t 4. j̸=i (αj )
t Meff =P
Both quantities are invariant under the sign change αt 7→ −αt . Hence 1{Ati (Qt )} = 1{Ati (T (Qt ))}. Since T preserves the proposal distribution and preserves the acceptance event, the accepted query design is also invariant under T : d
Qt | Ati = T (Qt ) | Ati . 14
Consequently, for every j ̸= i, E[αjt | Ati ] = E[αjt (T (Qt )) | Ati ] = −E[αjt | Ati ], which implies
E[αjt | Ati ] = 0.
Finally, the query design Qt is sampled independently of the client updates ∆t1 , . . . , ∆tK and the current global model, and the acceptance event Ati is a function only of the query design. Therefore the same identity holds after conditioning on the realized updates: E[αjt | ∆t1 , . . . , ∆tK , Ati ] = 0, Thus
j ̸= i.
h i X b it | ∆t1 , . . . , ∆tK , Ati = ∆ti + E ∆ E[αjt | ∆t1 , . . . , ∆tK , Ati ]∆tj j̸=i
= ∆ti . This proves the unbiasedness of the accepted estimator. B.2
Proof of Theorem 2 (Conditional variance under rejection sampling) P b t − ∆t . By Theorem 1, E[ζ | At , ∆t , . . . , ∆t ] = 0, so the Proof. Let ζ(Qt ) := j̸=i αjt ∆tj = ∆ 1 i i i K accepted covariance is b it | Ati , ∆t1 , . . . , ∆tK ) = E[ζζ ⊤ | Ati ] = Cov(∆
1 1 E[1A ζζ ⊤ ] ⪯ E[ζζ ⊤ ], pa pa
t where pta,i = Pr(Ati ). Each non-target part Um \ {i} is a simple random sample of size N drawn without replacement from the K − 1 non-target clients. By correcP the finite-population t tion formula [Cochran, 1977], each include-target sum Am := ∆ has covariance t j j∈Um \{i} Cov(Am ) = N (K − 1 − N )/(K − 2) Σt−i , and likewise for each exclude-target sum. The 2M sums are mutually independent, so
E[ζζ ⊤ ] =
2 N (K − 1 − N ) t · Σ−i . M K −2
Combining gives b it | Ati , ∆t1 , . . . , ∆tK ) ⪯ Cov(∆ B.3
1 2 N (K − 1 − N ) t · Σ−i . · pa M K −2
Proof of Theorem 3 (Stouffer concentration) (t)
(t)
(t)
(t)
Proof. Under Assumption 1, write zi = µi + ηi where ηi is conditionally ν 2 -sub-Gaussian PT (t) given Ft−1 . Define ST := By the conditional sub-Gaussian tower property, t=1 ηi . 2 2 E[exp(λST )] ≤ exp(T λ ν /2). Hence Wi := T −1/2 ST is ν 2 -sub-Gaussian. (t)
Clean client. Since |µi | ≤ ϵ, Zi ≤ √ for γ > T ϵ. (t)
Watermarked √ client. Since µi 2 2 γ) /(2ν )) for T m > γ.
C
√
T ϵ + Wi , so Pr(Zi ≥ γ) ≤ exp(−(γ −
≥ m, Zi ≥
√
√
T ϵ)2 /(2ν 2 ))
√ T m + Wi , so Pr(Zi ≤ γ) ≤ exp(−( T m −
Proofs for Privacy Analysis
This section adapts the proof strategy of Elkordy et al. [2023] to the client-level update estimates produced by FedAttr. We use the same data IID assumption and distributional assumptions as the independent-under-whitening case in Elkordy et al. [2023]. The difference of proof structure is that the equal-weight secure aggregation is replaced by a query-dependent estimation. Accordingly, the 15
t number of masking users in Elkordy et al. [2023] is replaced by the effective masking size Meff,i , where 2 X X (ct ) t Meff,i = it , cti := (αjt )2 , sti := (αjt )4 . si j̸=i
j̸=i
The proof follows the same high-level route as the independent-under-whitening case of Elkordy et al. [2023] in 4 stages: 1. Decompose the mutual information into a difference of two entropies. 2. Upper-bound the entropy of signal plus noise by Gaussian maximum entropy. 3. Lower-bound the entropy of the masking noise using the Bobkov–Chistyakov–Götze entropic Berry–Esseen bound Bobkov et al. [2011]. 4. Subtract the two bounds. C.1
FedAttr Notation
For convenience, we first list the notations in FedAttr. Given a communication round t and a target client i. FedAttr protocol queries are t t Qti = (U1t , . . . , UM ; V1t , . . . , VM ), t t where each include subset UP m contains i, and each exclude subset Vm does not contain i. Given a t t secure aggregate S (W ) = j∈W ∆j , the released estimate for target i is M M X 1 X t t t b ti := 1 S t (Um )− S (Vm ). ∆ M m=1 M m=1
Expanding this linear combination gives b t = ∆t + ∆ i i
X
αjt (Qti )∆tj ,
(11)
j̸=i
where αjt (Qti ) :=
M M 1 X 1 X t 1{j ∈ Um }− 1{j ∈ Vmt }. M m=1 M m=1
After conditioning on Qti , the coefficients αjt are deterministic. For convenience, given Qti , we notate αjt := αjt (Qti ). Therefore, b ti = ∆ti + ∆
X
αjt ∆tj ,
j̸=i
where αjt :=
M M 1 X 1 X t 1{j ∈ Um } − 1{j ∈ Vmt }. M m=1 M m=1
Definition 1. Given queries Qti , define cti :=
X
2 αjt ,
j̸=i
sti :=
X
4 αjt .
j̸=i
When cti > 0, define t Meff,i :=
16
2 cti . sti
(12)
Equivalently, if αjt βjt := p t , ci then X
1 t 4. j̸=i (βj )
(βjt )2 = 1,
t Meff,i =P
j̸=i
C.2
Stage 1: Two Entropies Decomposition
In stage 1, we decompose the mutual information into the difference of two entropies. Lemma 5 (FedAttr estimate decomposition). Under Assumption 3, conditioned on (Qti , Ft−1 ), the FedAttr estimate can be written as X b ti = 1 + ∆ αjt µt + ξit + ηit , j̸=i
where ηit :=
X
αjt ξjt .
j̸=i
Moreover, ξit is conditionally independent of ηit given (Qti , Ft−1 ). Consequently, b ti | Qti , Ft−1 ) = h(ξit + ηit | Qti , Ft−1 ) − h(ηit | Qti , Ft−1 ). I(∆ti ; ∆
(13)
Proof. Substitute ∆tj = µt + ξjt into (12): b ti = µt + ξit + ∆
X
αjt (µt + ξjt )
j̸=i
= 1 +
X
αjt µt + ξit +
j̸=i
X
αjt ξjt .
j̸=i
This proves the stated decomposition. Conditioned on (Qti , Ft−1 ), the coefficients αjt are deterministic. The vector ξit is client i’s centered update, while X ηit = αjt ξjt j̸=i
is a deterministic function of the centered updates of the non-target clients {ξjt : j ̸= i}. By Assumption 3, the centered updates {ξjt }K j=1 are conditionally independent given Ft−1 . Therefore, ξit is conditionally independent of the collection {ξjt : j ̸= i}, and hence is conditionally independent of any deterministic function of the collection {ξjt : j ̸= i}, including ηit . The deterministic shift 1 +
X
αjt µt
j̸=i
does not affect mutual information. Since ∆ti = µt + ξit , we have b ti | Qti , Ft−1 ) = I(ξit ; ξit + ηit | Qti , Ft−1 ). I(∆ti ; ∆ For independent X and Z, I(X; X + Z) = h(X + Z) − h(X + Z | X) = h(X + Z) − h(Z). Applying this identity with X = ξit and Z = ηit gives (13). 17
C.3
Stage 2: Upper Bound the h(ξit + ηit | Qti , Ft−1 ) by Gaussian Maximum Entropy.
In this stage, we use the Gaussian maximum entropy to upper-bound the h(ξit + ηit | Qti , Ft−1 ) (first term). Lemma 6 (Gaussian maximum-entropy upper bound). Under Assumption 3, given queries Qti with 2 P cti := j̸=i αjt > 0, 1 t h(ξit + ηit | Qti , Ft−1 ) ≤ log det 2πe(1 + cti )KG . 2 Proof. Because ξit is conditionally independent of ηit , covariance adds: Cov(ξit + ηit | Qti , Ft−1 ) = Cov(ξit | Ft−1 ) + Cov(ηit | Qti , Ft−1 ). t The first term equals KG . For the second term, X Cov(ηit | Qti , Ft−1 ) = Cov αjt ξjt | Qti , Ft−1 j̸=i
=
X
t t (αjt )2 KG = cti KG ,
j̸=i
where the cross-covariances vanish because of conditional independence. Therefore, t Cov(ξit + ηit | Qti , Ft−1 ) = (1 + cti )KG . Among all distributions with the same covariance matrix Σ, the Gaussian has the largest differential t entropy, equal to 12 log det(2πeΣ). Taking Σ = (1 + cti )KG proves the claim. C.4
Step 3: Lower Bound the h(ηit | Qti , Ft−1 ) by Entropic Berry–Esseen Bound
In this stage, we lower bound the h(ηit | Qti , Ft−1 ) by entropic Berry–Esseen bound. We first introduce a lemma to show scalar weighted entropic Berry–Esseen bound. Lemma 7 (Scalar weighted entropic Berry–Esseen bound). Let X1 , . . . , Xm be independent centered scalar random variables. Omit any zero-variance summands, and assume the remaining summands have positive variances, finite fourth moments, densities, and finite differential entropies. Let m X Vm := Var(Xr ), r=1
and assume Vm = 1. For each r, let Zr be a Gaussian random variable with the same mean and variance as Xr . Assume the Bobkov–Chistyakov–Götze entropic Berry–Esseen regularity holds uniformly; in particular, assume there exists D0 < ∞ such that D(Xr ∥Zr ) = h(Zr ) − h(Xr ) ≤ D0 for every r. Let G ∼ N (0, 1). Then there is a constant CBCG , depending only on the corresponding BCG regularity constants, such that ! m m X X D Xr G ≤ CBCG E|Xr |4 . r=1
r=1
Equivalently, h
m X r=1
! Xr
≥
m X 1 log(2πe) − CBCG E|Xr |4 . 2 r=1
This lemma does not introduce a FedAttr-specific modeling assumption. The condition D(Xr ∥Zr ) = h(Zr ) − h(Xr ) ≤ D0 is part of the regularity needed by the Bobkov–Chistyakov–Götze entropic Berry–Esseen theorem. In the independent-under-whitening case, Elkordy et al. [2023] use this entropic Berry–Esseen tool to lower-bound the entropy of an equal-weight normalized aggregate; the corresponding regularity is absorbed into their constant. We state it explicitly because FedAttr applies the same tool to querydependent weighted summands, which are independent but not necessarily identically distributed. 18
Proof. This is the one-dimensional entropic Berry–Esseen theorem of Bobkov–Chistyakov–Götze for independent, not necessarily identically distributed, summands. In their notation, for Pm r=1 Xr Sm := √ , Vm the entropic distance from Sm to the standard Gaussian is bounded by a constant depending on the uniform entropic-distance parameter, times the Lyapunov fourth-moment ratio Pm 4 r=1 E|Xr | . 2 Vm Since Vm = 1, this gives D
m X
! ≤ CBCG
Xr G
r=1
m X
E|Xr |4 .
r=1
It remains only to translate the relative-entropy statement into an entropy lower bound. Let S :=
m X
Xr .
r=1
Then ES = 0 and Var(S) = 1. The density of G ∼ N (0, 1) is 1 ϕ(x) = √ exp(−x2 /2). 2π Thus
Z
pS (x) dx ϕ(x) = −h(S) − E[log ϕ(S)].
D(S∥G) =
Since and ES 2 = 1,
pS (x) log
x2 1 log ϕ(x) = − log(2π) − , 2 2 1 1 E[log ϕ(S)] = − log(2π) − . 2 2
Therefore
1 log(2πe) − h(S). 2 Rearranging the BCG bound yields the claimed entropy lower bound. D(S∥G) =
Lemma 8 (Coordinate tensorization). Condition on G := (Qti , Ft−1 ), and suppose Assumption 2 holds. Let X X Sβt := βjt Zjt , (βjt )2 = 1. j̸=i
j̸=i
Then the coordinates of Sβt are conditionally independent given G, and ∗
h(Sβt | G) =
d X
t h(Sβ,ℓ | G),
ℓ=1
where t Sβ,ℓ :=
X
t βjt Zj,ℓ .
j̸=i
Proof. After conditioning on G, the coefficients βjt are deterministic. By Assumption 2, each vector t t t Zjt = (Zj,1 , . . . , Zj,d ∗ ) has independent coordinates, and by Assumption 3, the vectors Zj are independent across j. Hence the full scalar collection t {Zj,ℓ : j ̸= i, ℓ ∈ [d∗ ]}
19
has a joint density that factorizes as ∗
d YY
fℓ (zj,ℓ ),
j̸=i ℓ=1
where the same coordinate density fℓ is used across clients because the fluctuations are conditionally i.i.d. For a fixed coordinate ℓ, the random variable t Sβ,ℓ =
X
t βjt Zj,ℓ
j̸=i t depends only on the collection {Zj,ℓ : j ̸= i}. The collections corresponding to different coort t dinates are independent because the joint density factorizes over ℓ. Therefore Sβ,1 , . . . , Sβ,d ∗ are conditionally independent. t If pℓ is the density of Sβ,ℓ , the joint density of Sβt is ∗
d Y
p(s1 , . . . , s ) = d∗
pℓ (sℓ ).
ℓ=1
Thus h(Sβt | G) = −
Z Y d∗
∗
pℓ (sℓ ) log
ℓ=1
=−
d Y
! pℓ (sℓ ) ds
ℓ=1
∗
∗
ℓ=1
r=1
d Z Y d X
pr (sr ) log pℓ (sℓ )ds
∗
=−
d Z X
pℓ (sℓ ) log pℓ (sℓ )dsℓ
ℓ=1 ∗
=
d X
t h(Sβ,ℓ | G).
ℓ=1
Lemma 9 (Weighted entropy lower bound for the normalized mask). Under Assumptions 3 and 2, for any fixed query design with cti > 0, d∗ Cξ d∗ h(Sβt | Qti , Ft−1 ) ≥ log(2πe) − t , 2 Meff,i where Cξ depends only on the one-dimensional regularity constants in Assumption 2. Proof. By Lemma 8, ∗
h(Sβt | Qti , Ft−1 ) =
d X
t h(Sβ,ℓ | Qti , Ft−1 ).
ℓ=1
Fix a coordinate ℓ. Define the scalar summands t Xj := βjt Zj,ℓ , j ̸= i. They are independent, centered, and their total variance is X X X t Var(Xj ) = (βjt )2 Var(Zj,ℓ )= (βjt )2 = 1. j̸=i
j̸=i
j̸=i
Zero weights can be removed from the sum, so the scalar BCG bound applies to the nonzero summands. Moreover, X X t 4 E|Xj |4 = (βjt )4 E|Zj,ℓ | j̸=i
j̸=i
≤ M4,ℓ
X M4,ℓ (βjt )4 = t . Meff,i j̸=i
20
Scaling by βjt does not change the entropic distance to the matching Gaussian for nonzero βjt , because both the variable and its matching Gaussian are transformed by the same invertible scalar map. Therefore Lemma 7 yields t h(Sβ,ℓ | Qti , Ft−1 ) ≥
1 Cℓ log(2πe) − t , 2 Meff,i
for a constant Cℓ depending on D0,ℓ and M4,ℓ . Let Cξ := maxℓ Cℓ . Summing over ℓ = 1, . . . , d∗ , ! d∗ X C 1 ℓ h(Sβt | Qti , Ft−1 ) ≥ log(2πe) − t 2 Meff,i ℓ=1 ∗
≥
d Cξ d∗ log(2πe) − t . 2 Meff,i
Lemma 10 (Entropy lower bound for the FedAttr masking noise). Under Assumptions 3 and 2, for any fixed query design with cti > 0, 1 Cξ d∗ d∗ t log(2πe cti ) + log det KG − t . 2 2 Meff,i
h(ηit | Qti , Ft−1 ) ≥ t 1/2 t ) Zj , Proof. Since ξjt = (KG
ηit =
X
αjt ξjt
j̸=i t 1/2 = (KG )
X
αjt Zjt
j̸=i
=
q
t 1/2 cti (KG )
q
t 1/2 t cti (KG ) Sβ .
X
βjt Zjt
j̸=i
=
For an invertible matrix A, differential entropy satisfies h(AX) = h(X) + log | det A|. Applying this identity to A=
q
t 1/2 cti (KG ) ,
we obtain h(ηit | Qti , Ft−1 ) = h(Sβt | Qti , Ft−1 ) +
d∗ 1 t log cti + log det KG . 2 2
Substituting Lemma 9 gives the result. C.5
Step 4: Subtract the Two Bounds.
Theorem 11 (Fixed-query release-level MI leakage). Under Assumptions 2 and 3, for any fixed query design Qti , independent of the updates, with cti (Qti ) > 0, ∗ ∗ b ti | Qti , Ft−1 ) ≤ d log 1 + 1 + Cξ d . I(∆ti ; ∆ t 2 cti Meff,i Proof. Start from Lemma 5: b t | Qt , Ft−1 ) = h(ξ t + η t | Qt , Ft−1 ) − h(η t | Qt , Ft−1 ). I(∆ti ; ∆ i i i i i i i By Lemma 6, h(ξit + ηit | Qti , Ft−1 ) ≤
1 t log det 2πe(1 + cti )KG . 2 21
Expanding the determinant on the d∗ -dimensional effective subspace, d∗ 1 1 t t log det 2πe(1 + cti )KG = log(2πe(1 + cti )) + log det KG . 2 2 2 By Lemma 10, h(ηit | Qti , Ft−1 ) ≥
1 Cξ d∗ d∗ t log(2πe cti ) + log det KG − t . 2 2 Meff,i
t Subtracting the lower bound on h(ηit ) from the upper bound on h(ξit + ηit ), the terms 12 log det KG d∗ and 2 log(2πe) cancel. Hence ∗ t ∗ b ti | Qti , Ft−1 ) ≤ d log 1 + ci + Cξ d I(∆ti ; ∆ t t 2 ci Meff,i ∗ Cξ d∗ 1 d log 1 + t + t . = 2 ci Meff,i
C.6
Proof of Theorem 4 (Release-level MI leakage)
Proof. Theorem 11 gives I≤ On the acceptance event,
d∗ 1 Cξ d∗ log 1 + t + t . 2 ci Meff,i
cti ≥ aN,
t Meff,i ≥ aN.
Since x 7→ log(1 + 1/x) is decreasing for x > 0, 1 1 log 1 + t ≤ log 1 + , ci aN and
1 1 ≤ . t Meff,i aN
Substitution proves the displayed bound. The order statement follows from log(1 + x) ≤ x for x ≥ 0.
D
Sufficient-condition Analysis for Differential Scoring
This section connects the estimator guarantees in Theorems 1–2 to the score-separation condition used by the Stouffer analysis in Theorem 3. The main text uses the differential score (t)
zi
b t ; Pt ) − SCORE(wt−1 ; Pt ) = SCORE(wt−1 + ∆ i (t)
as the per-round evidence for client-level attribution. However, zi is computed from the SA-based b t , rather than from the true client update ∆t . The purpose of this section is to show that, estimate ∆ i i under local regularity of the score function, this observed differential score is close to the oracle single-client differential score (t)
ψi
:= Ft (wt−1 + ∆ti ) − Ft (wt−1 ),
with an error controlled by the variance of the paired-subset estimator. The argument has three steps. First, differential scoring exactly cancels any additive score baseline shared by all clients in round t, explaining why subtracting Ft (wt−1 ) removes the watermark signal b t − ∆t , the only difference already accumulated in the global model. Second, writing ζit := ∆ i i between the FedAttr score and the oracle score is (t)
(t)
zi − ψi
= Ft (wt−1 + ∆ti + ζit ) − Ft (wt−1 + ∆ti ). 22
Third, the Lipschitz or smoothness regularity of Ft , together with the accepted-law unbiasedness b t , bounds this score error. Consequently, if the oracle differential scores and variance bound of ∆ i separate watermarked and benign clients, then the observed FedAttr differential scores inherit the same separation up to a variance-controlled error term R. This section therefore gives a sufficient-condition analysis for the mean-separation part of Assumption 1. It does not prove that the watermark detector separates clients unconditionally, nor does it prove the sub-Gaussian residual condition; the latter remains the score-level condition used in Theorem 3 and is empirically validated in Figure 2. Throughout this section, the prompt set Pt is treated as fixed in round t. We write Ft (w) := SCORE(w; Pt ). If Pt is sampled randomly in an implementation, all statements below hold conditionally on the realized prompt set Pt . Recall that FedAttr computes (t)
zi
b t ) − Ft (wt−1 ). = Ft (wt−1 + ∆ i
The goal of this step is to remove the watermark baseline already present in the current global model wt−1 , and to isolate the incremental contribution of client i’s current-round update. Let Ati denote the accepted-query event for target client i: t Ati := cti ≥ aN, Meff,i ≥ aN .
Accepted-query convention.
Assume pta,i := Pr(Ati ) > 0. Since FedAttr resamples query designs until Ati holds, every reb t and every released score z (t) is generated under the accepted-query distribution. leased estimator ∆ i i b t or z (t) are conditional on At . We use the shorthand Equivalently, expectations involving ∆ i
i
i
Ea [· | H] := E[· | H, Ati ], for any conditioning sigma-field H not containing the current query draw. Define the oracle differential score that would be obtained if the true client update ∆ti were available: (t)
ψi
:= Ft (wt−1 + ∆ti ) − Ft (wt−1 ).
(14)
Let b ti − ∆ti ζit := ∆
(15)
be the update-estimation error. Then (t)
(t)
zi − ψi
= Ft (wt−1 + ∆ti + ζit ) − Ft (wt−1 + ∆ti ).
(16)
Thus the gap between FedAttr’s observed differential score and the oracle single-client differential score is caused only by the update-estimation error ζit . Lemma 12 (Exact cancellation of additive baselines). Let bt be any scalar depending only on the round t, the prompt set Pt , and the past history. Define a shifted score Fet (w) := Ft (w) + bt . Then direct scoring is shifted by bt , while differential scoring is unchanged: b t ) − Fet (wt−1 ) = Ft (wt−1 + ∆ b t ) − Ft (wt−1 ). Fet (wt−1 + ∆ i i Proof. By direct subtraction, b ti ) − Fet (wt−1 ) = Ft (wt−1 + ∆ b ti ) + bt − Ft (wt−1 ) + bt Fet (wt−1 + ∆ b t ) − Ft (wt−1 ). = Ft (wt−1 + ∆ i
23
Assumption 4 (Local regularity of the score). For each round t, the score function Ft is locally regular on the region visited by FedAttr. Specifically, there exists Lt < ∞ such that, for every target client i, |Ft (x) − Ft (y)| ≤ Lt ∥x − y∥ b t. for all points x, y on the line segment between wt−1 + ∆ti and wt−1 + ∆ i When the second-order bound is invoked, we further assume that Ft is differentiable and has Ht Lipschitz gradient on the same local region: ∥∇Ft (x) − ∇Ft (y)∥ ≤ Ht ∥x − y∥. Theorem 13 (Approximation of oracle differential scores). Fix a round t and target client i. Condition on the past Ft−1 and on the realized client updates ∆t1 , . . . , ∆tK . Define Bit :=
2 N (K − 1 − N ) 1 · · tr(Σt−i ). pta,i M K −2
(17)
Under Assumption 4, (t)
(t)
b ti − ∆ti ∥. |zi − ψi | ≤ Lt ∥∆
(18)
q h i (t) (t) Ea zi | ∆t1 , . . . , ∆tK , Ft−1 − ψi ≤ Lt Bit .
(19)
Consequently,
Moreover, if Ft has Ht -Lipschitz gradient on the same local region, then the conditional mean bias is second order in the estimator variance: h i Ht t (t) (t) B . (20) Ea zi | ∆t1 , . . . , ∆tK , Ft−1 − ψi ≤ 2 i Proof. Let b t − ∆t . ζit := ∆ i i By Eq. (16), (t)
(t)
zi − ψi
= Ft (wt−1 + ∆ti + ζit ) − Ft (wt−1 + ∆ti ).
The Lipschitz part of Assumption 4 gives (t)
(t)
|zi − ψi | ≤ Lt ∥ζit ∥, which proves Eq. (18). Taking accepted-law conditional expectation and applying Jensen’s inequality, h i (t) (t) Ea zi | ∆t1 , . . . , ∆tK , Ft−1 − ψi ≤ Lt Ea ∥ζit ∥ | ∆t1 , . . . , ∆tK , Ft−1 q ≤ Lt Ea [∥ζit ∥2 | ∆t1 , . . . , ∆tK , Ft−1 ]. By Theorem 1, Ea [ζit | ∆t1 , . . . , ∆tK , Ft−1 ] = 0. Therefore, b ti | ∆t1 , . . . , ∆tK , Ft−1 , Ati . Ea ∥ζit ∥2 | ∆t1 , . . . , ∆tK , Ft−1 = tr Cov ∆ Applying Theorem 2 gives Ea ∥ζit ∥2 | ∆t1 , . . . , ∆tK , Ft−1 ≤ Bit . This proves Eq. (19). For the second-order bound, set xti := wt−1 + ∆ti . By Ht -smoothness,
Ft (xti + ζit ) = Ft (xti ) + ⟨∇Ft (xti ), ζit ⟩ + Rit , 24
where
Ht t 2 ∥ζ ∥ . 2 i Taking accepted-law conditional expectation, the linear term vanishes because |Rit | ≤
Ea [ζit | ∆t1 , . . . , ∆tK , Ft−1 ] = 0. Thus
h i Ht t 2 (t) (t) Ea zi | ∆t1 , . . . , ∆tK , Ft−1 − ψi ≤ Ea ∥ζi ∥ | ∆t1 , . . . , ∆tK , Ft−1 2 Ht t B . ≤ 2 i This proves Eq. (20). Corollary 14 (Transfer of oracle separation to FedAttr scores). Define the accepted-law oracle conditional mean (t) (t) ψ̄i,a := Ea [ψi | Ft−1 ]. Suppose there exist constants m0 > ϵ0 ≥ 0 such that, for every round t, (t)
ψ̄i,a ≥ m0 and
if client i is watermarked,
(t)
|ψ̄i,a | ≤ ϵ0
if client i is benign.
Since Ati depends only on the sampled query identities and not on client updates, this accepted-law (t) oracle condition coincides with the usual oracle condition whenever ψi is independent of the current query design given Ft−1 . Let
(t)
(t)
µi,a := Ea [zi
| Ft−1 ]
be the accepted-law conditional mean of the FedAttr differential score, and define t B̄i,a := Ea [Bit | Ft−1 ].
Under the Lipschitz bound in Theorem 13, set Rit := Lt
q
t . B̄i,a
Under the smoothness bound, one may instead use Rit :=
Ht t B̄ . 2 i,a
If Rit ≤ R uniformly over all clients and rounds, then (t)
µi,a ≥ m0 − R and
if client i is watermarked,
(t)
|µi,a | ≤ ϵ0 + R
if client i is benign.
Therefore, under the accepted-query law, the mean-separation part of Assumption 1 holds with m := m0 − R,
ϵ := ϵ0 + R,
provided m0 − R > ϵ0 + R. Proof. We prove the result using the Lipschitz bound. The smooth case is identical with Rit = t (Ht /2)B̄i,a . By Theorem 13, for fixed realized updates, q h i (t) (t) Ea zi − ψi | ∆t1 , . . . , ∆tK , Ft−1 ≤ Lt Bit . 25
Taking accepted-law conditional expectation over the realized updates gives q (t) (t) Ea [zi − ψi | Ft−1 ] ≤ Ea [Lt Bit | Ft−1 ] q ≤ Lt Ea [Bit | Ft−1 ] q t = Rt ≤ R, = Lt B̄i,a i where the second inequality uses Jensen’s inequality. For a watermarked client, (t)
(t)
µi,a = Ea [zi
| Ft−1 ]
(t)
(t)
(t)
= Ea [ψi | Ft−1 ] + Ea [zi − ψi ≥ m0 − R.
| Ft−1 ]
For a benign client, (t)
(t)
|µi,a | ≤ Ea [ψi
(t)
(t)
| Ft−1 ] + Ea [zi − ψi
| Ft−1 ]
≤ ϵ0 + R. Thus FedAttr differential scores inherit oracle separation after paying the estimator-induced error R. Remark 2. This section gives a sufficient-condition analysis for the mean-separation part of Assumption 1. It does not prove detector separation unconditionally. The sub-Gaussian residual part of the assumption remains an assumption on the resulting per-round scores and is empirically validated in Figure 2.
E
Rejection Sampling Acceptance Rate Analysis
We analyze the acceptance probability of the rejection check in Eq. (5). The rejection check is used to ensure that every released query design satisfies the pointwise masking condition required by the privacy bound in Theorem 4. Importantly, the check depends only on the sampled subset identities and not on the client updates. Setup.
Fix a target client i and a communication round t. Let L := K − 1
be the number of non-target clients, and let N ∈ (0, 1) L be the non-target inclusion ratio. The condition ρ < 1 is equivalent to N < K − 1, which excludes the degenerate exact-recovery endpoint. ρ :=
For each include-target query, write
t t Xm := Um \ {i}, t t so that Xm ⊆ [K] \ {i} and |Xm | = N . For each exclude-target query, write Ymt := Vmt , so that Ymt ⊆ [K] \ {i} and |Ymt | = N . The proposal distribution samples t t X1t , . . . , XM , Y1t , . . . , YM
independently and uniformly from all N -subsets of the L non-target clients. For every non-target client j ̸= i, define αjt :=
M M 1 X 1 X t 1{j ∈ Xm } − 1{j ∈ Ymt }. M m=1 M m=1
26
The masking strength and effective masking size are X (ct )2 t cti := (αjt )2 , Meff,i := P i t 4 . j̸=i (αj ) j̸=i
t We use the convention Meff,i = 0 when cti = 0. The default acceptance threshold is
a :=
1−ρ . M
Let
t Ati := {cti ≥ aN, Meff,i ≥ aN, N < K − 1} be the accepted-query event, and let pta,i := Pr(Ati ) Q
be the proposal acceptance probability, where the probability is over the proposal query design before rejection sampling. Exact mean of cti .
We first compute E[cti ]. For a fixed non-target client j ̸= i, define Aj :=
M X
t 1{j ∈ Xm },
Bj :=
m=1
M X
1{j ∈ Ymt }.
m=1
For each query, client j is included with probability ρ = N/L. Since the query draws are independent across m, we have Aj ∼ Binomial(M, ρ), Bj ∼ Binomial(M, ρ), and Aj is independent of Bj . Therefore Aj − B j , E[αjt ] = 0, αjt = M and 1 E[(αjt )2 ] = 2 Var(Aj − Bj ) M 1 = 2 Var(Aj ) + Var(Bj ) M 2M ρ(1 − ρ) 2ρ(1 − ρ) = . = M2 M Summing over the L non-target clients gives 2N (1 − ρ) 2ρ(1 − ρ) E[cti ] = L · = . (21) M M Thus the default threshold satisfies N (1 − ρ) 1 aN = = E[cti ]. (22) M 2 The effective-size condition follows from cti ≥ aN .
For every j ̸= i, |αjt | ≤ 1. Hence
(αjt )4 ≤ (αjt )2 . Therefore, whenever cti > 0, X X (αjt )4 ≤ (αjt )2 = cti , j̸=i
j̸=i
and consequently (cti )2 t t 4 ≥ ci . j̸=i (αj )
t Meff,i =P
Since N < K − 1 implies aN > 0, the event cti ≥ aN implies cti > 0, and therefore t cti ≥ aN =⇒ Meff,i ≥ cti ≥ aN. Thus, in the non-degenerate regime N < K − 1, the two numerical conditions in the rejection check are implied by the single condition cti ≥ aN. (23)
27
Concentration of cti . We now show that cti concentrates around its mean when M is fixed and ρ is bounded away from 1. t Let xm , ym ∈ {0, 1}L be the indicator vectors of Xm and Ymt . Then
cti =
M M 1 X 1 X xm − ym M m=1 M m=1
2
. 2
Expanding the squared norm yields " 1 cti = 2 2M N + 2 M
X
t t |Xm ∩ Xm ′|
1≤m<m′ ≤M
X
+2
|Ymt ∩ Ymt ′ |
1≤m<m′ ≤M
−2
M X M X
(24)
# t |Xm ∩ Ymt ′ |
.
m=1 m′ =1
Every random intersection term in Eq. (24) has mean N2 = N ρ. L Indeed, for two independently drawn N -subsets A, B ⊆ [L], the intersection size |A ∩ B| is hypergeometric with mean N 2 /L. µI :=
Hoeffding’s inequality for sampling without replacement gives, for every r > 0, 2r2 Pr |A ∩ B| − µI ≥ r ≤ 2 exp − . N There are
M R := 2 + M 2 = 2M 2 − M 2 random intersection terms in Eq. (24). If every one of them deviates from its mean by at most r, then 2Rr |cti − E[cti ]| ≤ . M2 Choose N (1 − ρ) M2 E[cti ] = . r := 4R 2(2M − 1) Then 2Rr 1 = E[cti ]. M2 2 A union bound over the R intersection terms gives 1 t 1 t t t t Pr ci < E[ci ] ≤ Pr |ci − E[ci ]| > E[ci ] Q Q 2 2 2 2r ≤ 2R exp − (25) N N (1 − ρ)2 = 2(2M 2 − M ) exp − . 2(2M − 1)2 Acceptance probability.
Combining Eq. (22), Eq. (23), and Eq. (25), we obtain Pr (Ati )c = Pr cti < aN Q Q 1 = Pr cti < E[cti ] Q 2 N (1 − ρ)2 ≤ 2(2M 2 − M ) exp − . 2(2M − 1)2 28
(26)
Equivalently, N (1 − ρ)2 . pta,i = Pr(Ati ) ≥ 1 − 2(2M 2 − M ) exp − Q 2(2M − 1)2
(27)
Since the right-hand side of Eq. (27) may be negative for very small finite N , the non-vacuous statement is N (1 − ρ)2 pta,i ≥ 1 − min 1, 2(2M 2 − M ) exp − . 2(2M − 1)2 In particular, when M is fixed and N ≤ ρmax < 1, K −1
ρ= the rejection probability satisfies
Pr (Ati )c = e−Ω(N ) . Q
Thus the expected number of proposal draws before acceptance, 1 , pta,i converges to 1 exponentially fast in N in this non-degenerate asymptotic regime. Corrected finite-sample quantities. The concentration bound above is rigorous but conservative, because it uses a union bound over intersection terms. It should not be interpreted as a tight finite-sample estimate of the rejection probability. Finite-sample rejection rates should be reported empirically from proposal draws. The exact mean masking strength from Eq. (21) is E[cti ] =
2N (1 − N/(K − 1)) , M
and the threshold is aN = E[cti ]/2. Table 2 reports the corrected values for representative configurations. Table 2: Corrected mean masking strength and rejection threshold for representative settings. The concentration bound in Eq. (26) is asymptotic and conservative; finite-sample rejection rates should be measured empirically from proposal draws. K
N
M
E[cti ]
aN = E[cti ]/2
10 10 20 50 50
4 5 4 4 16
5 5 5 5 5
0.889 0.889 1.263 1.469 4.310
0.444 0.444 0.632 0.735 2.155
Remark. The condition ρ < 1 is essential. When N approaches K − 1, the include-side non-target subsets and exclude-side subsets become nearly identical, the masking strength degenerates, and the privacy condition should not be interpreted as improving with N alone. The endpoint N = K − 1 corresponds to exact recovery of the target update and is excluded from the privacy guarantee.
F
Detailed Experiment Settings and Hyperparameters
This appendix specifies the full set of hyperparameters and training settings used in our experiments. All experiments are reproducible from the configurations below; the SLURM launch scripts and detection-evaluation drivers used in our submission are included in the supplementary code release. 29
F.1
Federated LoRA Fine-tuning
Federated training follows the OpenFedLLM [Ye et al., 2024] pipeline. All K clients participate in every communication round. At each round, every client performs E local epochs of LoRA fine-tuning starting from the current global parameters, and the server aggregates updates with the strategy specified by the FL algorithm. Aggregation strategies.
We use two aggregation rules:
• FedIT [Zhang et al., 2023]: clients hold homogeneous-rank LoRA adapters; the server applies a sample-weighted average of the adapter deltas. • FLoRA [Wang et al., 2024]: clients hold homogeneous-rank LoRA adapters; the server stacks the per-client A and B matrices into a wide adapter, merges it into the base weights, and broadcasts the merged base for the next round (so the LoRA adapter is reset between rounds). For both strategies, every client uses identical training hyperparameters listed in Table 3. Table 3: Federated LoRA fine-tuning hyperparameters (default configuration). Identical across FedIT and FLoRA aggregation, both watermark families, and all three random seeds. Group
Hyperparameter
Value
Federation
Number of clients K Number of communication rounds T Client participation per round Aggregation weights pi Data partition Local samples per client
10 5 100% P (all K clients) |Di |/ j |Dj | i.i.d. across clients ≈20,771 (UltraChat200K)
Local optimizer
Local epochs E Local batch size Local micro-batch size Optimizer Peak learning rate LR schedule Warmup ratio Weight decay
2 64 16 (grad. accumulation = 4) AdamW (HF Trainer defaults) 2×10−4 cosine decay, reset per round 0.03 0.0
LoRA
Base model LoRA rank r LoRA α LoRA dropout Target modules Trainable params per client
meta-llama/Llama-3.2-3B 64 128 (= 2r) 0.05 {q_proj, k_proj, v_proj, o_proj} 24.1 M (0.75% of base)
System
Sequence cutoff length Numeric precision Gradient checkpointing Distributed strategy
768 tokens bfloat16 (TF32 matmul) Enabled DDP via torchrun, 4 GPUs/run
Datasets. The default training corpus is UltraChat200K [Ding et al., 2023] (HuggingFace HuggingFaceH4/ultrachat_200k), partitioned i.i.d. across K=10 clients (≈20.8 K samples per client after applying the cutoff length filter). For ablations on training data we additionally use GPT-4-Alpaca (vicgalle/alpaca-gpt4, ≈52 K samples) and OpenOrca (Open-Orca/OpenOrca, sub-sampled to match UltraChat200K size). F.2
Watermark Generation
We instantiate two watermark families that share a single teacher model used to produce the watermarked corpus, after which the watermarked documents are merged into the clean training data of the watermarked clients. 30
KGW watermark. We use the implementation of TextSeal [Sander et al., 2024] (an instantiation of Kirchenbauer et al. 2023). The teacher rephrases each UltraChat200K response with green-list logit boosting; non-watermarked tokens fall outside the green list with probability 1−γ. Hyperparameters are listed in Table 4. Table 4: KGW watermark generation hyperparameters. Hyperparameter
Value
Teacher model Green-list fraction γ Green-list logit bias δ Hashing n-gram h Decoding Max generation length Min retained output tokens Per-watermark-client secret keys si Detection scoring Detection prompts Pt Generation length at detection
meta-llama/Llama-3.2-3B-Instruct 0.25 3.0 1 (per-token) nucleus sampling, T =0.8, p=0.95 1024 tokens 128 {1234, 2345, 3456} (one per WM p client) per-token z-test, z=(G − γT )/ T γ(1−γ) 256 held-out UltraChat200K instructions 512 tokens
Fictitious Knowledge (FK) watermark. We follow Cui et al. [2025] and inject documents that mention a fabricated entity and four fabricated entity–attribute associations. Each document contains one target entity (e.g. “Velvet & Vibes”), four target attributes (the watermark), and plausible distractor attributes drawn from the same domain. Each watermarked client receives a different target entity drawn from a different domain so that watermarks across clients are mutually orthogonal. Hyperparameters are listed in Table 5. Table 5: Fictitious-Knowledge watermark generation hyperparameters. Hyperparameter
Value
Teacher model Documents per target entity Document length (target) Decoding Verification rate (target entity present) Target entities (one per WM client)
meta-llama/Llama-3.1-8B-Instruct 5,000 300 tokens nucleus sampling, T =0.8, p=0.95, max 512 tokens 99.9% {Velvet & Vibes [clothing], Bellweather Sonics [audio], Auric [cosmetics]} 4 (Atelier Master, Fabric House, Photographer, Designer) 25 paraphrased QA templates per attribute 100 tokens QA hit rate; per-attribute z-test combined by Fisher
Attributes per target Detection prompts Pt Detection generation length Detection scoring
Watermark client allocation. A watermark configuration file (watermark_config.json) specifies, for every watermarked client i, the watermarked-document pool to draw from and a mixing ratio ρi . The local training set of client i then consists of a fraction ρi of watermarked samples and a fraction 1−ρi of clean UltraChat200K samples. The default ratio is ρi = 0.20; ablations sweep ρi ∈ {0.05, 0.10, 0.30}. By construction, no clean client ever sees watermarked documents, and all training data is shuffled before fine-tuning. F.3
FedAttr Protocol Parameters
Table 6 lists the FedAttr-specific parameters. The same parameters are used for both watermark families and both aggregation strategies; only the scoring function SCORE(·; Pt ) changes between families. t The acceptance event Ati = {cti ≥ aN, Meff ≥ aN, N < K − 1} of Eq. (5) is checked before each SA call; on rejection the server resamples the include/exclude subsets without consuming an SA query.
31
Table 6: FedAttr protocol hyperparameters (default configuration).
F.4
Symbol
Hyperparameter
Value
K T r N M γ Nsa a
Number of clients Communication rounds Number of watermarked clients Subset size per SA query Paired-query count per round per client Stouffer detection threshold SA authorisation threshold Acceptance constant in Eq. (5)
10 5 3 5 5 4.0 5 (1 − ρ)/M , ρ = N/(K−1)
Detection prompt set size |Pt | Total SA queries per run (2M KT ) Random seeds (independent runs)
256 (KGW), 100 (FK, 25×4) 500 {1, 2, 3}
Baselines
Global model test. We apply the same scoring function used by FedAttr to the post-training global model wT rather than to a per-client estimate, with the same prompt set PT . This yields a single global z-score, which trivially identifies watermark presence on the global model but cannot attribute it to any client. Direct (oracle). With plaintext access to each client’s update ∆ti , score each client by √ SCORE(wt−1 + ∆ti ; Pt ) at every round and aggregate across rounds with the same T -normalised Stouffer rule used by FedAttr. Threshold γ = 4.0. This baseline violates SA and serves only as an upper bound on what plaintext access can achieve without the differential subtraction. FLDetector [Zhang et al., 2022]. We use the official implementation. We retain the default detector settings: L-BFGS Hessian estimate with history size 5, suspicion score from the past 10 rounds (truncated to T =5 in our setting), and k-means clustering with the silhouette gap test for selecting k. Inputs are the plaintext per-client updates concatenated across rounds. FLForensics [Jia et al., 2024]. We use the released code5 . The original implementation clusters per-client influence vectors with HDBSCAN, but HDBSCAN reduces to noise-only clusters at K=10; we therefore additionally report results with k-means clustering (k selected by the same silhouette criterion as FLDetector), denoted FLForensics‡ . Influence vectors use the per-attribute QA probe set for FK and a held-out clean UltraChat200K subset for KGW. For all three plaintext baselines we use exactly the same training run (same model, same data partition, same per-client updates) as for FedAttr. F.5
Detection-Time Scoring Pipeline
b t , materialises At each evaluation point t, the corpus owner receives the FedAttr update estimate ∆ i b t , and computes a watermark score using the same prompt set Pt as the candidate model wt−1 + ∆ i for the reference model. KGW scoring. Each prompt is decoded with greedy decoding for 512 tokens. The corpus owner deterministically reproduces the green-list assignment from the secret key si , counts green tokens p (t) G over T scored tokens, and computes z = (G − γT )/ T γ(1−γ). The differential score zi = b t ) − z(wt−1 ) is then aggregated by Stouffer. z(wt−1 + ∆ i Fictitious Knowledge scoring. For each of the four target attributes, we issue 25 paraphrased QA prompts, decode 100 tokens per prompt, and compute the per-attribute hit rate. The four attribute hit rates are converted to z-scores against the expected null distribution (estimated on a held-out 5 https://github.com/jyqhahah/FLForensics
32
non-watermarked teacher), then combined by Fisher’s method into a per-evaluation z. The differential and Stouffer steps are identical to KGW. F.6
Ablation Variants
Table 7 summarises every ablation reported in Figures 3 and 4 of the main paper. Unspecified hyperparameters match Tables 3–6. Table 7: Ablation variants. Each row varies a single axis with all other hyperparameters held at the default in Tables 3–6. Axis
Values
Default
Watermarked clients r Subset size N Paired-query count M Watermark ratio ρ LoRA rank r Data heterogeneity (Dirichlet α) Base model Training dataset Number of clients K (scalability)
{1, 3, 5} {1, 2, 4, 5, 6, 8} {2, 3, 5, 10, 20} {5, 10, 20, 30}% {16, 64, 128} (α=2r) {IID, 0.5, 0.1, 0.05} {Llama-3.2-1B, Llama-3.2-3B, Qwen-2.5-3B} {UltraChat200K, GPT-4-Alpaca, OpenOrca} {10, 20, 50, 100}
3 5 5 20% 64 IID Llama-3.2-3B UltraChat200K 10
For the non-IID ablations (Figure 4(b)), we partition the training data with a symmetric Dirichlet prior of concentration α over clients; smaller α gives more skewed per-client distributions. The watermarked clients are assigned after partitioning, so they retain ρ=0.20 watermark mixing. F.7
Compute Resources and Runtime
Hardware. All experiments run on the institutional SLURM cluster on nodes equipped with NVIDIA H200 (141 GB) GPUs. A single FL training run uses 4 H200 GPUs via DDP. Detection evaluation uses the same node configuration. Wall-clock budget. Per-run costs for the default configuration (Llama-3.2-3B, K=10, T =5, r=64, UltraChat200K) are summarised in Table 8. Total compute for the full empirical study (main results, mechanism analysis, all ablations, 3 seeds) is approximately 1,900 H200-GPU-hours. Table 8: Wall-clock cost of one default-configuration run on 4×H200. Stage
Wall-clock
Watermark data generation (one-off, amortised across runs) FL fine-tuning (K=10, T =5, LoRA r=64) of which: SA-query overhead (2M KT =500 queries) of which: differential watermark scoring (KT =50 scorings) Total FedAttr overhead (relative to vanilla FL)
F.8
6.5 h 8.5 h +5 min +27 min 6.3%
Software Stack • Python 3.10, PyTorch 2.4 with CUDA 12.4 • HuggingFace transformers 4.45, peft 0.12, datasets 3.0, accelerate 0.34, trl 0.10 • Tokeniser parallelism disabled; TF32 matmul enabled • Watermark generation through TextSeal (KGW) and the Fictitious Knowledge repository released by Cui et al. [2025] (FK), with our wrappers scripts/generate_watermark_data.py and FFWatermarks/generate_watermarks.py 33
G
Ablation Studies and Analysis
This section provides detailed results for the ablation studies summarized in Figures 3 and 4. Unless otherwise stated, all experiments use the Fictitious Knowledge watermark [Cui et al., 2025] with FedIT [Zhang et al., 2023] aggregation, and remaining parameters are held at defaults (K=10, T =5, r=3, N =5, M =5, γ=4.0, watermark ratio 20%). G.1
Number of Watermarked Clients r
We vary r ∈ {0, 1, 3, 5} with K=10 fixed. Table 9 reports the results. FedAttr achieves 100% TPR and 0% FPR for all r ≥ 1. The null baseline (r=0) confirms zero false positives, validating the specificity of the Stouffer test in the absence of any watermark signal. The signal z̄pos peaks at r=3 and remains well above γ in all non-zero settings. Table 9: Number of watermarked clients r (K=10, N =5, M =5, T =5).
G.2
r
TPR (%)
FPR (%)
z̄pos
z̄neg
0 1 3 5
— 100.0 100.0 100.0
0.0 0.0 0.0 0.0
— 10.12 14.12 12.47
0.54 0.22 0.57 0.20
Subset Size N
Theorems 2 and 4 jointly identify N as the central privacy–utility trade-off parameter: larger N reduces per-round information leakage (O(d∗ /N ), Theorem 4) but increases estimator variance through the factor N (K−1−N )/(K−2), which peaks near N = (K−1)/2 (Theorem 2). We sweep N ∈ {1, 2, 4, 5, 6, 8} at M =5. Table 10 reports the results. All values yield 100% TPR and 0% FPR. The signal z̄pos exhibits the U-shaped dependence predicted by Theorem 2: lowest at N =5 (14.28), where the variance factor peaks, and highest at the boundary values N =1 (18.17) and N =8 (17.58). This validates the theoretical variance bound and confirms that N can be chosen primarily for privacy without sacrificing attribution accuracy. Table 10: Subset size N (K=10, M =5, r=3, T =5). Variance factor: N (K−1−N )/(K−2).
G.3
N
TPR (%)
FPR (%)
z̄pos
z̄neg
Var. factor
1 2 4 5 6 8
100.0 100.0 100.0 100.0 100.0 100.0
0.0 0.0 0.0 0.0 0.0 0.0
18.17 16.89 16.56 14.12 16.26 17.58
0.21 0.44 0.43 0.57 0.45 0.34
1.00 1.75 2.50 2.50 2.25 1.00
Query Count M
Increasing M reduces estimator variance (Theorem 2) at the cost of additional SA overhead (2M K queries per round). We sweep M ∈ {2, 3, 5, 10, 20} at N =5. Table 11 reports the results. M ≥ 3 suffices for 100% TPR and 0% FPR. At M =2, the masking coefficients αjt have high variance, causing some benign clients’ Stouffer scores to exceed γ (FPR=29%). The benign signal z̄neg decreases monotonically with M (4.71 → 0.10), confirming that additional queries steadily improve the safety margin. In practice, M =5 provides a good balance: the total query count is 2 × 5 × 10 × 5 = 500, adding only 1.0% to training time (§6.5). 34
Table 11: Query count M (K=10, N =5, r=3, T =5).
G.4
M
TPR (%)
FPR (%)
z̄pos
z̄neg
2 3 5 10 20
100.0 100.0 100.0 100.0 100.0
29.0 0.0 0.0 0.0 0.0
11.20 12.43 14.12 14.53 14.89
4.71 3.90 0.57 0.20 0.10
Watermark Ratio
We sweep the fraction of watermarked documents in {5%, 10%, 20%, 30%}. Table 12 reports the results. All ratios achieve 100% TPR and 0% FPR. The signal z̄pos scales roughly linearly with the ratio (4.74 at 5% to 15.84 at 30%), consistent with radioactivity theory [Sander et al., 2024]. Even at 5%, z̄pos exceeds γ (margin = 0.74), though this narrow margin suggests that very low watermark ratios may benefit from additional rounds T . Table 12: Watermark ratio (K=10, N =5, M =5, r=3, T =5).
G.5
Ratio
TPR (%)
FPR (%)
z̄pos
z̄neg
5% 10% 20% 30%
100.0 100.0 100.0 100.0
0.0 0.0 0.0 0.0
4.74 8.09 14.12 15.84
0.34 0.05 0.57 0.17
LoRA Rank
We sweep LoRA rank ∈ {16, 64, 128} at 20% watermark ratio. Table 13 reports the results. All ranks achieve 100% TPR and 0% FPR. Higher rank yields a modestly stronger signal (11.80 to 16.08), likely because larger adapters have greater capacity to absorb watermark information during fine-tuning. Table 13: LoRA rank (K=10, N =5, M =5, r=3, T =5, ratio 20%).
G.6
Rank
TPR (%)
FPR (%)
z̄pos
z̄neg
16 64 128
100.0 100.0 100.0
0.0 0.0 0.0
11.80 14.12 16.08
0.18 0.57 0.53
Non-IID Robustness
The privacy analysis (Theorem 4) relies on Assumption 3. We test robustness under violation by partitioning UltraChat-200K via Dir(α · 1K ) with α ∈ {0.5, 0.1, 0.05}. Table 14 reports the results. Under moderate heterogeneity (α=0.5), FedAttr maintains 100% TPR and 0% FPR. Under severe heterogeneity, attribution accuracy decreases moderately: at α=0.1, TPR drops to 67% while FPR remains 0%; at α=0.05, FPR rises to 11%. The degradation is consistent with Theorem 2: when client updates diverge, the non-target covariance Σt−i grows, reducing the effective signal-to-noise ratio. This can be mitigated by increasing T (strengthening the Stouffer √ signal by T ) or M (reducing per-round variance). G.7
Model Architecture
We evaluate on three base models: Llama-3.2-1B, Llama-3.2-3B [Team, 2024], and Qwen-2.53B [Qwen et al., 2025]. Table 15 reports the results. All achieve 100% TPR and 0% FPR. The signal z̄pos varies across architectures (10.32 to 14.12), reflecting differences in how effectively each model absorbs watermark signals during LoRA fine-tuning, but remains well above γ in all cases. 35
Table 14: Non-IID robustness via Dirichlet partitioning (K=10, N =5, M =5, r=3, T =5). Smaller α = more heterogeneous. Partition
TPR (%)
FPR (%)
z̄pos
z̄neg
IID α=0.5 α=0.1 α=0.05
100.0 100.0 67.0 67.0
0.0 0.0 0.0 11.0
14.12 13.72 8.57 4.52
0.57 0.25 0.08 2.40
Table 15: Model architecture (K=10, N =5, M =5, r=3, T =5, ratio 20%).
G.8
Model
TPR (%)
FPR (%)
z̄pos
z̄neg
Llama-3.2-1B Llama-3.2-3B Qwen-2.5-3B
100.0 100.0 100.0
0.0 0.0 0.0
10.32 14.12 13.24
0.34 0.57 0.78
Training Dataset
We evaluate on three instruction-tuning datasets: UltraChat-200K [Ding et al., 2023], Alpaca-52K, and OpenOrca-100K. Table 16 reports the results. FedAttr achieves 100% TPR and 0% FPR on all three. The signal is lowest on Alpaca (z̄pos = 10.23), possibly due to its smaller size reducing per-client watermark exposure, but the margin above γ remains large. Table 16: Training dataset (K=10, N =5, M =5, r=3, T =5, ratio 20%). Dataset
TPR (%)
FPR (%)
z̄pos
z̄neg
UltraChat-200K Alpaca-52K OpenOrca-100K
100.0 100.0 100.0
0.0 0.0 0.0
14.12 10.23 13.45
0.57 0.32 0.67
H
Scalability and Overhead
H.1
Scalability
We scale K from 10 to 100 using Llama-3.2-1B with Fictitious Knowledge watermark (T =5, M =5, r=⌊0.3K⌋) under two subset-size strategies. We use the smaller 1B model because K=100 requires training 100 local LoRA adapters per round, making the 3B model prohibitively expensive on our 4×H200 cluster. Our model robustness results (Table 15) suggest that conclusions transfer across model scales. Table 17 reports the results. Under fixed N =4, the signal z̄pos decreases from 10.12 to 7.83 as K grows from 10 to 100, consistent with Theorem 2: more non-target clients increase the masking noise in the estimator. However, z̄pos remains well above γ=4 even at K=100 (margin = 3.83), and FedAttr maintains 100% TPR and 0% FPR throughout. The benign signal z̄neg increases modestly (0.57 → 1.54), but stays far below γ. Under proportional N =⌊K/3⌋, the privacy–variance trade-off adapts to the cohort size: larger N provides stronger privacy (O(d∗ /N ) leakage) while the variance factor N (K−1−N )/(K−2) stays controlled. At K=100 with N =33, the signal z̄pos =7.92 is comparable to the fixed-N setting (7.83), while the per-round MI leakage is reduced by a factor of 33/4 ≈ 8×. H.2
Overhead
Table 18 breaks down FedAttr’s computational overhead. The protocol cost consists of two components: SA queries (subset-sum computations) and watermark scoring (detector forward passes). SA query scaling. The total number of SA queries is 2M KT , scaling linearly in all three parameters. Table 19 reports the query count and estimated time for different configurations. 36
Table 17: Scalability with increasing K (Llama-3.2-1B, Fictitious Knowledge watermark, FedIT, T =5, M =5, r=⌊0.3K⌋). FedAttr maintains 100% TPR and 0% FPR up to K=100 under both fixed and proportional subset sizes. K
N
r
TPR
FPR
z̄pos / z̄neg
Fixed subset size (N =4) 10 4 3 100% 20 4 6 100% 50 4 15 100% 100 4 30 100%
0% 0% 0% 0%
10.12 / 0.43 9.40 / 1.02 7.86 / 1.32 7.83 / 1.54
Proportional (N =⌊K/3⌋) 10 3 3 100% 20 6 6 100% 50 16 15 100% 100 33 30 100%
0% 0% 0% 0%
10.41 / 0.45 9.56 / 0.56 7.55 / 1.02 7.92 / 1.03
Table 18: Overhead breakdown (Llama-3.2-3B, LoRA rank 64, 4×H200 GPUs). Component
Time
% of training
FL training (5 rounds)
8.5 hr
—
SA queries (500 total) per query FF scoring (55 evals) per eval
5 min 0.6 s 27 min ∼30 s
1.0%
FedAttr total
32 min
6.3%
5.3%
Scoring cost. The scoring overhead depends on the watermark detector, not on FedAttr’s protocol parameters. Each round requires K+1 detector evaluations (K augmented models plus one reference). Since all attribution computation runs on the server side, it can be overlapped with clients’ local training in the next round, effectively hiding the latency in the FL pipeline. Rejection rate. The rejection check (Section 4.1) ensures that the privacy bound (Theorem 4) holds pointwise for every accepted query design. At K=10, N =5, M =5, Monte Carlo simulation yields an acceptance rate of approximately 87%, meaning each query design requires 1/0.87 ≈ 1.15 sampling attempts on average. Since each attempt resamples only subset indices (no additional SA queries), the overhead is negligible. At K=50, the acceptance rate exceeds 99.8%. Without the rejection check, the privacy bound still holds in expectation over the query design, matching the guarantee of Elkordy et al. [2023]. H.3
Partial Participation
In practice, not all clients may be available every round. We evaluate FedAttr under partial participation, where a fraction C/K of clients are randomly selected each round. For client i participating in rounds Ti ⊆ [T ], the Stouffer statistic uses only those rounds: 1 X (t) Zi = p zi . |Ti | t∈Ti
(28)
Theorem 3 applies with T replaced by |Ti |: fewer rounds weaken the signal but the error bound retains the same exponential form. We experiment with K=20, r=6, N =5, M =5, T =10, and participation rates C/K ∈ {0.5, 0.7, 1.0}. Table 20 reports the results. FedAttr maintains 100% TPR and 0% FPR at all three rates. p The signal z̄pos decreases from 14.5 (full participation) to 10.8 (C/K=0.5), consistent with the |Ti | scaling: p at C/K=0.5, each client participates in ≈ 5 rounds, and 14.5× 5/10 ≈ 10.3, close to the observed 10.8. The benign signal z̄neg increases modestly (0.67 → 1.34), reflecting the reduced averaging effect, but remains far below γ=4. 37
Table 19: SA query count and estimated time (M =5, T =5, 0.6s per query). K
Queries (2M KT )
Time
% of training
10 20 50
500 1000 2500
5 min 10 min 25 min
1.0% 2.0% 4.9%
These results confirm that FedAttr naturally accommodates partial participation: the Stouffer aggregation adapts to each client’s participation history, and attribution remains reliable as long as each client accumulates sufficient rounds. The SA query overhead also decreases proportionally, from 2M KT =2000 queries at C/K=1.0 to 2M CT =1000 at C/K=0.5. Table 20: Partial participation (K=20, r=6, N =5, M =5, T =10, Llama-3.2-1B, Fictitious Knowledge watermark). C/K: fraction of clients participating per round; E[|Ti |]: expected number of rounds per client. C/K
E[|Ti |]
TPR (%)
FPR (%)
z̄pos / z̄neg
1.0 0.7 0.5
10 7 5
100.0 100.0 100.0
0.0 0.0 0.0
14.5 / 0.67 12.8 / 1.23 10.8 / 1.34
Discussion. At C/K=0.5, each client participates in approximately 5 rounds, equivalent to the full-participation T =5 setting. The signal strength should therefore be comparable. At C/K=0.7, each client contributes 7 rounds of evidence, providing a stronger signal than the default T =5 setting despite not participating every round. This demonstrates that FedAttr naturally accommodates partial participation: the Stouffer aggregation automatically adapts to each client’s participation history, and the theoretical guarantees extend with |Ti | replacing T . The SA query overhead under partial participation is 2M C · T (only participating clients are queried), reducing the total overhead proportionally. Table 21: Recovery under non-IID (α=0.1) with increasing T (K=10, N=5, M=5, r=3, γ=4.0).
H.4
T
TPR (%)↑
FPR (%)↓
z̄pos
z̄neg
5 10 15 20
67.0 100.0 100.0 100.0
0.0 0.0 0.0 0.0
8.57 12.9 15.8 18.4
0.08 0.54 1.02 1.32
Non-IID Recovery via Increasing Communication Rounds
Fig. 4 shows that under severe non-IID heterogeneity (α=0.1),√ FedAttr’s TPR degrades to 67% at T =5. Theorem 3 predicts that the Stouffer statistic grows as T , so increasing communication rounds should recover attribution accuracy. We verify this by sweeping T ∈ {5, 10, 15, 20} at α=0.1, with all other parameters held at defaults. Table 21 confirms the theoretical prediction. Doubling the rounds from T =5 to T =10 restores 100% TPR and 0% FPR, with z̄posp increasing from 8.57 to 12.9. The observed growth ratio 12.9/8.57 ≈ 1.51 √ is close to the theoretical 10/5 ≈ 1.41, consistent with the T scaling predicted by Theorem 3. Further increasing T to 15 and 20 continues to widen the margin (z̄pos − γ reaches 14.4 at T =20). The benign signal z̄neg increases modestly (0.08 to 1.32) but remains well below γ=4 throughout, confirming that additional rounds selectively amplify the watermark signal without inflating false positives. These results suggest a practical guideline for non-IID deployments: the server can monitor the Stouffer margin across rounds and continue attribution until a target confidence level is reached, rather than fixing T in advance. 38
I
Limitations
Corpus owner involvement. FedAttr requires the corpus owner to participate in the attribution phase by evaluating the scoring function with its private detection key. This introduces an operational dependency: attribution cannot proceed without the corpus owner’s cooperation. In settings where the corpus owner is unavailable or unwilling to participate, a delegated or threshold-based key-sharing mechanism would be needed, which we leave to future work.
J
Broader Impacts
FedAttr is designed to enforce data-use license compliance in federated learning, helping corpus owners identify unauthorized use of their intellectual property without compromising the privacy of honest participants. By operating entirely within the secure aggregation framework, FedAttr preserves the core privacy guarantees that make federated learning attractive for privacy-sensitive domains such as healthcare and finance. On the positive side, FedAttr strengthens the trust ecosystem between data providers and model trainers: corpus owners gain a practical enforcement tool, which in turn may encourage broader data sharing under clear licensing terms and foster more open collaboration in federated settings. A potential concern is that the attribution mechanism could be repurposed beyond its intended licenseenforcement scope—for instance, to monitor or profile individual clients’ training behavior. We note that FedAttr’s design mitigates this risk in two ways: (i) the corpus owner must hold the watermark detection key to produce attribution decisions, so the server alone cannot perform attribution; and (ii) the mutual-information bound (Theorem 4) formally limits what the released estimator reveals about any individual client’s update. Nonetheless, deployment guidelines should clearly specify the permissible scope of attribution queries to prevent misuse.
39