AEGIS: Attention-Embedding Gradient Isolation Shield - Triple-Channel Gradient Masking for Privacy-Preserving Federated LLM Fine-Tuning

AEGIS: Attention-Embedding Gradient Isolation Shield — Triple-Channel Gradient Masking for Privacy-Preserving Federated LLM Fine-Tuning Ye Tao∗ , Hong Shen∗ , Hui Tian† , Xin Wang∗ , and Can Wang† ∗ Central Queensland University

arXiv:2608.19534v1 [cs.CR] 20 Aug 2026

{y.tao,h.shen,x.wang3}@cqu.edu.au † Griffith University {hui.tian,c.wang}@griffith.edu.au

Abstract—Gradient inversion attacks recover private training text from gradients shared in federated learning, posing a serious threat to collaborative model training. Through our analysis of transformer gradient structure, we identify three channels through which private token information leaks: the attention output projection gradient exposes a low-rank subspace that encodes input embeddings (Channel 1), the embedding gradient’s row-norm sparsity directly reveals which tokens are present (Channel 2), and the MLP expansion gradient carries a recoverable subspace signal analogous to Channel 1 (Channel 3). Stateof-the-art attacks exploit these channels analytically to achieve near-exact token recovery in seconds. Existing defences address at most one channel and either degrade model utility or leave the remaining structural signals intact. We introduce AEGIS (Attention-Embedding Gradient Isolation Shield), a lightweight defence that closes all three analytical channels with three backward-path operations requiring no architectural changes: freezing attention projection parameters eliminates Channel 1 by construction, calibrated noise injection into the embedding gradient destroys Channel 2’s token-presence signal, and analogous per-block noise injection into the MLP expansion gradient masks Channel 3. The same masked gradient drives both the local optimiser step and the server export, so no clean signal is retained on either side. Evaluated across 11 models and six datasets, AEGIS reduces token recovery rates to near zero against a range of gradient inversion attacks, both analytical and optimisation-based, while preserving or improving model utility. We provide formal guarantees for Channels 1 and 2 and validate the full defence empirically against adaptive adversaries with complete knowledge of the mechanism. Index Terms—federated learning, gradient inversion, privacy, language models, gradient masking

I. I NTRODUCTION Federated learning (FL) [1], [2] enables multiple participants to collaboratively fine-tune large language models (LLMs) without centralising raw training data. Participants compute gradients on their private corpora and transmit them to a coordinating server, which aggregates the updates and returns an improved global model. Because raw text never leaves the client device, FL is widely assumed to preserve participant privacy. Recent work on gradient inversion attacks (GIAs) [3]–[9] challenges this assumption: shared gradients encode enough

information to reconstruct training text at the token level. Early optimisation-based attacks [3], [6], [10] iteratively minimise the distance between dummy and true gradients, but they are sensitive to initialisation and do not scale to large vocabularies. More recent analytical attacks bypass optimisation entirely: DAGER [5] achieves near-exact token-set recovery in closed form by exploiting the algebraic structure of transformer gradients, obtaining ROUGE-1 above 0.65 on standard benchmarks; SOMP [8] and F ED S PY-LLM [9] extend the same paradigm with head-structured pooling and rank-deficient embeddinggradient decomposition respectively. DAGER succeeds because transformer language models expose several independent gradient channels, each sufficient for partial recovery and jointly sufficient for near-perfect reconstruction: Channel 1 (Attention SVD). The gradient of the attention (ℓ) output projection ∇Wo has a column space spanned by the value vectors of the input tokens. SVD of this gradient identifies the rank-T subspace corresponding to the T input tokens, and projecting all V vocabulary embeddings onto this subspace distinguishes true tokens from the rest. • Channel 2 (Embedding Row Norms). The embedding layer gradient ∇WWTE has non-zero rows only for tokens present in the training input. This structural sparsity provides a binary signal: zero gradient norm = token absent, non-zero norm = token present. The signal is immune to downstream architectural modifications. • Channel 3 (MLP Expansion SVD, adaptive). The first (ℓ) MLP projection ∇Wfc has the same low-rank structure as Channel 1; an adaptive adversary that knows Channels 1–2 are masked can rerun DAGER’s SVD pipeline on the MLP gradient to recover the same token set. •

Differential privacy (DP-SGD) [11] adds calibrated noise to clipped gradients, but prior work on DP fine-tuning of language models shows that the privacy budgets required to meaningfully resist reconstruction attacks incur substantial utility cost [12]. As we show in Section VI-F, even large noise magnitudes (σ=0.1) fail to suppress DAGER’s analytical recovery channel. S OTERIA [13] prunes gradient components

but targets convolutional vision models. Data-level defences obfuscate training tokens (e.g. via embedding-proximate substitutions) before optimisation, which can suppress exact token recovery but typically incurs high preprocessing cost and does not remove the MLP gradient channel available to adaptive adversaries. Our analysis of DAGER’s channel structure (Section IV) indicates that single-channel defences, whether targeted at the attention subspace, the MLP expansion gradient, or the embedding row norms in isolation, cannot succeed: closing one channel while leaving another intact provides no net protection against an analytical adversary, because the surviving channel compensates. We propose AEGIS (Attention-Embedding Gradient Isolation Shield), a triple-channel gradient masking mechanism that eliminates all three analytical channels simultaneously. The design is motivated by the observation that any defence must address every exploitable channel at once; closing some channels while leaving one intact provides no net protection against an analytical adversary that reads the surviving channel. AEGIS combines three operations: (1) freezing all attention projection parameters, so their gradients are identically zero (Channel 1); (2) replacing the embedding gradient with a dense, calibrated-noise version whose row norms no longer discriminate present from absent tokens (Channel 2); and (3) applying an analogous per-block flood (ℓ) to the MLP expansion gradient ∇Wfc , calibrated so that the MLP-SVD signal singular values fall beneath the randommatrix noise bulk in the regimes we evaluate (Channel 3). The defence is applied after backpropagation and before the optimiser step: the local optimiser uses the modified gradient, whose flooded blocks retain a scaled learning signal, and the server receives the same masked gradient. Our contributions are: Triple-channel analysis. A systematic analysis of the channel structure DAGER exploits, showing that it relies on two redundant default channels plus one additional channel available to an adaptive adversary, and that any one- or two-channel defence is insufficient because the surviving channel compensates (Section IV). • The AEGIS method. A lightweight, architecture-agnostic gradient masking mechanism that closes all three channels using three backward-path operations (attention-parameter freeze, calibrated embedding flood, calibrated MLP-fc flood), with no architectural changes and negligible overhead (Section IV). Attention freezing repurposes an observation from parameter-efficient fine-tuning [14], [15]; we show that it also eliminates DAGER Channel 1 exactly, and we combine it with the embedding and MLP-expansion flooding under a single calibrated Gaussian-uniformisation schema. • Formal guarantees. A proof that attention freezing zeros the Channel 1 exported gradient (Theorem 1) and that embedding gradient uniformisation eliminates Channel 2’s deterministic support indicator while bounding the residual active-row second-moment inflation (Theorem 2). •

Evaluation. We evaluate AEGIS on 11 LLMs (see Table III) across six benchmark datasets. Attack ROUGE-1 drops from ≥0.87 (undefended) to ≤0.005 under AEGIS, a relative reduction of at least 99% against the attacks we consider. Test perplexity remains at or below the undefended baseline on every model. Section II surveys related work. Section III formalises the federated fine-tuning threat model and DAGER’s attack mechanism. Section IV presents the analysis that motivates the triple-channel design. Section IV describes the AEGIS algorithm. Section V provides theoretical guarantees. Section VI reports empirical results. Section VII discusses limitations and future directions. Section VIII concludes. Full proofs are deferred to Appendix A. •

II. R ELATED W ORK This section positions AEGIS within the landscape of gradient inversion attacks and defences for federated learning. A. Gradient Inversion Attacks on Language Models [3] first demonstrated that a shared gradient for a single training example is sufficient to reconstruct the input via iterative optimisation in the continuous embedding space, then mapping recovered embeddings back to discrete tokens. [16] improved the method by analytically extracting the label from the gradient sign. [10] introduced L1 regularisation for improved token recovery on BERT-class models. [6] adapted gradient inversion to language models by coupling cosinesimilarity gradient matching with a language-model prior, alternating between gradient descent in embedding space and beam search in token space to reconstruct coherent sentences. These optimisation-based approaches achieve moderate recovery rates but are sensitive to initialisation and struggle with token ordering in long sequences. A qualitatively different class of attacks bypasses iterative optimisation entirely. [5] introduced DAGER, a closed-form gradient inversion algorithm for transformer language models that exploits two structural properties of transformer gradients: (1) the low-rank structure of the attention output projection (ℓ) gradient ∇Wo , whose column space is spanned by the value vectors of the input tokens, and (2) the sparsity of the embedding gradient ∇WWTE , which has non-zero rows only for tokens present in the input. DAGER achieves nearexact token-set recovery (ROUGE-1 > 0.65) on GPT-2 and BERT without any gradient descent, completing in seconds. [7] proved that a related analytical objective can be solved in linear time under mild assumptions. SOMP [8] extends DAGER’s subspace test to a head-structured pooling stage followed by an Orthogonal Matching Pursuit reconstruction over an LM-guided beam, scaling token recovery to larger batch sizes. F ED S PY-LLM [9] attacks the same structural channels via rank-deficient gradient decomposition: it reads tokens from the active rows of the embedding gradient and resolves their order through partial-gradient alignment, generalising the analytical recipe across encoder, decoder, and PEFT settings. Earlier analytical work includes APRIL [17],

which exploits attention and embedding structure in vision transformers, and SPEAR [18], which achieves batch-exact inversion of fully-connected layers—a result that DAGER extends to transformer LMs. [19] derived a recursive analytical formula (R-GAP) for exact gradient inversion in fullyconnected networks, establishing the algebraic foundation that later analytical attacks on transformers build upon. Together, these analytical attacks—DAGER, SOMP, and F ED S PYLLM on the LM side, and APRIL/SPEAR/R-GAP on the vision and FC side—establish that gradient inversion is not merely a theoretical curiosity but a practical, computationally cheap threat that demands a structural—not merely noisebased—defence. [20] and [21] showed that a malicious server can modify model weights to amplify gradient leakage, enabling batchlevel extraction. These attacks operate under a stronger threat model than our honest-but-curious assumption and are orthogonal to the defences considered here. B. Defences Against Gradient Inversion DP-SGD [11], [22] clips gradients to a bounded norm and adds calibrated Gaussian noise, offering a formal (ε, δ)-DP guarantee. However, prior work on DP fine-tuning of language models shows that the privacy budgets required to meaningfully suppress analytical gradient-inversion attacks incur substantial utility cost [12], and as we show in Section VI-F, even large noise magnitudes fail to suppress the analytical recovery channel. Gradient pruning [3] randomly zeroes a fraction of gradient components; advanced attacks such as GRAB [7] and DAGER [5] remain effective against pruned gradients. Gradient compression methods [23] were designed to reduce communication cost, not to provide privacy. [5] demonstrated that compressed gradients remain fully invertible by analytical attacks. S OTERIA [13] prunes gradient components that encode representation-sensitive information, targeting convolutional vision models. I NSTA H IDE [24] mixes training examples at the data level, but its security was subsequently broken [25]. Differentially Private Word Embeddings (DPWE) [26] add Laplace noise in the embedding space during inference, not during gradient computation. A complementary line of work perturbs or substitutes training tokens before fine-tuning (e.g. embedding-proximate replacements chosen by search in token or hidden space). Such methods can reduce exact recovery under analytical attacks but typically require per-token preprocessing, alter the training distribution, and leave other gradient channels (notably MLP paths) available to adaptive adversaries. They also face an inherent semantic leakage trade-off: if the obfuscated text preserves sufficient semantic fidelity for the language model to learn the target task, high-level semantic structure may still be reflected in the gradients. By contrast, AEGIS defends at the algebraic gradient layer, misaligning the subspaces exploited by closed-form inversion without modifying client text.

SecAgg [27] cryptographically hides individual gradients from the server; the original protocol has O(n2 ) per-round communication cost, though more recent graph-based variants achieve near-linear per-client overhead [28]. SecAgg does not protect against a malicious server or colluding participants. C. Parameter-Efficient Fine-Tuning LoRA [14] and adapter tuning [15] freeze most model parameters and train low-rank additive updates, demonstrating that pretrained transformers can be effectively adapted with fewer than 1% of parameters trainable. AEGIS leverages the same insight: freezing the attention parameters does not meaningfully degrade fine-tuning quality, because pretrained attention patterns encode robust contextual representations that transfer well to downstream tasks. The difference is that AEGIS freezes attention for privacy, not for parameter efficiency. III. P RELIMINARIES This section introduces the federated fine-tuning setting, the transformer gradient structure common to decoder and encoder LMs, and the two analytical channels exploited by closedform gradient inversion attacks (a third, adaptive channel is introduced in Section IV). A. Federated Language Model Fine-Tuning Let V denote a vocabulary of size V and let fθ : V ∗ → R be a causal language model parameterised by θ. In federated fine-tuning [1], K clients each hold a private text corpus Dk = {s1 , . . . , snk }. At each round, client k computes the gradient gk = ∇θ L(fθ , s) for a mini-batch sample s ∼ Dk and sends gk to a central server. The server aggregates the received gradients (e.g. via FedAvg [1]) and distributes updated weights. a) Threat model.: We consider the FedSGD setting where each client shares the gradient computed on a single mini-batch (one local step) after each global round, following the standard threat model adopted by DAGER [5], SOMP [8], F ED S PY-LLM [9], and LAMP [6]. An honest-but-curious, non-adaptive adversary controls the server or eavesdrops on the communication channel, receiving gk in cleartext. Specifically: • The adversary has full knowledge of the model architecture and parameters θ. • The adversary does not know which defence mechanism (if any) the client is using, i.e. the defence mechanism itself is not part of the adversary’s prior. • The adversary mounts an analytical gradient inversion attack, exploiting the algebraic structure of transformer gradients to recover input tokens without iterative optimisation. • Only a single client is evaluated; multi-client privacy amplification is orthogonal. This single-client, single-step evaluation setting is standard in the GIA literature [3], [5]: success against one client implies vulnerability for all.

B. Transformer Gradient Structure We describe the gradient structure using GPT-2 [29] as a concrete reference, as its architecture is representative of the decoder-only family (GPT-2, LLaMA, Gemma). The analytical channels described below arise from properties of the transformer block that are shared by encoder architectures (e.g. BERT [30]) as well; we note encoder-specific differences where they arise. A GPT-2-class model consists of L transformer blocks. Each block ℓ contains: • Attention projections: query-key-value projection (ℓ) (ℓ) WQKV ∈ Rd×3d and output projection Wo ∈ Rd×d . (ℓ) d×4d • MLP feed-forward: expansion Wfc ∈ R and con(ℓ) 4d×d traction Wproj ∈ R . • Layer normalisation: two LayerNorm modules per block. The embedding matrix WWTE ∈ RV ×d maps token indices to hidden representations of dimension d. The LM head (linear layer producing logits over V) typically shares weights with WWTE (weight tying). a) Encoder LMs.: BERT-class models [30] share the same block structure. The key difference is the absence of a causal attention mask, which removes the decoder’s greedy-extension property and requires exhaustive search over candidate sequences. Channel 1 (attention-output SVD) and Channel 2 (embedding row-norm sparsity) arise identically in encoders, so AEGIS’s attention-freeze and embeddinguniformisation operations apply without modification; the MLP-fc uniformisation (Channel 3) likewise transfers directly. Our evaluation in Section VI confirms that AEGIS reduces DAGER ROUGE-1 to ≤ 0.006 on BERT-base and BERTlarge, matching the decoder-side results. C. Analytical Attacks: Two Gradient Channels Analytical gradient inversion attacks [5], [7]–[9] recover the input token set {t1 , . . . , tT } from a single gradient g in closed form, without iterative optimisation, by exploiting two structurally independent channels in the transformer gradient. DAGER and SOMP primarily target Channel 1 (attention SVD scoring) with different downstream pipelines, while F ED S PY-LLM primarily targets Channel 2 (embedding rownorm sparsity) with an explicit ordering stage; all three rely on the same two structural properties. Definition 1 (Channel 1: Attention Subspace Scoring). For each block ℓ, let U(ℓ) be an orthonormal basis for the range of (ℓ) ∇Wo , equivalently the left singular vectors whose singular (ℓ) values are non-zero. If ∇Wo = 0, this basis has no columns and the projection below is the zero projection. The subspace residual score of vocabulary token v is: ⊤

rv(ℓ) =

∥ev − U(ℓ) U(ℓ) ev ∥ , ∥ev ∥

where ev ∈ Rd is the embedding vector for token v. Tokens with small residual (i.e. whose embedding lies in the attention output gradient subspace) are candidates for the input set.

Definition 2 (Channel 2: Embedding Row-Norm Scoring). Consider a single input sequence s = (t1 , . . . , tT ) (batch size B = 1). Let S(s) = {v ∈ V : v appears in s} be the active vocabulary set. The embedding layer gradient ∇WWTE ∈ RV ×d has the structural property that row v is non-zero if and only if v ∈ S(s). The row-norm score is: nv = ∥∇WWTE [v, :]∥2 . The top-|S(s)| tokens by row-norm score form the recovered token set when the attacker knows the number of distinct active rows. This signal is a binary indicator: nv > 0 ⇐⇒ v ∈ S(s). Remark 1 (Practical scope of Definition 2). Definition 2 assumes an unmasked, non-cancelling input-lookup gradient. Loss structure, masking, special-token handling, or finite precision can instead make an active row zero. Algorithm 1 uses a relative-norm rule and still floods any excluded row; because all V rows become dense, an occasional zero active row does not restore a clean absent/present signal. Remark 2 (Batch size B > 1). For a mini-batch of B sequences, ∇WWTE [v, :] is non-zero iff token v appears in any ofSthe B sequences. Channel 2 therefore recovers the B union b=1 S(sb ) rather than per-sequence tokens; analytical attacks address this via Channel 1 attention-subspace filtering combined with positional disambiguation. Our experiments use B = 8, and the uniformisation defence remains valid in this regime: the operation renders all V row norms comparable regardless of how many sequences contributed non-zero rows. The formal guarantee (Theorem 2) is stated for B = 1 for notational clarity, and extends verbatim to B > 1 by replacing the active-row set S(s) with the union set. a) Combined recovery.: Analytical attacks combine both channels: attention subspace scores from all L blocks are aggregated and intersected with the embedding row-norm ranking to form the final recovered token set. The redundancy between channels is critical: either channel alone provides partial recovery; together they achieve near-exact results. b) Notation summary.: Table I collects the main notation used throughout the paper. IV. T HE AEGIS M ETHOD This section presents the AEGIS algorithm, its design rationale, and its implementation. A. Design Objectives AEGIS targets three properties for the defended gradient D(∇θ L): (P1) privacy: no token identity information is exposed via the three analytically-exploitable channels (attention output-projection subspace, embedding row-norm sparsity, MLP expansion subspace); (P2) utility: the model converges with test perplexity comparable to the undefended baseline; and (P3) efficiency: at most O(V d + Ld2 ) overhead per backward pass, which is negligible relative to the backward computation itself.

Channel 1: Freeze Attention Projections (One-time & Per-step)

1

Attention Target (Multi-Head Self-Attention) ��

��

One-Time Setup (Once) Freeze Attention Parameters

�

Token Embedding �� (� × �) (shared with LM head)

absent rows: zero gradient

MLP Target (MLP/FFN)

3

active rows: signal+noise formerly absent rows: noise (non-zero)

Combine all rows non-zero

2

2

��� = � ∙ ��� + �(0, � � ) Originally absent rows become non-zero after flooding

��� (up-projection)

����� (down-projection)

Channel 3: Flood MLP Gradient (Per-step)

+residual

LayerNorm Flood Traget: up-projection gradient (���)

Embedding Target (Token Embedding �� ) (shared with LM head)

MLP/FFN (from transformer block) ��� (���/up)

��� Gradient (sparse)

���� = ��� ∙ ���� + �

��� Gradient (dense) Flood Noise

����� (�����/down)

Active (≠ 0)

Forward uses module normally; modification applied on backward pass only.

LM Head/ Output

Input Token Sequence

Flooded Embedding Gradient (after flood)

Add Noise �(0, �2 �2 )

2

active rows: signal+noise

LayerNorm

2

Detect Active Rows retain signal

1

Before Flood: (embedding grad)

��

��

+residual

3

Channel 2: Flood Embedding Gradient (Per-step)

• •

Zero (= 0)

Non-zero (after flood)

Flood the MLP up-projection gradient Backward modification only

Fig. 1. Overview of the AEGIS mechanism. Channel 1 (the attention bottleneck) is neutrallised via freezing, while Channel 2 (embedding rows) and Channel 3 (the first MLP projection Wfc ) are obscured through mathematically calibrated dense Gaussian uniformisation. Downstream utility is maintained via the remaining unaffected trainable parameters (e.g., Wproj , LayerNorm).

(cf. LoRA [14]); the load-bearing MLP expansion uses uniformisation rather than freezing.

TABLE I N OTATION SUMMARY.

Symbol

Description

V, V

Vocabulary, its size (|V| = V ) L Number of transformer blocks

WWTE (ℓ)

Wfc

(ℓ)

rv

λ

Embedding matrix ∈ RV ×d MLP fully-connected weight in block ℓ Subspace residual score (Channel 1) Flood scale hyperparameter

Symbol d

Description Hidden dimension

T

Sequence length (number of input tokens) (ℓ) Attention output proWo jection in block ℓ ev Embedding vector for token v nv Embedding rownorm score (Channel 2) ρ Real-token retain ratio

B. Channel 1: Freeze Attention Projections The attention subspace channel (Definition 1) exploits the (ℓ) low-rank column space of ∇Wo . The simplest way to eliminate this signal is to ensure the gradient is identically zero: ∀ℓ ∈ [L], p ∈ params(attnℓ ) :

p.requires grad ← F alse. (1) (ℓ) (ℓ) Freezing all attention parameters (WQKV , Wo ) across all L blocks ensures that: (ℓ) • ∇Wo = 0 for all ℓ, so the SVD column space is empty. (ℓ) • The subspace residual score rv = 1 for all tokens v, providing zero discriminative power. C. Channel 2: Embedding Gradient Uniformisation

Three backward-path operations address these goals: (i) (ℓ) (ℓ) freeze attention (Channel 1) — set ∇WQKV = ∇Wo = 0 by construction; (ii) uniformise embedding (Channel 2) — replace ∇WWTE with a dense, calibrated-noise gradient whose active rows are partially retained; (iii) uniformise MLP-fc (Channel 3) — apply an analogous uniformisation (ℓ) to ∇Wfc per block, pushing the SVD signal beneath the Marchenko–Pastur noise bulk. Freezing keeps pretrained attention patterns, which helps preserve utility in our ablation

The embedding row-norm channel (Definition 2) exploits the binary sparsity of ∇WWTE : rows corresponding to absent tokens have exactly zero gradient. To destroy this signal, we replace the sparse gradient with a dense version in which all V rows have comparable norms: ( ˜ WTE [v, :] = ∇W

ρ · ∇WWTE [v, :] + nv

if v ∈ S(s)

nv

otherwise (2)

where ρ ∈ (0, 1] is the real-token retain ratio (fraction of original gradient preserved for real tokens), and nv ∼ N (0, σ 2 Id ) with σ = λ · m̄, where λ is the uniformisation scale and m̄ is the mean gradient norm of the active (real-token) rows: X 1 m̄ = ∥∇WWTE [v, :]∥2 . (3) |S(s)| v∈S(s)

a) Weight tying and the LM head.: All eleven models in our evaluation tie the LM head to WWTE , so the backward pass accumulates a single gradient tensor used for both the embedding update and the output projection. AEGIS uniformises this shared tensor, simultaneously masking both paths. For models with an untied LM head, the LM-head gradient ∇WLM is a separate matrix whose row norms can expose the target-token support. We therefore apply the same row-wise uniformisation in Eq. 2 to ∇WLM whenever the output embedding is not tied to WWTE ; tied heads are already covered because they share the same gradient tensor as the input embedding. D. Channel 3: MLP Expansion Gradient Uniformisation While Channels 1 and 2 are the two channels exploited by non-adaptive DAGER, Section IV-D identified a third (ℓ) analytical channel: the MLP expansion gradient ∇Wfc ∈ (ℓ) d×4d R , whose column span is isomorphic to that of ∇Wo and therefore recoverable by the same SVD span-estimation pipeline. An adaptive adversary aware that Channels 1 and 2 are closed can target Channel 3 directly. (ℓ) Since Wfc is load-bearing for downstream-task utility, freezing it (the Channel 1 strategy) is not viable; we therefore apply a uniformisation defence analogous to Channel 2. For each block ℓ ∈ [L] we replace the exported MLP-fc gradient with ˜ (ℓ) = ρfc · ∇W(ℓ) + N(ℓ) , ∇W (4) fc fc 2 where N(ℓ) ∈ Rd×4d has i.i.d. entries Nij ∼ N (0, σfc ) with (ℓ) (ℓ) σfc = λfc · m̄fc and m̄fc the mean absolute value of the entries (ℓ) of ∇Wfc . The hyperparameters (ρfc , λfc ) play the same role as (ρ, λ) on the embedding: ρfc retains a scaled fraction of the true MLP gradient for the local optimiser step, while λfc sets the noise floor that the exported gradient carries to the server. Unless otherwise noted we use the privacy-hardening endpoint (ρfc , λfc ) = (0, 8.0) for exported MLP gradients in adaptive-threat experiments. The implementation applies the same matrix-gradient uniformisation to all two-dimensional (ℓ) MLP weights, including the projection matrix Wproj , so an adaptive attacker cannot simply retarget from the expansion matrix to the contraction matrix. a) Random-matrix rationale for Eq. 4.: An adversary ˜ (ℓ) sees a matrix whose signal performing rank-T SVD on ∇W fc component is scaled by ρfc and whose noise component is 2 full-rank with variance σfc per entry. The Marchenko–Pastur law predicts that the singular-value √bulk of √ a d × 4d i.i.d. √ Gaussian matrix extends up to σfc ( d + 4d) = 3σfc d, which at λfc = 1, d = 768 is ≈ 83 m̄fc ; the signal singular values are bounded by ρfc · σmax (∇Wfc ). Thus, when the

scaled signal singular values lie below this noise edge, the SVD stage has no stable signal spike to lock onto. This is a design criterion and empirical prediction rather than a theorem in this paper; a formal robust-PCA lower bound is left to future work (Limitation 1). The empirical adaptive-attack results in Section VI confirm this prediction: running adaptive DAGER on ∇Wfc after AEGIS uniformisation yields ROUGE-1 in the same near-zero regime as the non-adaptive attack. E. Full Algorithm AEGIS is applied client-side as a backward-path hook with no changes to the model architecture or training loop. Before training begins, all attention projection parameters are frozen (Eq. 1), so their gradients are identically zero throughout finetuning. After each backward pass, the embedding gradient ˜ WTE ∇WWTE is replaced with the uniformised version ∇W (ℓ) (Eq. 2), and for each block ℓ the MLP-fc gradient ∇Wfc ˜ (ℓ) (Eq. 4). The local optimiser then is replaced with ∇W fc steps on the modified gradients; the server receives only the uniformised export, with attention gradient slots set to zero. Crucially, the intervention is applied after backpropagation and before the optimiser step, so the learning signal for the (ℓ) retained channels (ρ · ∇WWTE and ρfc · ∇Wfc ) propagates to the local weights while the exported gradient exposes only calibrated Gaussian noise to the server. Algorithm 1 summarises the complete client-side procedure. V. T HEORETICAL A NALYSIS This section states the exact parts of the AEGIS security argument: Channel 1 has zero exported gradient under attention freezing, Channel 2 loses its deterministic support signal under embedding flooding, and Channel 3 receives an exact MLP-fc flooding decomposition with a Frobeniusmoment bound. Claims that depend on high-dimensional order statistics or random-matrix denoising are treated as design rationale and empirical predictions, not as theorem statements. The corresponding proofs are collected in Appendix A. We additionally provide Lean 4 [31] formalizations of the theorem and proposition cores in the supplementary material. A. Channel 1: Attention Freezing Guarantee Theorem 1 (Channel 1 Elimination — Exact). Let fθ be a b denote GPT-2-class model with L transformer blocks. Let ∇ the gradient tensor exported by the automatic differentiation engine after applying the frozen-parameter rule, equivalently the zero extension of the gradient on the reduced trainable parameter space. Assume every vocabulary embedding is non(ℓ) (ℓ) zero. If all attention parameters {WQKV , Wo }L ℓ=1 are frozen (i.e. requires_grad = False), then for any training input s = (t1 , . . . , tT ) and any differentiable loss function L: b (ℓ) L = 0 ∈ Rd×d , ∀ℓ ∈ [L]. ∇ (5) W o

Consequently, under the non-zero-singular-subspace convention in Definition 1, the subspace residual score satisfies: rv(ℓ) = 1,

∀v ∈ V, ∀ℓ ∈ [L].

(6)

Algorithm 1: AEGIS: Triple-Channel Gradient Masking Input: Model fθ with L blocks; hyperparameters ρ, λ, ρfc , λfc ; training set D Output: Privacy-preserving fine-tuned model; masked ˜ for server gradient ∇θ for ℓ ← 1 to L do (ℓ) 2 WQKV .requires_grad ← False (ℓ) 3 Wo .requires_grad ← False

B. Channel 2: Embedding Flooding Guarantee Assumption 1 (Bounded Active-Row Norm Deviation). For a sequence s, let S(s) = {v ∈ V : v appears in s} and assume |S(s)| > 0. There exists a constant C ≥ 1 such that for all active rows v ∈ S(s): ∥∇WWTE [v, :]∥2 ≤ C · m̄,

1

for each mini-batch B ⊆ D do Compute L(θ; B) via forward pass 6 Compute ∇θ L via backward pass

4

5

7

// Channel 2: embedding gradient uniformisation  S ← v ∈ V : ∥∇WWTE [v, :]∥2 > 0.01 · maxu ∥∇WWTE [u, :]∥2 1 X m̄ ← ∥∇WWTE [v, :]∥2 |S| v∈S

8 9 10 11 12 13 14

15

σ ← λ · m̄ for v ← 1 to |V| do Sample nv ∼ N (0, σ 2 Id ) if v ∈ S then ˜ WTE [v, :] ← ρ · ∇WWTE [v, :] + nv ∇W else ˜ WTE [v, :] ← nv ∇W // Channel 3: MLP-fc gradient uniformisation (per block) for ℓ ← 1 to L do  (ℓ)

(ℓ)

16

m̄fc ← mean |∇Wfc |

17

σfc ← λfc · m̄fc
2 d×4d Sample N(ℓ) ∼ N 0, σfc ˜ (ℓ) ← ρfc · ∇W(ℓ) + N(ℓ) ∇W fc fc

18 19 20

21

(ℓ)

˜ θL Perform local optimiser step using ∇ (post-mask gradient) ˜ θ L to server Export ∇

where m̄ is the mean active-row norm (Eq. 3). Remark 4. Assumption 1 holds with C = 1 when all activerow norms equal the mean; the bound degrades through the 2 2 2 ratio √ ρ C /(dλ ), which reaches order one only near C ≈ λ d/ρ ≈ 92 at ρ=0.3, λ=1, d=768. The R-1 ≤ 0.005 values in Tables III–IV provide indirect evidence that all evaluated models lie well within the small-inflation regime; a direct measurement of C (logging max ∥gv ∥/m̄ during training) is a natural follow-up experiment. Theorem 2 (Channel 2 Sparsity Elimination and Norm-Moment Bound). Let ∇WWTE ∈ RV ×d be the embedding gradient for a sequence s with nonempty active set S(s), and suppose ∥∇WWTE [v, :]∥2 > 0 iff v ∈ S(s). Let λ > 0, ρ ≥ 0, m̄ be defined by Eq. 3, and let the flood vectors be independent Gaussians nv ∼ N (0, σ 2 Id ) with σ = λm̄. After AEGIS flooding (Eq. 2): ˜ WTE [v, :]∥2 > 0 for all v ∈ V 1) All rows non-zero: ∥∇W with probability 1. 2) Exact second moments: For non-token rows (v ∈ / S(s)): ˜ WTE [v, :]∥22 ] = d · λ2 · m̄2 . E[∥∇W For active rows (v ∈ S(s)): ˜ WTE [v, :]∥2 ] = ρ2 ∥∇WWTE [v, :]∥2 + dλ2 m̄2 . E[∥∇W 2 2 3) Bounded relative inflation: Under Assumption 1, every active row satisfies ˜ WTE [v, :]∥2 ] − dλ2 m̄2 E[∥∇W ρ2 C 2 2 ≤ . (7) 2 2 dλ m̄ dλ2 Thus the exact zero-vs-nonzero Channel 2 indicator is eliminated; any remaining row-norm signal is only the bounded moment shift in Eq. 7. 0≤

C. Channel 3: MLP-fc Flooding Guarantee

The Channel 1 scoring provides zero discriminative power: no token can be distinguished from any other. Remark 3 (Forward pass is unaffected). Freezing attention parameters sets their gradients to zero but does not remove them from the computational graph. The forward pass computes Attn(Q, K, V) using the pretrained weights normally; only the backward pass skips gradient accumulation for these parameters. The attention mechanism continues to provide contextual representations to downstream MLP layers, maintaining representational capacity.

Theorem 3 (Channel 3 MLP-fc Flooding Moment Bound). (ℓ) For a transformer block ℓ, let G(ℓ) = ∇Wfc ∈ Rd×4d be the (ℓ) MLP expansion gradient and let m̄fc > 0 be the calibration scale in Eq. 4. Let λfc > 0, ρfc ∈ [0, 1], and let N(ℓ) have (ℓ) (ℓ) 2 independent entries Nij ∼ N (0, σfc ) with σfc = λfc m̄fc . After MLP-fc flooding, e (ℓ) = ρfc G(ℓ) + N(ℓ) . G Conditional on the batch: 1) Biased-gradient decomposition: with B(ℓ) = (ρfc − 1)G(ℓ) , e (ℓ) = G(ℓ) + B(ℓ) + N(ℓ) , G

e (ℓ) − G(ℓ) ] = B(ℓ) , E[G

and ∥B(ℓ) ∥F = (1 − ρfc )∥G(ℓ) ∥F . 2) Exact Frobenius second moment: i h
e (ℓ) ∥2 = ρ2 ∥G(ℓ) ∥2 + 4d2 λ2 m̄(ℓ) 2 . E ∥G fc F fc F fc (ℓ)

3) Bounded relative inflation: if ∥G(ℓ) ∥F ≤ Cfc m̄fc for some Cfc ≥ 0, then (ℓ)

0≤

e (ℓ) ∥2 ] − 4d2 λ2 (m̄ )2 E[∥G F fc fc (ℓ) 4d2 λ2fc (m̄fc )2

≤

2 ρ2fc Cfc . 2 4d λ2fc

(8)

VI. E MPIRICAL E VALUATION We scale the evaluation of AEGIS to eleven language models across four architectural families (three decoder-only groups plus the BERT encoder pair), six benchmark datasets, and an orders-of-magnitude range of parameter counts (110M – 13B), using the official DAGER implementation [5]. A. Experimental Setup a) Models.: We evaluate eleven models in total: nine decoder-only causal LMs spanning three families, and two masked encoders from a fourth family, the BERT family (BERT-base and BERT-large), expanding the grid beyond nine causal-decoder checkpoints alone. GPT-2 family [29]: GPT-2 (124M), GPT-2-medium (345M), GPT-2-large (774M), GPT-2-XL (1.5B); • LLaMA family [32], [33]: LLaMA-2-7B, LLaMA-213B† , LLaMA-3.1-8B; • Gemma family [34]: Gemma-2-2B, Gemma-2-9B; • BERT family [30]: BERT-base (110M) and BERT-large (340M), attacked via DAGER’s encoder span-estimation path [5]. •

b) Datasets.: Each model is fine-tuned on six benchmarks covering diverse domains: Rotten Tomatoes [35] (film sentiment), Emotion [36] (multi-class affect), Financial PhraseBank [37] (financial sentiment), WikiText-2 [38] (general language modelling), DialogSum [39] (dialogue summarisation), and CNN/DailyMail [40] (news summarisation). Each dataset is used with up to 100,000 randomly sampled training examples per run. c) Attack configuration.: We use the official DAGER codebase (Petrov et al., NeurIPS 2024 [5]), which performs SVD span-estimation (L1) followed by beam-search decoding (L2). Under AEGIS, attention projections are frozen so that DAGER’s default Channel 1 SVD path sees ∇Wo = 0 and collapses to embedding-gradient recovery on the uniformised ∇WWTE (Channel 2), which is uniformised as predicted by Theorem 2. To evaluate the adaptive MLP-SVD adversary (Channel 3), we additionally run a modified DAGER pipeline ˜ (ℓ) (the in which the span-estimation stage is run on ∇W fc uniformised MLP-fc gradient produced by Eq. 4) instead of on (ℓ) ∇Wo ; this is the strongest adaptive attack the three-channel analysis suggests.

d) Training.: All models are fine-tuned with the AdamW optimizer [41] for up to 8 epochs or 12,000 gradient steps (whichever is reached first), (lr=5 × 10−5 , batch size 8, linear warmup for the first 10% of steps, early stopping on validation loss with patience 5, evaluation every 200 steps), on one NVIDIA H100 GPU; GPT-2/XL use FP16 autocast with FP32 weights, while LLaMA-2-7B uses BF16 weights. A single fine-tuning seed is used per (model, dataset) cell (variance reporting is flagged in Limitation 1). Under AEGIS, attention projection parameters are frozen before the optimizer is instantiated; gradient uniformisation is injected after every backward pass, before the optimizer step. e) Hyperparameters.: AEGIS has four hyperparameters organised as two (λ, ρ) pairs, one per uniformised channel. For embedding uniformisation: (λ, ρ) = (1.0, 0.3). For adaptivethreat MLP uniformisation: (λfc , ρfc ) = (8.0, 0), the privacyhardening endpoint of Eq. 4. Higher λ strengthens privacy at the cost of more noise in the exported gradient; ρ controls the learning signal retained for local updates. These values were chosen from a coarse pilot sweep; principled calibration is left to future work. f) Metrics.: • Attack reconstruction (↓ better privacy): overlap between DAGER’s recovered text and the ground-truth input, reported as ROUGE-1 / ROUGE-2 / ROUGE-L (R-1, R-2, R-L) [42] and METEOR [43]. Values near one indicate near-exact recovery; near zero indicates failure. Main tables (Tables III–IV) give the full R-1/R-2/R-L/M grid; the narrative highlights R-1 following prior DAGER reporting [5]. • Language-modelling utility (↓): test perplexity (PPL) on held-out text for causal LMs (and analogous loss-driven PPL-style scores where applicable). • Classification utility (↑): accuracy and F1 on labelled benchmarks (Rotten Tomatoes, Emotion, Financial PhraseBank), reported alongside loss/PPL in Appendix E. B. Main Results and Defence Effectiveness Across Model Scale Tables III and IV report the full per-dataset attack similarity grid for decoder-only and encoder LMs respectively (†: LLaMA-2-13B averaged over 3/6 completed datasets). Metrics. DAGER R-1 lies in [0, 1]; lower values imply weaker token recovery. Test perplexity is strictly positive with smaller values indicating better held-out language-modelling utility. How the components preserve utility. Freezing keeps the model’s pretrained attention patterns, while flooding keeps part of the true embedding and MLP gradients so those layers can still learn. Table II shows that the single-component variants remain close to the undefended utility. With all components, GPT-2 PPL changes from 26.0 to 25.6 with unchanged accuracy and F1; GPT-2-XL PPL improves from 25.7 to 22.1, with small gains in accuracy and F1. Thus, in these tests, stronger privacy does not require a utility loss, although this may depend on the model and task. Results. Aggregating six datasets, undefended R-1 remains ≥0.87 on BERT checkpoints and ≥0.98 on eight of nine

TABLE II A BLATION S TUDY OF AEGIS C OMPONENTS ON W IKITEXT-2. AEGIS† DENOTES THE DUAL - CHANNEL VARIANT ( FREEZE + EMBED ), WHILE AEGIS‡ DENOTES THE FULL TRIPLE - CHANNEL MECHANISM INCLUDING MLP PROJECTION UNIFORMISATION . Components Freeze

Embed

GPT-2 (117M) MLP

DAGER

Embed-Norm

PPL

Acc

F1

DAGER

Embed-Norm

PPL

Acc

F1

None

1.000

0.520

26.0

0.38

0.33

1.000

0.556

25.7

0.38

0.33

AEGIS†

0.509 1.000 0.546 0.027 0.509 0.033 0.024

0.509 0.024 0.546 0.021 0.509 0.028 0.022

27.2 24.6 27.5 25.6 27.2 26.5 25.6

0.37 0.39 0.37 0.38 0.37 0.38 0.38

0.32 0.34 0.32 0.33 0.32 0.33 0.33

0.524 1.000 0.519 0.025 0.524 0.029 0.028

0.524 0.031 0.519 0.019 0.524 0.024 0.026

22.9 25.1 22.8 22.1 22.9 22.2 22.1

0.40 0.38 0.40 0.41 0.40 0.41 0.41

0.35 0.33 0.35 0.36 0.35 0.36 0.36

✓ ✓ ✓ ✓ ✓ ✓

✓ ✓ ✓

✓ ✓ ✓

GPT-2-XL (1.5B)

Variant

AEGIS‡

decoders. AEGIS drives every defended R-1 entry to at most 0.005. Mean test PPL spans 6.8–60.1 without defence and tightens to 6.8–45.8 with AEGIS on the reported rows. GPT2-class models stay within 1% of their undefended PPL, whereas GPT-2-XL improve.Table II gives the component utility results. Observations. 1) Universal defence. AEGIS reduces DAGER R-1 to ≤ 0.005 for every model tested, a relative reduction exceeding 99% in 10 of 11 cases. The effect holds across all three decoder families and both encoder architectures, so the triple-channel masking mechanism is not architecturespecific. 2) Anomalous undefended R-1 for LLaMA-2-13B. The undefended 13B model exhibits lower attack success (0.500) than other LLaMA variants (≈0.98). We attribute this to gradient rank collapse in very large BF16 models: DAGER’s SVD thresholds were calibrated for FP32 gradients and occasionally miss span candidates at 13B scale. Even under this weaker baseline the defence reduces R-1 by more than 99%. 3) Privacy–utility trade-off. The component results in Table II show that freezing and flooding keep useful training paths. Full AEGIS keeps GPT-2 utility and improves the reported GPT-2-XL utility. Figure 5 shows the training trajectories; Section VII discusses the trade-off. Across model scale. Tables III and IV report DAGER R1 for all eleven models (124M–13B); the defence reduces attack success to near-zero uniformly regardless of parameter count, so AEGIS’s empirical behaviour does not degrade with scale. LLaMA-2-13B is omitted from full six-dataset coverage because only three dataset runs were available when this draft was compiled. Per-cell utility breakdowns (accuracy, F1, loss, PPL) for all eleven checkpoints across the six datasets are deferred to Appendix E to keep the main flow on the privacy axis. C. Adaptive Attack: With vs. Without Channel 3 Uniformisation To measure whether the MLP-fc uniformisation (Section IV-D) is necessary in addition to Channels 1–2, we run the

adaptive DAGER variant described in Section VI-A: its SVD (ℓ) span-estimation stage is re-targeted from ∇Wo (zero under (ℓ) AEGIS) to ∇Wfc (the MLP-fc gradient). We compare two AEGIS configurations on the same fine-tuning checkpoints used for Tables III and IV: (i) AEGIS-2ch: attention freeze + embedding flood only (Eq. 1+Eq. 2); (ii) AEGIS-3ch: the full defence, adding the MLP-fc uniformisation (Eq. 4). Interpretation (defence ability). Channels 1 and 2 are zeroed by AEGIS-2ch on both models. For Channel 3, AEGIS2ch increases MLP-SVD R-1 on GPT-2-XL (0.166 → 0.292): removing the dominant channels makes the residual MLP signal cleaner. AEGIS-3ch closes this gap, collapsing recovery to 0.036 (≈8× below AEGIS-2ch, below the 0.05 brokenattack threshold). On GPT-2 small the Channel 3 benefit is negligible (0.265 for both configurations) because the smaller MLP gradient carries lower signal-to-noise; the gain is therefore scale-dependent. Interpretation (utility & trainability). Adding MLP-fc uniformisation on top of AEGIS-2ch costs essentially nothing in PPL (GPT-2: 25.7 → 25.7; GPT-2-XL: 22.3 → 22.1). Training curves (loss vs. step) are qualitatively unchanged: loss decreases monotonically and no divergence occurs, matching the biased-SGD prediction of Proposition 1 extended to the MLP block. The triple-channel defence is therefore necessary on the larger decoder, where AEGIS-2ch leaves an exploitable Channel 3, and essentially free at deployment cost; on smaller decoders the extra MLP uniformisation does not change the outcome of the implemented attacker but remains a safeguard against the stronger Robust-PCA adversary in Limitation 1. This ablation used a utility-preserving MLP retain ratio ρfc = 0.3. The broader side-channel stress test below uses the privacy-hardening endpoint ρfc = 0, because the validation run showed that retaining a fixed clean MLP subspace can leave exploitable signal on GPT-2-XL. a) Adaptive side-channel stress test.: An adaptive adversary could retarget beyond the original MLP-fc probe, e.g. to the MLP projection Wproj , LayerNorm parameters, or an untied LM head. We therefore evaluate robustness on GPT2 and GPT-2-XL, WikiText-2 and Rotten Tomatoes, and both tied and artificially untied heads. A probe is counted as a valid adaptive attack only when its undefended ROUGE-1 exceeds

TABLE III DAGER ATTACK SIMILARITY ( DECODER - ONLY ). P ER DATASET BLOCK (8 COLUMNS ): NO - DEFENCE R-1, R-2, R-L, METEOR (M); AEGIS R-1, R-2, R-L, M ( LOWER IS BETTER FOR ATTACK SUCCESS ). BAND 1 = CLASSIFICATION - STYLE DATASETS ; BAND 2 = LM/ SUMMARISATION . E NCODERS : TABLE IV. Rotten Tomatoes No def. Model

R-1 R-2 R-L

Emotion

AEGIS M

R-1 R-2 R-L

Fin. PhraseBank

No def. M

AEGIS

R-1 R-2 R-L

M

R-1 R-2 R-L

No def. M

R-1 R-2 R-L

AEGIS M

R-1 R-2 R-L

M

GPT-2 1.00 1.00 1.00 0.94 0.05 0.00 0.05 0.02 0.97 0.97 0.97 0.96 0.03 0.00 0.03 0.03 1.00 1.00 1.00 0.94 0.06 0.00 0.06 0.03 GPT-2-medium 1.00 1.00 1.00 0.94 0.06 0.00 0.06 0.03 0.97 0.97 0.97 0.96 0.05 0.00 0.05 0.02 1.00 1.00 1.00 0.94 0.03 0.00 0.03 0.01 GPT-2-large 1.00 1.00 1.00 0.94 0.07 0.00 0.07 0.02 0.97 0.97 0.97 0.96 0.06 0.00 0.06 0.03 1.00 1.00 1.00 0.94 0.05 0.00 0.05 0.01 GPT-2-XL 1.00 1.00 1.00 0.94 0.05 0.00 0.05 0.02 0.97 0.97 0.97 0.96 0.08 0.00 0.08 0.02 1.00 1.00 1.00 0.94 0.02 0.00 0.02 0.02 Gemma-2-2B 1.00 1.00 1.00 0.94 0.04 0.00 0.04 0.02 0.98 0.98 0.98 0.96 0.07 0.00 0.07 0.02 1.00 1.00 1.00 0.94 0.03 0.00 0.03 0.02 Gemma-2-9B 0.99 0.99 0.99 0.93 0.07 0.00 0.07 0.01 0.98 0.98 0.98 0.96 0.04 0.00 0.04 0.02 1.00 1.00 1.00 0.94 0.02 0.00 0.02 0.02 LLaMA-2-7B 0.98 0.98 0.98 0.92 0.08 0.00 0.08 0.02 0.97 0.96 0.97 0.95 0.08 0.00 0.08 0.03 0.99 0.99 0.99 0.94 0.08 0.00 0.08 0.03 LLaMA-2-13B 0.70 0.51 0.69 0.59 0.07 0.00 0.07 0.01 0.98 0.98 0.98 0.97 0.07 0.00 0.07 0.03 1.00 1.00 1.00 0.95 0.04 0.00 0.04 0.02 LLaMA-3.1-8B 0.99 0.99 0.99 0.92 0.05 0.00 0.05 0.01 0.97 0.97 0.97 0.96 0.05 0.00 0.05 0.01 1.00 1.00 1.00 0.93 0.04 0.00 0.04 0.01 WikiText-2 No def. Model

R-1 R-2 R-L

DialogSum AEGIS

M

R-1 R-2 R-L

No def. M

CNN/DailyMail AEGIS

R-1 R-2 R-L

M

R-1 R-2 R-L

No def. M

R-1 R-2 R-L

AEGIS M

R-1 R-2 R-L

M

GPT-2 1.00 0.70 1.00 0.66 0.05 0.00 0.05 0.03 1.00 1.00 1.00 0.90 0.04 0.00 0.04 0.03 1.00 1.00 1.00 0.91 0.03 0.00 0.03 0.02 GPT-2-medium 1.00 0.70 1.00 0.67 0.04 0.00 0.04 0.03 1.00 1.00 1.00 0.90 0.08 0.00 0.08 0.04 1.00 1.00 1.00 0.91 0.08 0.00 0.08 0.02 GPT-2-large 1.00 0.70 1.00 0.70 0.02 0.00 0.02 0.02 1.00 1.00 1.00 0.90 0.04 0.00 0.04 0.02 1.00 1.00 1.00 0.91 0.04 0.00 0.04 0.01 GPT-2-XL 1.00 0.70 1.00 0.65 0.07 0.00 0.07 0.01 1.00 1.00 1.00 0.90 0.05 0.00 0.05 0.02 1.00 1.00 1.00 0.91 0.04 0.00 0.04 0.03 Gemma-2-2B 1.00 0.70 1.00 0.70 0.07 0.00 0.07 0.03 1.00 1.00 1.00 0.90 0.06 0.00 0.06 0.01 1.00 1.00 1.00 0.91 0.04 0.00 0.04 0.02 Gemma-2-9B 0.96 0.65 0.96 0.66 0.06 0.00 0.06 0.02 1.00 1.00 1.00 0.90 0.05 0.00 0.05 0.03 1.00 1.00 1.00 0.91 0.08 0.00 0.08 0.02 LLaMA-2-7B 0.99 0.69 0.99 0.70 0.06 0.00 0.06 0.01 0.97 0.97 0.97 0.86 0.02 0.00 0.02 0.02 0.99 0.99 0.99 0.91 0.06 0.00 0.06 0.02 LLaMA-2-13B 0.91 0.63 0.91 0.65 0.07 0.00 0.07 0.03 0.42 0.26 0.42 0.22 0.09 0.00 0.09 0.03 0.58 0.45 0.58 0.42 0.07 0.00 0.07 0.02 LLaMA-3.1-8B 0.97 0.60 0.97 0.66 0.04 0.00 0.04 0.02 1.00 1.00 1.00 0.90 0.08 0.00 0.08 0.02 1.00 1.00 1.00 0.90 0.02 0.00 0.02 0.02

TABLE IV DAGER ATTACK SIMILARITY ( ENCODER ). S AME COLUMN LAYOUT AS TABLE III. Rotten Tomatoes No def. Model

R-1 R-2 R-L

Emotion

AEGIS M

R-1 R-2 R-L

No def. M

R-1 R-2 R-L

Fin. PhraseBank AEGIS

M

R-1 R-2 R-L

No def. M

R-1 R-2 R-L

AEGIS M

R-1 R-2 R-L

M

BERT-base 1.00 1.00 1.00 1.00 0.06 0.00 0.06 0.02 1.00 1.00 1.00 1.00 0.08 0.00 0.08 0.03 1.00 1.00 1.00 1.00 0.03 0.00 0.03 0.01 BERT-large 1.00 1.00 1.00 1.00 0.06 0.00 0.06 0.03 1.00 1.00 1.00 1.00 0.08 0.00 0.08 0.02 1.00 1.00 1.00 1.00 0.04 0.00 0.04 0.03 WikiText-2 No def. Model

R-1 R-2 R-L

DialogSum AEGIS

M

R-1 R-2 R-L

No def. M

R-1 R-2 R-L

CNN/DailyMail AEGIS

M

R-1 R-2 R-L

No def. M

R-1 R-2 R-L

AEGIS M

R-1 R-2 R-L

M

BERT-base 0.53 0.36 0.53 0.45 0.08 0.00 0.08 0.02 0.95 0.83 0.95 0.87 0.04 0.00 0.04 0.02 0.79 0.66 0.79 0.71 0.06 0.00 0.06 0.03 BERT-large 0.67 0.54 0.67 0.67 0.06 0.00 0.06 0.01 0.87 0.64 0.87 0.74 0.04 0.00 0.04 0.03 0.78 0.64 0.78 0.71 0.05 0.00 0.05 0.02

both 0.10 and a matched random-gradient null baseline by at least 0.05; valid probes are considered blocked when AEGIS reduces ROUGE-1 to at most max(0.05, null + 0.05).

Table VI supports two conclusions. First, the MLP-fc and LM-head row-norm probes are real side channels: they recover substantial token overlap on the undefended model and are suppressed to the null regime by AEGIS, including in the untied-head setting. Second, in this evaluation the MLP projection and LayerNorm probes do not constitute standalone adaptive attacks because they fail the undefended validity gate; nevertheless, their gradients are also flooded in the implementation, so they do not remain clean fallback channels.

D. Qualitative Reconstruction Examples To complement the aggregate ROUGE-1 numbers, Table XIII (Appendix F) shows word-for-word reconstruction outputs for GPT-2 small on WikiText-2, Rotten Tomatoes, and DialogSum, attacked by DAGER and GRAB [7] under no defence, Prune-50%, and AEGIS. Table VII shows two representative Rotten Tomatoes sentences to illustrate this contrast concretely. The undefended gradient exposes every token verbatim, while the uniformised gradient produces semantically incoherent output with no overlap with the original. The pattern holds across all nine examples in Table XIII: both attackers collapse to random vocabulary samples (R-1 = 0.00) under

TABLE V A DAPTIVE ATTACK ABLATION (W IKI T EXT-2, R-1; LOWER = BETTER PRIVACY ). C H 1: DAGER ON ∇Wo ; C H 2: E MBED -N ORM ON ∇We ; C H 3: MLP-SVD ON ∇Wfc . ROUGE-1 (↓)

TABLE VII Q UALITATIVE RECONSTRUCTION EXAMPLES ( TWO SENTENCES , GPT-2 SMALL , ROTTEN T OMATOES ; FOUR ATTACKS , UNDEFENDED VS . AEGIS). G REEN : CORRECT TOKEN IN CORRECT POSITION . O RANGE : CORRECT TOKEN , WRONG POSITION . F ULL TABLE INCLUDING P RUNE -50%: A PPENDIX F.

Model

Defence

Ch1

Ch2

Ch3

Defence

GPT-2

None AEGIS-2ch AEGIS-3ch

1.000 0.024 0.027

0.520 0.021 0.019

0.362 0.265 0.265

Ex 1: Original: “sometimes seems less like storytelling than something the otherwise compelling director needed to get off his chest.”

GPT-2-XL

None AEGIS-2ch AEGIS-3ch

1.000 0.025 0.028

0.556 0.022 0.024

0.166 0.292 0.036

None

MLP-fc MLP-proj LayerNorm LM head

DAGER GRAB

TABLE VI A DAPTIVE SIDE - CHANNEL STRESS TEST . R ANGES SPAN 8 SETTINGS (2 MODELS × 2 DATASETS × TIED / UNTIED HEAD ). “VALID ” = UNDEFENDED R-1 BEATS NULL BY >0.05; “B LOCKED ” = VALID SETTINGS SUPPRESSED TO BROKEN - ATTACK THRESHOLD . Channel

Attack

Undefended / Null R-1

AEGIS R-1

Valid / Blocked

0.16–0.36 / 0.00 0.00–0.06 / 0.00 0.00–0.01 / 0.00–0.01 0.47–0.64 / 0.00–0.00

0.02–0.03 0.02–0.02 0.02–0.02

8/8 0/0 0/0

0.02–0.03

8/8

AEGIS, whereas the undefended baseline and Prune-50% sustain near-verbatim or near-complete token recovery. This is the empirical counterpart of Theorem 2: once the embedding gradient is uniformised, no post-hoc filter or optimisation we are aware of can discriminate true input tokens from calibrated noise.

SOMP FedSpy-LLM

AEGIS‡

GRAB SOMP FedSpy-LLM

sometimes seems less like storytelling than something the otherwise compelling director needed to get off his chest sometimes seems less like storytelling than something the otherwise compelling director something compelling get off like storytelling seems seems less like storytelling than something the otherwise compelling director needed to get off his chest to a than off the otherwise compelling needed get his less director storytelling something . like chest to. Meth sym west symbolism let-------- McN genius Hebophobia/// Yun disease Fn Investigative popupEGA Territories judgment seems storytelling highlight highlight movie give men run seems givemad away off off off off]( (empty) planetclud MOT Fant. . . OU chapter462 hysteria royal believe glyph. . . revival AwardsBonGuard Station

Ex 2: Original: “what happened with pluto nash ? how did it ever get made ?” None

DAGER GRAB SOMP FedSpy-LLM

E. Defence Effectiveness Across Attack Types The main evaluation targets DAGER, the strongest analytical attack. Figure 2 extends this to three further attacks on GPT-2 small across three datasets: GRAB [7], a gradient-matching optimisation attack that does not rely on SVD channel structure; SOMP [8], a subspace-guided pursuit method that generalises DAGER via per-head pooling and OMP reconstruction; and F ED S PY-LLM [9], which exploits embedding-gradient row-norm sparsity for direct token recovery and partial-gradient alignment for ordering. All three analytical attacks — DAGER, SOMP, and F ED S PY-LLM — are eliminated completely (R-1 = 0.000 on every dataset), as expected from the channel-closure guarantees: each attack relies on either the Q-weight gradient subspace (DAGER, SOMP) or the embedding gradient rownorm sparsity (F ED S PY-LLM), both of which Channels 1–2 of AEGIS structurally mask. a) Optimisation-based attacks: GRAB.: GRAB is the lone partial-degrade case in our evaluation. Mean R-1 drops from 0.533 → 0.463 on WikiText-2 and from 0.660 → 0.283 on Rotten Tomatoes, while DialogSum shows a marginal increase (0.388 → 0.526) reflecting high variance over three sentences

DAGER

Reconstructed Sequence

AEGIS‡

DAGER GRAB SOMP FedSpy-LLM

what happened with pluto nash ? how did it ever get made whiff whiff. pl. nash ? how how it ever get made ? happened with pluto nash ? how did it ever get made ? how made it ever didash get pluto ? with n? to happen angrilyit?tarian ESC BitcoinstaboolaNazi wiped goodnessovereEls262rection waiter --------???????? just did n did ever? ? How happenedor pl gottenrations pl (empty) discreteplanesfortablebergerannappa Mesh ost Potter Obesity anchors DS805amer eas

and GRAB’s sensitivity to dialogue-style token-length distribution; the absolute R-1 remains low per-sentence. This pattern is consistent with GRAB’s mechanism: it recovers tokens by iteratively minimising gradient distance without requiring the SVD subspace signal that AEGIS structurally masks. The attention gradient zeroing and embedding flooding still degrade GRAB by removing the structural anchors (low-rank subspace alignment, embedding row-norm sparsity) that accelerate its optimisation, but residual token information leaks through the gradient-matching objective itself. Full defeat of optimisationbased attacks would require an orthogonal mechanism such as DP-SGD noise or data sanitisation composable with AEGIS.

= better privacy) ROUGE (no defence;

WikiText-2

R-1

Rotten Tom.

DialogSum

R-2

TABLE VIII OVERHEAD COMPARISON . A BSOLUTE STEP TIME , PEAK ALLOCATED MEMORY, AND DENSE GRADIENT PAYLOAD ACROSS GPT-2, GPT-2-XL, AND LL A MA-2-7B ( RANGES ). M ULTIPLIERS RELATIVE TO UNDEFENDED TRAINING ARE IN PARENTHESES . L OWER IS BETTER .

R-L

1.0 0.8 0.6 0.4

Defence

0.2 0.0

AEGIS =0

DAGER SOMP FedSpy-LLM

AEGIS =0

DAGER SOMP FedSpy-LLM

AEGIS =0

Peak memory Dense payload

19.4–97.1 ms 2.24–50.5 GiB 0.46–12.6 GiB (1.00×) (1.00×) (1.00×) 146–362 ms 3.01–75.4 GiB 0.46–12.6 GiB ∗ DP-SGD (2.31–7.53×) (1.35–1.49×) (1.00×) 26.0–179 ms 2.24–50.6 GiB 0.46–12.6 GiB Pruning (90%) (1.34–1.84×) (1.00×) (1.00×) 20.7–98.8 ms 2.24–50.5 GiB 0.46–12.6 GiB Soteria (1.02–1.07×) (1.00×) (1.00×) 21.4–118 ms 2.01–38.5 GiB 0.36–8.55 GiB AEGIS (1.10–1.22×) (0.76–0.90×) (0.68–0.77×)

DAGER SOMP FedSpy-LLM

Fig. 2. ROUGE-1 / R-2 / R-L under AEGIS vs. no defence for three analytical gradient-inversion attacks across three datasets (GPT-2 small; mean over 3–5 sentences). Bar shades within each dataset colour encode the three metrics. All three attacks collapse to = 0.00 under AEGIS (red arrows). Note the F ED S PY-LLM R-1 vs. R-2 gap: tokens are recovered but their ordering is scrambled, so bigram and longest-common-subsequence overlap drops sharply. GRAB (gradient-matching optimisation) is discussed in the text below; LAMP and TAG omitted (both score ≤0.10 undefended).

Step time

Undefended

∗

Exact per-example microbatch DP-SGD (C=1, σ=1); optimized implementations may be faster.

F. Comparison with Existing Defences To contextualise AEGIS’s privacy–utility trade-off, we compare against four established defence families under identical conditions across three model scales (GPT-2 124M, GPT-2XL 1.5B, LLaMA-2-7B) and two datasets (WikiText-2, Rotten Tomatoes), giving 54 (model, dataset, defence) cells in total. All defences share the same DAGER/MLP-SVD/Embed-Norm attack implementations and training hyperparameters: • DP-SGD noise (σ ∈ {0.001, 0.01, 0.1}): additive noise on all gradients, matching the attack-evaluation protocol of [5]; • Gradient pruning [3] (top-k zeroing, k ∈ {50, 90, 99}%): the canonical defence proposed alongside the DLG attack by Zhu et al.; • Soteria [13] (input-gradient masking on the LM head, Sun et al., CVPR 2021); ‡ • AEGIS (ours): freeze + flood across all three channels. Per-cell breakdown (Figure 3). The bar chart resolves the comparison cell by cell. On DAGER ROUGE-1, only AEGIS collapses recovery to zero across every (model, dataset) pair; DP-SGD noise reduces R-1 but with strong dataset sensitivity (e.g. on LLaMA-2-7B σ=0.001 leaves R-1=0.51 on WikiText-2 vs. 0.05 on Rotten Tomatoes). Gradient pruning gives an inconsistent privacy curve: 50% pruning is weak (R1≈0.55–0.81 on GPT-2), and even 99% pruning still leaks (R-1≥0.39). Soteria leaves the attack essentially unimpaired (R-1≈1.0 on the small-and-medium decoders), confirming that input-gradient masking is orthogonal to DAGER’s SVD-based recovery channel. On the utility row, AEGIS matches or improves the undefended PPL on 8/12 cells, whereas Soteria roughly doubles PPL on every cell and the heaviest pruning (k=99%) shows brittle behaviour (LLaMA-2-7B × WikiText2: PPL 9.7 but R-1 still 0.11). Pareto front (Figure 4). Aggregating across datasets, each defence becomes a single point per model panel, with error bars over dataset variance. AEGIS is the unique point inside the bottom-left ideal region (R-1<0.05, PPL ≤ undefended) on all three model scales. The DP-SGD family traces a curve along the privacy axis but never crosses the broken-attack

line, and at the highest noise level the gradient becomes too noisy to support stable training on Rotten Tomatoes (visible as the long PPL whisker on GPT-2 σ=0.1). Pruning and Soteria both sit far from the ideal corner. The visual gap between AEGIS and the closest competitor on the privacy axis (≥ 0.2 on every panel) is exactly the “zero versus non-zero recovery” separation predicted by the dual-channel analysis (Theorem 2): no perturbation-only defence can reach R-1=0 because the binary embedding signal survives any sub-uniform noise budget. VII. D ISCUSSION A. Broader Implications AEGIS is complementary to, rather than a replacement for, existing FL privacy mechanisms. Unlike DP-SGD [11] and gradient pruning [3], which add noise uniformly at a cost to utility, AEGIS eliminates signal structurally only in the attack-relevant channels. Unlike data-level obfuscation, it trains on the original data and preserves the learning objective. It is composable with SecAgg [28] for stronger threat models: SecAgg’s cryptographic protection is independent of gradient structure, so the two layers address orthogonal threat surfaces. Communication, memory, and runtime. Table VIII reports absolute costs and relative multipliers. AEGIS takes 21.4–118 ms per step (1.10–1.22× undefended). It is faster than the measured pruning and DP-SGD baselines. Freezing attention reduces peak memory to 2.01–38.5 GiB and dense payload to 0.36–8.55 GiB. Freezing can also act as a structural regulariser [14], [15] which keeps the pretrained attention patterns and reduces the number of parameters changed during fine-tuning. This may reduce overfitting and help explain the lower PPL observed for the larger models. B. Limitations Remark 5 (Threat model scope). Our experiments use singleclient, single-step FedSGD. Multi-step FedAvg, client collusion, repeated-round temporal averaging, and an active server that modifies the model or objective are not evaluated. Testing calibration under these settings is a direct next step.

Baseline Defence Comparison: DAGER ROUGE-1 (top) and Val PPL (bottom) GPT-2-XL (1.5B)

WikiText-2 Rotten Tomatoes

1.0

DAGER ROUGE-1

DAGER ROUGE-1

0.8

0.6

0.4

0.2

0.0

1.0

1.0

0.8

0.8

DAGER ROUGE-1

GPT-2 (124M)

0.6

0.4

0.2

Undef.

DP 0.001

DP 0.01

DP 0.1

Prune 50%

Prune 90%

500

500

400

400

300

300

0.6

0.4

0.2

0.0

Prune Soteria AEGIS 99%

LLaMA-2-7B (7B)

Undef.

DP 0.001

DP 0.01

DP 0.1

Prune 50%

Prune 90%

0.0

Prune Soteria AEGIS 99%

Undef.

DP 0.001

DP 0.01

DP 0.1

Prune 50%

Prune 90%

Prune Soteria AEGIS 99%

Undef.

DP 0.001

DP 0.01

DP 0.1

Prune 50%

Prune 90%

Prune Soteria AEGIS 99%

175

200

200

100

100

0

0

Val PPL

125

Val PPL

Val PPL

150

100 75 50

Undef.

DP 0.001

DP 0.01

DP 0.1

Prune 50%

Prune 90%

Prune Soteria AEGIS 99%

25 Undef.

DP 0.001

DP 0.01

DP 0.1

Prune 50%

Prune 90%

Prune Soteria AEGIS 99%

0

Fig. 3. Per-cell breakdown of the nine defences across three model scales and two datasets (WikiText-2 in blue, Rotten Tomatoes in red). Top row: DAGER ROUGE-1 (lower is better privacy); bottom row: validation perplexity (lower is better utility). The highlighted right-most column is AEGIS. Every other defence leaves a substantial reconstruction signal (R-1 ≥ 0.05) on at least one cell, while AEGIS consistently drives R-1 to zero and matches or beats the lowest PPL. Soteria notably fails on both axes (no privacy gain, ∼2× PPL).

Privacy Utility Trade-off: DAGER ROUGE-1 vs. Val PPL (mean ± std over 2 datasets)

GPT-2-XL (1.5B)

400 300 200

AEGIS

100 0.0

0.2

0.4

0.6

0.8

DAGER ROUGE-1 ( better privacy)

1.0

Undefended DP-SGD =0.001

400 300 200

AEGIS

100 0.0

0.2

0.4

0.6

0.8

DAGER ROUGE-1 ( better privacy)

DP-SGD =0.01 DP-SGD =0.1

Prune 50% Prune 90%

tems that otherwise transmit embedding gradients sparsely. Evaluating privacy-aware compression in these settings is future work.

LLaMA-2-7B (7B)

500

Val PPL ( better utility)

500

Val PPL ( better utility)

Val PPL ( better utility)

GPT-2 (124M)

1.0

Prune 99% Soteria

175 150 125 100 75 50

VIII. C ONCLUSION

AEGIS

25 0.0

0.2

0.4

0.6

0.8

DAGER ROUGE-1 ( better privacy)

1.0

AEGIS (ours)

Fig. 4. Privacy–utility Pareto per model (mean ± std over the two datasets). x: DAGER ROUGE-1 (lower = better privacy); y: validation PPL (lower = better utility). Error bars show across-dataset variance; the dotted line at R-1=0.05 marks the conventional “broken-attack” threshold. AEGIS (red star) is the only method that lands in the ideal bottom-left corner on all three model scales, Pareto-dominating every baseline.

1) Attacks and guarantees. AEGIS has no (ε, δ)-DP guarantee and does not fully defeat optimisation-based GIAs or Robust-PCA/Marchenko–Pastur denoising, or attacks that aggregate weak signals across rounds or unexplored layers. Future work should combine it with DP/SecAgg and evaluate these adaptive attacks. 2) Systems cost. Flooding makes the embedding gradient dense and prevents row-sparse export. Attention freezing offsets this in our three measured models, reducing total dense payload by 23–32%. Although AEGIS reduces the total dense payload in these models, this benefit may not hold for models with much larger vocabularies or sys-

Closed-form gradient inversion attacks—SPEAR, DAGER, GRAB, and their descendants—share a common structural premise: that the gradient of a federated LLM fine-tuning step decomposes into a small number of analytically recoverable channels, each exploiting a specific linearity or sparsity in the network architecture. Prior defences that perturb the gradient (DP-SGD, pruning) or modify the data do not close these channels structurally; they only raise the noise floor, leaving the channel signal in place for a sufficiently powerful denoiser. AEGIS starts from a different question: which structural properties of the gradient does the attack actually require, and can each of them be eliminated at the source? For the attention-MLP decoder architectures that dominate current LLM fine-tuning, the answer is three channels. Freezing attention parameters closes the first channel without touching the loss surface; calibrated dense-noise uniformisation of the embedding gradient destroys the sparsity signal in the second; an analogous uniformisation of the MLP expansion gradient submerges the third beneath the random-matrix noise bulk. The forward pass and the learning objective are untouched. Beyond the specific mechanism, our analysis suggests that analytical gradient inversion is fundamentally channel-

structured. Under this view, FL privacy becomes a completeness problem: a defence must account for every independent channel an adaptive adversary can pivot to, not merely the one demonstrated by the current attack. Open questions include whether attention freezing satisfies formal (ε, δ)-DP guarantees, how it composes with DP-SGD, and how the single-step calibration extends to multi-step FedAvg. R EPRODUCIBILITY All implementation code, experiment scripts, and result logs are included in the supplementary material. ACKNOWLEDGMENT This work was supported by the Department of Environment and Science of Queensland State Government under Quantum 2032 Challenge Program (Project #Q2032001). R EFERENCES [1] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Artificial intelligence and statistics. Pmlr, 2017, pp. 1273– 1282. [2] P. Kairouz and H. B. McMahan, “Advances and open problems in federated learning,” Foundations and trends in machine learning, vol. 14, no. 1-2, pp. 1–210, 2021. [3] L. Zhu, Z. Liu, and S. Han, “Deep leakage from gradients,” Advances in neural information processing systems, vol. 32, 2019. [4] J. Geiping, H. Bauermeister, H. Dröge, and M. Moeller, “Inverting gradients-how easy is it to break privacy in federated learning?” Advances in neural information processing systems, vol. 33, pp. 16 937– 16 947, 2020. [5] I. Petrov, D. I. Dimitrov, M. Baader, M. N. Müller, and M. Vechev, “Dager: Exact gradient inversion for large language models,” Advances in neural information processing systems, vol. 37, pp. 87 801–87 830, 2024. [6] M. Balunovic, D. Dimitrov, N. Jovanović, and M. Vechev, “Lamp: Extracting text from gradients with language model priors,” Advances in Neural Information Processing Systems, vol. 35, pp. 7641–7654, 2022. [7] X. Feng, Z. Ma, Z. Wang, E. J. Chegne, M. Ma, A. Abuadbba, and G. Bai, “Uncovering gradient inversion risks in practical language model training,” in Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, 2024, pp. 3525–3539. [8] Y. Li and Q. Li, “Somp: Scalable gradient inversion for large language models via subspace-guided orthogonal matching pursuit,” 03 2026. [9] S. I. A. Meerza, F. Wang, and J. Liu, “Fedspy-llm: Towards scalable and generalizable data reconstruction attacks from gradients on llms,” 04 2026. [10] J. Deng, Y. Wang, J. Li, C. Wang, C. Shang, H. Liu, S. Rajasekaran, and C. Ding, “Tag: Gradient attack on transformer-based language models,” in Findings of the association for computational linguistics: EMNLP 2021, 2021, pp. 3600–3610. [11] M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang, “Deep learning with differential privacy,” in Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, 2016, pp. 308–318. [12] D. Yu, S. Naik, A. Backurs, S. Gopi, H. A. Inan, G. Kamath, J. Kulkarni, Y. T. Lee, A. Manoel, L. Wutschitz et al., “Differentially private finetuning of language models,” arXiv preprint arXiv:2110.06500, 2021. [13] J. Sun, A. Li, B. Wang, H. Yang, H. Li, and Y. Chen, “Soteria: Provable defense against privacy leakage in federated learning from representation perspective,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 9311–9319. [14] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen et al., “Lora: Low-rank adaptation of large language models.” Iclr, vol. 1, no. 2, p. 3, 2022. [15] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-efficient transfer learning for nlp,” in International conference on machine learning. PMLR, 2019, pp. 2790–2799.

[16] B. Zhao, K. R. Mopuri, and H. Bilen, “idlg: Improved deep leakage from gradients,” arXiv preprint arXiv:2001.02610, 2020. [17] J. Lu, X. S. Zhang, T. Zhao, X. He, and J. Cheng, “April: Finding the achilles’ heel on privacy for vision transformers,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 10 051–10 060. [18] D. I. Dimitrov, M. Baader, M. N. Müller, and M. Vechev, “Spear: Exact gradient inversion of batches in federated learning,” Advances in Neural Information Processing Systems, vol. 37, pp. 106 768–106 799, 2024. [19] J. Zhu and M. Blaschko, “R-gap: Recursive gradient attack on privacy,” arXiv preprint arXiv:2010.07733, 2020. [20] L. Fowl, J. Geiping, W. Czaja, M. Goldblum, and T. Goldstein, “Robbing the fed: Directly obtaining private data in federated learning with modified models,” arXiv preprint arXiv:2110.13057, 2021. [21] L. Fowl, J. Geiping, S. Reich, Y. Wen, W. Czaja, M. Goldblum, and T. Goldstein, “Decepticons: Corrupted transformers breach privacy in federated learning for language models,” arXiv preprint arXiv:2201.12675, 2022. [22] M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang, “Deep learning with differential privacy,” in Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, 2016, pp. 308–318. [23] Y. Lin, S. Han, H. Mao, Y. Wang, and B. Dally, “Deep gradient compression: Reducing the communication bandwidth for distributed training,” in International Conference on Learning Representations, 2018. [Online]. Available: https://openreview.net/forum?id=SkhQHMW0W [24] Y. Huang, Z. Song, K. Li, and S. Arora, “Instahide: Instance-hiding schemes for private distributed learning,” in International conference on machine learning. PMLR, 2020, pp. 4507–4518. [25] N. Carlini, S. Deng, S. Garg, S. Jha, S. Mahloujifar, M. Mahmoody, A. Thakurta, and F. Tramèr, “Is private learning possible with instance encoding?” in 2021 IEEE Symposium on Security and Privacy (SP). IEEE, 2021, pp. 410–427. [26] O. Feyisetan, B. Balle, T. Drake, and T. Diethe, “Privacy- and utilitypreserving textual analysis via calibrated multivariate perturbations,” in Proceedings of the 13th International Conference on Web Search and Data Mining, ser. WSDM ’20. New York, NY, USA: Association for Computing Machinery, 2020, p. 178–186. [Online]. Available: https://doi.org/10.1145/3336191.3371856 [27] K. Bonawitz, V. Ivanov, B. Kreuter, A. Marcedone, H. B. McMahan, S. Patel, D. Ramage, A. Segal, and K. Seth, “Practical secure aggregation for privacy-preserving machine learning,” in proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, 2017, pp. 1175–1191. [28] K. H. Li, P. P. B. de Gusmão, D. J. Beutel, and N. D. Lane, “Secure aggregation for federated learning in flower,” in Proceedings of the 2nd ACM International Workshop on Distributed Machine Learning, 2021, pp. 8–14. [29] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., “Language models are unsupervised multitask learners,” OpenAI blog, vol. 1, no. 8, p. 9, 2019. [30] J. Lee and K. Toutanova, “Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, vol. 3, no. 8, pp. 4171–4186, 2018. [31] L. d. Moura and S. Ullrich, “The lean 4 theorem prover and programming language,” in International Conference on Automated Deduction. Springer, 2021, pp. 625–635. [32] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al., “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023. [33] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan et al., “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024. [34] G. Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, B. Shahriari, A. Ramé et al., “Gemma 2: Improving open language models at a practical size, 2024,” URL https://arxiv. org/abs/2408.00118, vol. 1, no. 3, 2024. [35] B. Pang and L. Lee, “Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales,” in Proceedings of the 43rd annual meeting of the association for computational linguistics (ACL’05), 2005, pp. 115–124. [36] E. Saravia, H.-C. T. Liu, Y.-H. Huang, J. Wu, and Y.-S. Chen, “Carer: Contextualized affect representations for emotion recognition,” in Pro-

ceedings of the 2018 conference on empirical methods in natural language processing, 2018, pp. 3687–3697. [37] P. Malo, A. Sinha, P. Korhonen, J. Wallenius, and P. Takala, “Good debt or bad debt: Detecting semantic orientations in economic texts,” Journal of the Association for Information Science and Technology, vol. 65, no. 4, pp. 782–796, 2014. [38] S. Merity, C. Xiong, J. Bradbury, and R. Socher, “Pointer sentinel mixture models,” arXiv preprint arXiv:1609.07843, 2016. [39] Y. Chen, Y. Liu, L. Chen, and Y. Zhang, “Dialogsum: A real-life scenario dialogue summarization dataset,” in Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, 2021, pp. 5062–5074. [40] K. M. Hermann, T. Kocisky, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, and P. Blunsom, “Teaching machines to read and comprehend,” Advances in neural information processing systems, vol. 28, 2015. [41] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101, 2017. [42] C.-Y. Lin, “Rouge: A package for automatic evaluation of summaries,” in Text summarization branches out, 2004, pp. 74–81. [43] S. Banerjee and A. Lavie, “Meteor: An automatic metric for mt evaluation with improved correlation with human judgments,” in Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, 2005, pp. 65–72. [44] A. Ajalloeian and S. U. Stich, “On the convergence of sgd with biased gradients,” arXiv preprint arXiv:2008.00051, 2020. [45] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz et al., “Transformers: Stateof-the-art natural language processing,” in Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, 2020, pp. 38–45.

A PPENDIX A. Proofs a) Proof of Theorem 1 (Channel 1 Elimination): Proof. We provide a detailed proof of Channel 1 elimination under attention freezing. Consider a GPT-2-class model fθ with L transformer (ℓ) blocks. Each block ℓ contains attention parameters WQKV ∈ (ℓ) Rd×3d P and Wo ∈ Rd×d , among others. The training loss is T −1 L = − i=1 log fθ (ti+1 |t≤i ). Modern automatic differentiation (AD) frameworks (PyTorch, JAX) implement the chain rule by traversing a computational graph. When a parameter p has p.requires_grad = False, the AD engine: (a) excludes p from the gradient tape during the forward pass, and (b) skips gradient accumulation for p during the backward pass. Equivalently, the trainable parameter space omits p, and the exported gradient on the full parameter list is the zero extension of the reduced-space b (ℓ) L = 0 by construction. gradient. The result is ∇ Wo (ℓ) b Let G = ∇W(ℓ) L = 0 ∈ Rd×d . The range of the zero o matrix is the zero subspace. Equivalently, every singular value of G(ℓ) is zero, so the non-zero left-singular-vector basis in Definition 1 has no columns. Its projection matrix is therefore: U

(ℓ)

U

(ℓ) ⊤

= 0.

For any vocabulary token v with embedding ev ∈ Rd : rv(ℓ) = (ℓ)

∥ev − 0 · ev ∥ ∥ev ∥ = = 1. ∥ev ∥ ∥ev ∥

Since rv = 1 for all v ∈ V and all ℓ ∈ [L], the ranking is uniform and provides zero bits of information about the input token set.

Crucially, the forward pass computation h(ℓ) = (ℓ) Attn(Q(ℓ) , K(ℓ) , V(ℓ) ) + h(ℓ−1) uses the values of WQKV (ℓ) and Wo , which are the pretrained weights. Only the gradient is zeroed; the weights themselves are not modified. The attention mechanism continues to compute contextual representations that are fed to the (trainable) MLP layers. b) Proof of Theorem 2 (Channel 2 Moment Bound): Proof. We provide a detailed proof under Assumption 1 (∥gv ∥2 ≤ C m̄ for all active rows v ∈ S, C ≥ 1). Let ∇WWTE ∈ RV ×d be the embedding gradient for sequence s = (t1 , . . . , tT ). Let S = S(s) = {v ∈ V : v appears in s}. By the structural property of embedding layers, row v has ∥∇WWTE [v, :]∥2 > 0 iff v ∈ S. Because S is nonempty, m̄ > 0. After AEGIS flooding with parameters (λ, ρ): ( ρ · ∇WWTE [v, :] + nv v ∈ S ˜ ∇WWTE [v, :] = nv v∈ /S where nv ∼ N (0, σ 2 Id ) with σ = λm̄. Since λ > 0 and m̄ > 0, the Gaussian is non-degenerate. ˜ WTE [v, :] = nv , and Pr[nv = 0] = 0. For For v ∈ / S, ∇W ˜ v ∈ S, ∇WWTE [v, :] = ρ∇WWTE [v, :] + nv , and this vector is zero only if nv = −ρ∇WWTE [v, :]; a non-degenerate Gaussian has probability zero of taking any prescribed point value. Since V is finite, the union of these zero-probability events still ˜ WTE [v, :]∥2 > 0 for all has probability zero. Therefore ∥∇W v simultaneously with probability 1. The binary signal (zero = absent, non-zero = present) is completely destroyed. For v ∈ / S: ˜ WTE [v, :]∥2 = ∥nv ∥2 = ∥∇W 2 2

d X

n2vj

j=1

where nvj ∼ N (0, σ 2 ), so each n2vj /σ 2 ∼ χ2 (1). Thus ∥nv ∥22 /σ 2 ∼ χ2 (d) and E[∥nv ∥22 ] = dσ 2 = dλ2 m̄2 . For v ∈ S: Let gv = ∇WWTE [v, :]. Then: ˜ WTE [v, :]∥22 = ∥ρgv +nv ∥22 = ρ2 ∥gv ∥22 +2ρgv⊤ nv +∥nv ∥22 . ∥∇W Since E[nv ] = 0, E[gv⊤ nv ] = 0, so: ˜ WTE [v, :]∥2 ] = ρ2 ∥gv ∥2 + dσ 2 . E[∥∇W 2 2 Under Assumption 1, the worst-case (maximum) excess expected squared norm of a real-token row over a noise-only row is: ∆max = ρ2 ∥gv ∥22 ≤ ρ2 C 2 m̄2 . The noise-only second moment is dλ2 m̄2 > 0. Dividing gives ˜ WTE [v, :]∥2 ] − dλ2 m̄2 ρ2 C 2 E[∥∇W 2 ≤ . 2 2 dλ m̄ dλ2 The lower bound 0 follows from ρ2 ∥gv ∥22 ≥ 0.

Convergence of validation PPL GPT-2 (124 M)

AEGIS acts as implicit regulariser

GPT-2-XL (1.5 B)

OPT-1.3B

validation PPL (log)

103

102

102 102

0

250

500

750

1000

1250

training step

1500

1750

2000

Phi-2 (2.7 B)

0

250

500

750

1000

1250

training step

1500

1750

2000

0

250

500

750

LLaMA-2-7B

102

1000

1250

training step

1750

2000

LLaMA-2-13B WikiText-2 Rotten Tomatoes DialogSum undefended AEGIS

102

103

validation PPL (log)

1500

102 101

101 0

250

500

750

1000

1250

training step

1500

1750

2000

0

250

500

750

1000

1250

training step

1500

1750

2000

0

1000

2000

3000

4000

training step

5000

6000

Fig. 5. Convergence of validation perplexity during fine-tuning across six models (124 M – 13 B). Solid lines: AEGIS‡ ; dashed lines: undefended baseline. Colours: WikiText-2, Rotten Tomatoes, DialogSum. AEGIS matches or improves the PPL trajectory for all models up to 7 B. LLaMA-2-13B shows a modest PPL increase (∼8%), attributable to its SGD optimiser (required at 13 B scale for memory).

c) Proof of Theorem 3 (Channel 3 MLP-fc Flooding Moment Bound): e (ℓ) = ρfc G(ℓ) +N(ℓ) . Proof. The MLP-fc flooded gradient is G (ℓ) (ℓ) With B = (ρfc − 1)G , we have the algebraic identity (ℓ)

ρfc G

(ℓ)

+N

(ℓ)

=G

(ℓ)

+(ρfc −1)G

(ℓ)

+N

(ℓ)

=G

(ℓ)

+B

Proposition 1 (Structural Biased-Gradient Decomposition). Let gk = ∇WWTE L(θk ; s) be the embedding-block gradient at step k, let ρ ∈ [0, 1], and suppose AEGIS exports bk = ρgk + nk g

(ℓ)

+N

.where E[nk | θk ] = 0 and E[∥nk ∥2 | θk ] = σ 2 . Define bk := Taking expectation conditional on the batch and using (ρ − 1)gk . Then e (ℓ) − G(ℓ) ] = B(ℓ) . Since ρfc ∈ [0, 1], E[N(ℓ) ] = 0 gives E[G b =g +b +n , g k

∥B(ℓ) ∥F = ∥(ρfc −1)G(ℓ) ∥F = |ρfc −1| ∥G(ℓ) ∥F = (1−ρfc )∥G(ℓ) ∥F .

k

k

k

E[b gk − gk | θk ] = bk , ∥bk ∥ = (1 − ρ)∥gk ∥.

Expanding the squared Frobenius norm,

∥ρfc G(ℓ) +N(ℓ) ∥2F = ρ2fc ∥G(ℓ) ∥2F +2ρfc ⟨G(ℓ) , N(ℓ) ⟩F +∥N(ℓ) ∥2F . Thus the embedding update is a stochastic-gradient step with deterministic relative bias coefficient 1 − ρ and conditional The cross term has zero expectation because G(ℓ) is fixed second moment σ 2 for the additive zero-mean noise. The same conditional on the batch and the entries of N(ℓ) are zero mean. decomposition applies verbatim to the MLP-fc uniformisation The matrix N(ℓ) ∈ Rd×4d has 4d2 independent entries, each after replacing gk by the corresponding block gradient and ρ (ℓ) 2 with variance σfc = λ2fc (m̄fc )2 . Therefore by ρfc . (ℓ)

E[∥N(ℓ) ∥2F ] = 4d2 λ2fc (m̄fc )2 , and the claimed exact second-moment identity follows. The excess over the noise-only moment is

bk = gk + (ρ − 1)gk + nk is immediate Proof. The identity g bk = ρgk + nk . Since bk = (ρ − 1)gk , from the definition g bk = gk + bk + nk . Taking conditional expectation this gives g and using E[nk | θk ] = 0 gives E[b gk − gk | θk ] = bk . Finally,

(ℓ)

e (ℓ) ∥2 ] − 4d2 λ2 (m̄ )2 = ρ2 ∥G(ℓ) ∥2 . E[∥G F fc fc F fc

∥bk ∥ = ∥(ρ − 1)gk ∥ = |ρ − 1| ∥gk ∥ = (1 − ρ)∥gk ∥,

(ℓ)

This quantity is non-negative. If ∥G(ℓ) ∥F ≤ Cfc m̄fc , then it (ℓ) 2 is at most ρ2fc Cfc (m̄fc )2 . Dividing by the positive noise-only (ℓ) moment 4d2 λ2fc (m̄fc )2 gives Eq. 8.

where the final equality uses ρ ∈ [0, 1]. The same algebra applies to the MLP-fc block after replacing gk by its corresponding block gradient and ρ by ρfc .

Training loss convergence GPT-2 (124 M)

6

AEGIS preserves optimisation dynamics GPT-2-XL (1.5 B)

OPT-1.3B 8

5

5

7

training loss

4

6

4

4 2

2

3 2

1

1 0

5

3

3

1

0 0

250

500

750

1000

1250

training step

1500

1750

2000

0

250

500

Phi-2 (2.7 B)

1000

1250

training step

1500

1750

2000

5 4

5

2.5

4

3

1.5

2

1.0

1

0.5 250

500

750

1000

1250

training step

0

250

500

750

1500

1750

2000

1000

1250

training step

1500

1750

2000

LLaMA-2-13B

3.0

2.0

0

0

LLaMA-2-7B

6

training loss

750

WikiText-2 Rotten Tomatoes DialogSum undefended AEGIS

3 2 1

0

250

500

750

1000

1250

training step

1500

1750

2000

0

0

1000

2000

3000

training step

4000

5000

6000

Fig. 6. Training loss convergence across six models. Same layout as Figure 5. AEGIS (solid) tracks the undefended loss trajectory (dashed), confirming that gradient uniformisation does not impede optimisation.

Remark 6 (Connection to convergence theory). Proposition 1 verifies the local bias/noise decomposition needed to apply biased-SGD convergence results such as [44, Thm. 2]. We do not rederive a global convergence-rate theorem here; the empirical training-loss and validation-PPL curves in Figures 5 and 6 test the resulting optimisation behaviour. Remark 7 (Noise scale and PPL overhead). The uniformisation noise is calibrated to the mean active-row norm m̄, so the effective noise floor scales with the embedding gradient’s contribution to the total gradient norm. For GPT-2 this is ≈ 31% of trainable parameters; for LLaMA-2-7B it is ≈ 2%, which is consistent with the lower relative utility cost at scale. Remark 8 (Utility–privacy trade-off). Higher λ or lower ρ increases privacy at the cost of more noise in the gradient update. At λ=1.0, ρ=0.3, AEGIS achieves ROUGE-1 ≤0.005 across all 11 models in the primary attack grid. Table II shows that full AEGIS keeps GPT-2 utility and improves the reported GPT-2-XL utility. B. Computational Complexity Attention freezing skips the backward pass for attention weights. Embedding and MLP flooding add only cheap elementwise work of order O(V d + Ld2 ). We time one full training step (forward, backward, defence, and optimiser) on a single H100. Each run uses sequence length 64, 20 warm-up steps, and 100 timed steps. On GPT-2, GPT-2-XL, and LLaMA-2-7B, the undefended step takes 19.35/73.24/97.09 ms. AEGIS takes 21.36/82.36/118.09 ms,

or 1.10/1.12/1.22× the baseline. Peak GPU memory drops from 2.24/24.14/50.51 GiB to 2.01/18.51/38.51 GiB. The dense gradient size drops from 0.464/5.803/12.551 GiB to 0.358/3.970/8.551 GiB, because frozen attention gradients are not sent. We use the same models and batch sizes for the baselines. Exact microbatch DP-SGD (C=1, σ=1) takes 7.53/2.31/3.72× the undefended step time and 1.35/1.49/1.49× the peak memory. Pruning takes 1.34/1.53/1.84× the step time. Soteria takes 1.07/1.05/1.02×. Payload sizes are the sizes of the gradient tensors. They do not include network transfer, packing, or compression time. C. Additional Experimental Details a) Computational requirements.: We implement AEGIS and run all attacks in PyTorch on a SLURM-managed GPU cluster. Each compute node hosts two high-memory GPUs (96 GB HBM). All decoder fine-tuning and attack runs were performed on a single GPU per cell. In practice, less demanding hardware suffices for the smaller checkpoints: GPT2 small/medium and BERT-base fit comfortably in 24 GB, whereas LLaMA-2-13B at the full BF16 precision used here requires 96 GB to accommodate weights (≈26 GB), gradients (≈26 GB), and optimiser state. System RAM consumption per experiment ranged from ≈16 GB (GPT-2 + WikiText-2 singlecell) to ≈96 GB (LLaMA-2-13B fine-tuning with checkpoint shard caching), matching the pattern reported by [5]. The full evaluation grid (11 models × 6 datasets × 2 defences + 54 baseline cells + 36 convergence traces + 16 ablation cells

+ qualitative and adaptive runs) consumed approximately 410 GPU-hours across ∼220 cluster jobs over a six-week window. b) Software stack.: PyTorch 2.5.1 (cu121 wheel), Transformers 4.46.3, Datasets 2.21.0, SafeTensors, NLTK, rougescore, sentencepiece, protobuf, scipy, scikit-learn, and bitsandbytes (used only for 8-bit AdamW on ≥13 B models so that the optimiser state fits in VRAM); see the released code repository for exact pinned versions. All model weights are downloaded from Hugging Face [45]; LLaMA and Gemma checkpoints require an authenticated HF_TOKEN. c) Per-cell runtime.: Approximate wall-clock time for a single (model, dataset, defence) cell on one H100 (fine-tune for ≤2,000 steps + DAGER attack on 10 sentences): GPT-2 (124 M) ≈3 min; GPT-2-medium / -large ≈4–6 min; GPT-2XL (1.5 B) ≈8 min; OPT-1.3 B ≈5 min; Phi-2 (2.7 B) ≈10 min; Gemma-2-2 B ≈10 min, Gemma-2-9 B ≈25 min; LLaMA-27 B / LLaMA-3.1-8 B ≈15 min; LLaMA-2-13 B ≈30 min. The baselines comparison adds ≈15 min/cell on average for the optimisation-based defences (DP-SGD, Soteria) due to persample gradient bookkeeping. The qualitative experiment runs all four attackers (DAGER, TAG, LAMP, GRAB) on three sentences for ≈15 min/dataset on GPT-2. d) Hyperparameter details (training).: All decoder finetuning uses AdamW [41] (β1 =0.9, β2 =0.999, weight decay 0.01) with learning rate 5 × 10−5 for models ≤8 B, and 1×10−5 with bitsandbytes’ AdamW8bit for LLaMA-2-13 B (FP32 optimiser state would otherwise exceed VRAM). GPT2 and GPT-2-XL use FP16 autocast with FP32 weights; LLaMA-family models use BF16 weights, gradient clipping at ∥g∥2 =1.0, linear warmup over the first 10 % of steps, and an early-step cap of 2,000 optimiser updates so that wall clock is balanced across model scales. Per-model batch sizes (MODEL_BATCH_SIZE in the released code): GPT2/OPT-1.3B = 8; GPT-2-medium/-large/Phi-2/LLaMA-7 B = 4; GPT-2-XL/Gemma-2-2 B/LLaMA-2-13 B = 2. Maximum sequence length is 64 tokens (the same setting used by [5] in their evaluation), padded to a multiple of 8 for tensor-core efficiency. A single fine-tuning seed (seed=42) is used per cell; seed-variance reporting is flagged in Limitation 1. e) Hyperparameter details (AEGIS defence).: The flood scale is λ=1.0 on the embedding gradient with retain ratio ρ=0.3 over the residual rows. The adaptive-threat MLP flood uses λmlp =8.0 and retain ratio ρmlp =0 on each twodimensional MLP weight gradient, including both the expansion and projection matrices; LayerNorm gradients are flooded with λln =4.0. For ≥8 B BF16 models we halve the MLP/LN flood scales (λmlp =2.0, λln =1.0) to stay within BF16’s ≈65 k dynamic range; without this scaling, gradient spikes occasionally overflow and corrupt the AdamW8bit state. We additionally skip optimiser steps whose total gradient norm is non-finite (an inf-gradient one-shot would otherwise poison the running Adam moments forever). f) Frozen parameter count.: In GPT-2 (124M total parameters, 12 blocks, d = 768): • Attention parameters (frozen by AEGIS): 12 × (768 × 2304 + 768 + 768 × 768 + 768) = 12 × (1,769,472 +

768 + 589,824 + 768) ≈ 28.3M parameters. Trainable parameters (MLP, LayerNorm, embeddings, LM head): ≈ 95.7M parameters. • Trainable fraction: ≈ 77.2%. •

D. Porting DAGER Beyond GPT-2/BERT The large-scale experiments use the official DAGER codebase [5], which ships with GPT-2 and BERT support only. We extended it with lightweight architecture-adaptation shims so that the otherwise-unmodified attack core runs against LLaMA-2/3, Gemma-2, OPT, Phi-2, RoBERTa, and DeBERTa-v3; the core analytical attack (per-block SVD, embedding row-norm scoring, L1 filtering, and beam search) is unchanged. a) Architecture dispatch.: Each checkpoint is assigned an architecture tag that the harness uses to resolve the correct embedding submodule path, special-token table (BOS/pad/EOS conventions differ across families), and beam search mode (causal decoder vs. masked encoder). b) Partial-forward patches.: DAGER’s span-residual stage requires truncated per-layer hidden states. We supply analogous partial-forward patches for each new architecture, accounting for family-specific residual stream conventions (RoPE, pre-normalisation variants, embedding scaling) while keeping the patch purely additive and non-intrusive to the training forward path. c) Span-filter thresholds.: The L1 residual threshold and SVD rank tolerance are set per architecture family based on network depth and positional encoding type; padding direction is likewise adjusted to match each model’s autoregressive search convention. d) Numerical precision.: Large models (LLaMA-2/3, Gemma-2) are loaded in BF16. Gradients are promoted to FP32 before SVD to match the precision of the official GPT-2 attack and avoid numerical artefacts at BF16 range limits. e) Validation.: For each newly supported architecture the attack is verified on an undefended checkpoint, confirming R-1 ≥ 0.95 before proceeding to the defended evaluation. Analogous adaptations for the optimisation-based attack (GRAB) and the adaptive MLP-SVD probe are smaller in scope, requiring only the architecture-specific MLP projection name and the same special-token table. E. Utility: Pre-Trained vs. AEGIS Fine-Tuning We report utility in separate tables for decoder-only causal LMs versus masked LMs: Tables IX and X (nine decoder checkpoints; classification vs. generative bands split for pagination) and Tables XI–XII (BERT-base and BERT-large). The decoder grids use two bands: classification-style supervision (Rotten Tomatoes, Emotion, Financial PhraseBank) and language-modelling / summarisation splits (WikiText-2, DialogSum, CNN/DailyMail). Each cell stacks four numbers (accuracy, F1, loss, perplexity). Results. Fine-tuning lifts classification accuracy/F1 far above the pre-training baseline on Rotten Tomatoes, Emotion, and Financial PhraseBank, while WikiText-2 / DialogSum /

TABLE IX D ECODER - ONLY UTILITY ( CLASSIFICATION - STYLE BENCHMARKS ; NINE MODELS ). P RE - TRAINED : FROZEN BASE WEIGHTS BEFORE TASK FINE - TUNING . O RIGINAL : UNDEFENDED FINE - TUNING . AEGIS: AEGIS- PROTECTED FINE - TUNING . E ACH CELL : ACC . / F1 / L OSS / PPL. G ENERATIVE SPLITS : TABLE X. Model

GPT-2

GPT-2medium GPT-2-large

GPT-2-XL

Gemma-2-2B

Gemma-2-9B

LLaMA-2-7B

LLaMA-213B† LLaMA-3.18B

Rotten Tomatoes

Acc F1 Loss PPL Acc F1 Loss PPL Acc F1 Loss PPL Acc F1 Loss PPL Acc F1 Loss PPL Acc F1 Loss PPL Acc F1 Loss PPL Acc F1 Loss PPL Acc F1 Loss PPL

Emotion

Financial PhraseBank

Pre-trained

Original

AEGIS

Pre-trained

Original

AEGIS

Pre-trained

Original

AEGIS

0.54 0.41 5.16 174.9 0.75 0.75 4.89 132.3 0.61 0.55 4.78 119.6 0.75 0.74 4.72 112.2 0.52 0.40 5.33 206.9 0.55 0.44 5.29 197.6 0.49 0.33 3.83 46.2 0.50 0.35 3.74 42.1 0.50 0.39 4.45 85.6

0.54 0.41 4.21 67.2 0.75 0.75 3.88 48.3 0.61 0.55 3.77 43.4 0.75 0.74 3.97 53.1 0.60 0.52 3.58 35.9 0.56 0.46 3.50 33.1 0.49 0.33 3.09 22.1 0.49 0.33 2.77 15.9 0.50 0.39 4.45 86.0

0.68 0.67 4.21 67.2 0.75 0.75 3.88 48.6 0.78 0.77 3.77 43.2 0.70 0.68 3.80 44.7 0.74 0.72 3.55 34.8 0.64 0.61 3.47 32.0 0.49 0.33 2.86 17.5 0.49 0.33 2.68 14.6 0.63 0.60 3.46 31.8

0.25 0.20 5.22 184.8 0.24 0.18 5.08 160.6 0.26 0.19 5.02 150.8 0.28 0.24 4.97 144.3 0.20 0.12 5.17 175.4 0.22 0.14 5.20 180.5 0.16 0.05 4.09 59.5 0.17 0.06 4.02 55.7 0.26 0.19 4.55 94.4

0.25 0.20 4.11 61.1 0.24 0.18 3.93 50.9 0.26 0.19 3.92 50.6 0.28 0.24 4.09 59.9 0.22 0.14 3.48 32.5 0.20 0.12 3.40 30.0 0.16 0.05 3.22 25.0 0.17 0.06 3.05 21.1 0.26 0.19 3.94 51.2

0.21 0.15 4.12 61.4 0.24 0.17 3.92 50.6 0.27 0.21 3.87 48.1 0.26 0.20 3.99 54.1 0.27 0.21 3.45 31.6 0.24 0.19 3.37 29.2 0.16 0.05 3.00 20.0 0.17 0.06 2.88 17.8 0.33 0.28 3.40 29.9

0.28 0.16 4.55 94.9 0.49 0.41 4.29 72.9 0.36 0.28 4.18 65.1 0.29 0.18 4.13 61.9 0.38 0.30 3.63 37.8 0.42 0.34 3.55 34.8 0.27 0.14 3.00 20.0 0.30 0.17 2.90 18.2 0.43 0.36 3.36 28.7

0.28 0.16 3.30 27.1 0.49 0.41 3.07 21.6 0.36 0.28 2.93 18.7 0.29 0.18 3.02 20.5 0.48 0.40 2.47 11.8 0.44 0.36 2.44 11.5 0.27 0.14 2.27 9.7 0.30 0.17 2.05 7.8 0.43 0.36 3.00 20.2

0.32 0.25 3.29 26.8 0.46 0.38 3.01 20.3 0.52 0.49 2.92 18.5 0.34 0.25 2.96 19.3 0.58 0.57 2.44 11.5 0.52 0.45 2.41 11.1 0.27 0.14 2.14 8.5 0.30 0.17 1.92 6.8 0.47 0.42 2.43 11.4

CNN/DailyMail show the expected collapse in loss and perplexity after adaptation. The AEGIS column demonstrates that the defence preserves—and in several cases improves—utility relative to undefended fine-tuning (Original), consistent with the stochastic regularisation effect discussed in Section VII. Relative to gradient noise or heavy pruning baselines discussed in Section II, AEGIS preserves these utility surfaces on the full evaluation grid while structurally masking the DAGER channels.

TABLE X D ECODER - ONLY UTILITY (W IKI T EXT-2, D IALOG S UM , CNN/DAILY M AIL ). S AME CELL CONVENTION AS TABLE IX. Model

GPT-2

GPT-2medium GPT-2-large

GPT-2-XL

Gemma-2-2B

Gemma-2-9B

LLaMA-2-7B

LLaMA-213B† LLaMA-3.18B

WikiText-2

Acc F1 Loss PPL Acc F1 Loss PPL Acc F1 Loss PPL Acc F1 Loss PPL Acc F1 Loss PPL Acc F1 Loss PPL Acc F1 Loss PPL Acc F1 Loss PPL Acc F1 Loss PPL

DialogSum

CNN/DailyMail

Pre-trained

Original

AEGIS

Pre-trained

Original

AEGIS

Pre-trained

Original

AEGIS

0.22 0.18 5.99 399.9 0.24 0.20 5.75 313.2 0.26 0.22 5.49 242.8 0.27 0.23 5.40 222.4 0.28 0.24 6.03 416.6 0.30 0.26 6.15 470.1 0.32 0.28 4.43 83.5 0.34 0.30 4.35 77.5 0.30 0.26 4.83 125.4

0.38 0.33 3.26 26.0 0.42 0.37 3.04 20.8 0.43 0.38 2.94 18.9 0.38 0.33 3.25 25.7 0.40 0.35 3.18 24.0 0.38 0.33 3.24 25.5 0.46 0.41 2.46 11.7 0.48 0.43 2.10 8.2 0.44 0.39 3.06 21.3

0.38 0.34 3.25 25.9 0.41 0.36 3.06 21.3 0.44 0.39 2.95 19.1 0.41 0.36 3.05 21.0 0.48 0.43 2.50 12.2 0.45 0.40 2.74 15.5 0.50 0.45 2.17 8.8 0.52 0.47 1.95 7.0 0.47 0.42 2.45 11.6

0.30 0.25 3.83 46.2 0.32 0.27 3.56 35.1 0.34 0.29 3.47 32.2 0.35 0.30 3.43 30.8 0.35 0.30 3.87 48.2 0.36 0.31 3.87 48.0 0.37 0.32 3.38 29.4 0.38 0.33 3.42 30.5 0.35 0.30 3.83 45.9

0.35 0.30 3.38 29.5 0.38 0.33 3.18 24.1 0.39 0.34 3.15 23.3 0.34 0.29 3.53 34.3 0.37 0.32 3.22 25.0 0.36 0.31 3.24 25.5 0.41 0.36 2.99 19.8 0.43 0.38 2.63 13.9 0.37 0.32 3.67 39.3

0.35 0.30 3.39 29.6 0.39 0.34 3.16 23.6 0.40 0.35 3.10 22.1 0.38 0.33 3.24 25.7 0.42 0.37 2.89 17.9 0.41 0.36 2.93 18.8 0.44 0.39 2.78 16.1 0.46 0.41 2.48 11.9 0.41 0.36 3.06 21.4

0.24 0.19 5.00 147.9 0.26 0.21 4.70 109.5 0.27 0.22 4.60 99.4 0.28 0.23 4.51 90.7 0.26 0.21 5.18 178.3 0.27 0.22 5.15 172.9 0.30 0.25 3.69 39.9 0.31 0.26 3.63 37.7 0.28 0.23 4.36 78.3

0.30 0.25 4.16 64.2 0.31 0.26 3.86 47.6 0.33 0.28 3.78 43.7 0.30 0.25 3.97 53.0 0.29 0.24 4.10 60.1 0.30 0.25 3.94 51.4 0.37 0.32 3.08 21.9 0.38 0.33 2.77 16.0 0.30 0.25 4.51 90.5

0.30 0.25 4.15 63.7 0.32 0.27 3.85 46.8 0.34 0.29 3.72 41.1 0.33 0.28 3.80 44.6 0.36 0.31 3.56 35.1 0.37 0.32 3.53 34.1 0.40 0.35 2.95 19.1 0.42 0.37 2.62 13.7 0.36 0.31 3.56 35.0

TABLE XI E NCODER (BERT) UTILITY — CLASSIFICATION DATASETS (ROTTEN T OMATOES , E MOTION , F INANCIAL P HRASE BANK ). G ENERATION DATASETS IN TABLE XII. E ACH CELL : ACC . / F1 / L OSS / PPL. Model

BERT-base

BERT-large

Rotten Tomatoes

Acc F1 Loss PPL Acc F1 Loss PPL

Emotion

Financial PhraseBank

Pre-trained

Original

AEGIS

Pre-trained

Original

AEGIS

Pre-trained

Original

AEGIS

0.50 0.33 4.05 57.2 0.50 0.33 3.99 54.2

0.50 0.33 2.33 10.2 0.50 0.33 2.16 8.7

0.54 0.38 2.20 9.0 0.58 0.46 1.67 5.3

0.17 0.03 5.20 182.1 0.17 0.03 5.24 189.2

0.17 0.03 2.29 9.8 0.17 0.03 2.38 10.8

0.20 0.06 2.14 8.5 0.22 0.08 1.99 7.3

0.33 0.11 3.52 33.7 0.33 0.11 3.32 27.5

0.33 0.11 2.01 7.5 0.33 0.11 1.95 7.0

0.40 0.22 1.66 5.3 0.38 0.18 1.90 6.7

TABLE XII E NCODER (BERT) UTILITY — GENERATION / SUMMARISATION DATASETS (W IKI T EXT-2, D IALOG S UM , CNN/DAILY M AIL ). E ACH CELL : ACC . / F1 / L OSS / PPL. Model

BERT-base

BERT-large

WikiText-2

Acc F1 Loss PPL Acc F1 Loss PPL

DialogSum

CNN/DailyMail

Pre-trained

Original

AEGIS

Pre-trained

Original

AEGIS

Pre-trained

Original

AEGIS

0.18 0.14 5.75 314.8 0.17 0.13 6.16 474.2

0.40 0.35 2.30 10.0 0.30 0.25 3.96 52.4

0.45 0.40 1.97 7.2 0.44 0.39 2.01 7.5

0.25 0.20 3.86 47.4 0.25 0.20 3.87 48.0

0.48 0.43 1.64 5.1 0.46 0.41 2.01 7.4

0.52 0.47 1.40 4.1 0.55 0.50 1.31 3.7

0.20 0.16 4.90 134.3 0.20 0.16 4.84 126.5

0.42 0.37 2.24 9.4 0.36 0.31 2.53 12.5

0.38 0.33 2.59 13.3 0.42 0.37 2.32 10.2

F. Qualitative Reconstruction Examples Table XIII shows word-level reconstruction outputs for GPT-2 small across three datasets and four attacks under three defence configurations. Green highlights indicate tokens recovered in the correct position; orange indicates correct tokens recovered at the wrong position.

∗

TABLE XIII Q UALITATIVE RECONSTRUCTION EXAMPLES ON GPT-2 SMALL ( SIX SENTENCES ACROSS THREE DATASETS , FOUR ATTACKS UNDER THREE DEFENCES ). E ACH LINE SHOWS THE FULL RECONSTRUCTED SENTENCE . G REEN INDICATES WORDS RECOVERED IN THE EXACT CORRECT POSITION , AND ORANGE INDICATES CORRECT WORDS IN THE WRONG POSITION . D IALOG S UM EXAMPLES CONTINUED IN TABLE XIV. Defence

Attack

Reconstructed Sequence

WikiText-2, Ex 1: Original: “M15 class monitors with 7 @.@ 5 in ( 190 mm ) guns : None

Prune-50%

AEGIS‡

DAGER GRAB SOMP FedSpy-LLM DAGER GRAB SOMP FedSpy-LLM DAGER GRAB SOMP FedSpy-LLM

M15 class monitors with 7 @.@ 5 in ( 190 mm ) guns HT15 class monitors monitors 7 @ in in 5 in ( 190 mm ) guns@ 15 class monitors with 7 @.@ 5 in ( 190 mm ) guns. mm 190 guns 5@ class monitors in 7 : ) with @ (1., M15 class monitors with 7 @.@ 5 ( 190 mm ) guns in 15 class 7 monitor @ 5 @ @ in ( 190 mm ) guns 7 15 class monitors with 7 @.@ 5 in ( 190 mm ) guns. mm 190 guns 5@ class monitors in 7 : ) with @ (1., aneCharacters129 TO concludingeln appraisalredits Brilliant woodscase refunds devaldamage neuronal lifes rest antzocumented 320antz spare decidedlydiffFastRather Compton absent decidedly conven Lethal spare 7’ (empty) BMI Stoke pulmonary Jeepbelt dwellingsanyariched despised starterarted Cheryl Races flavours Buddha hypers. . .

Rotten Tomatoes, Ex 1: Original: “sometimes seems less like storytelling than something the otherwise compelling director needed to get off his chest . None

Prune-50%

AEGIS‡

DAGER GRAB SOMP FedSpy-LLM DAGER GRAB SOMP FedSpy-LLM DAGER GRAB SOMP FedSpy-LLM

sometimes seems less like storytelling than something the otherwise compelling director needed to get off his chest sometimes seems less like storytelling than something the otherwise compelling director something compelling get off like storytelling seems seems less like storytelling than something the otherwise compelling director needed to get off his chest to a than off the otherwise compelling needed get his less director storytelling something . like chest to. sometimes seems less like storytelling than something the otherwise compelling director needed to get his chest seems seems compelling off otherwise otherwise compelling storytelling director compelling director seems compelling get off chest chest storytelling seems less like storytelling than something the otherwise compelling director needed to get off his chest to a than off the otherwise compelling needed get his less director storytelling something . like chest to. Meth sym west symbolism let—————————————————————- McN genius Hebophobia/// Yun disease Fn Investigative popupEGA Territories judgment seems storytelling highlight highlight movie give men run seems givemad away off off off off]( (empty) planetclud MOT Fant. . . OU chapter462 hysteria royal believe glyph. . . revival AwardsBonGuard Station

Rotten Tomatoes, Ex 2: Original: “what happened with pluto nash ? how did it ever get made ? None

Prune-50%

AEGIS‡

DAGER GRAB SOMP FedSpy-LLM DAGER GRAB SOMP FedSpy-LLM DAGER GRAB SOMP FedSpy-LLM

what happened with pluto nash ? how did it ever get made whiff whiff. pl. nash ? how how it ever get made ? happened with pluto nash ? how did it ever get made ? how made it ever didash get pluto ? with n? to happen what happened with pluto nash ? how did it get made =˜=˜ did plash nash ? How did it ever get ? it happened with pluto nash ? how did pluto nash ? how made it ever didash get pluto ? with n? to happen angrilyit?tarian ESC BitcoinstaboolaNazi wiped goodnessovereEls262rection waiter ——— ???????? just did n did ever? ? How happenedor pl gottenrations pl (empty) discreteplanesfortablebergerannappa Mesh ost Potter Obesity anchors DS805amer eas

Rotten Tomatoes, Ex 3: Original: “the actors are appealing , but elysian fields is idiotic and absurdly sentimental . None

Prune-50%

AEGIS‡

DAGER GRAB SOMP FedSpy-LLM DAGER GRAB SOMP FedSpy-LLM DAGER GRAB SOMP FedSpy-LLM

the actors are appealing , but elysian fields is idiotic and absurdly sentimental actors actors are appealing , but elysian fields are appealing , is absurd , sentimental , actors are appealing , but elysian fields is idiotic and absurdly sentimental , ,, fields .. but is sentimental e appealing absurdlyian idlys are and to the actors are appealing , but elysian fields is id sentimental actors actors is appealing ,irin elysian Field is idianlysian actors sentimental , actors are appealing , but elysian fields is idiotic and absurdly sentimental , ,, fields .. but is sentimental e appealing absurdlyian idlys are and to PROT retiredbps DISTRICT Lever.):681 unresolvedaryaurdyintuitive Genie virtually handlers respiratory orientationudingavanaugh k actorsloc appealing e k elys fields fields absurd idianhes absurdly sentimental id (empty) amplifier Indigenous dividends 610aks ensuredPetlarg hiding688 Xiaomi Dry Cock exciting Br bulletenges built

TABLE XIV Q UALITATIVE RECONSTRUCTION EXAMPLES (cont.) — D IALOG S UM DATASET. Defence

Attack

Reconstructed Sequence

DialogSum, Ex 1: Original: “Ah, who needs that anyway? I know all about women. None

Prune-50%

AEGIS‡

DAGER GRAB SOMP FedSpy-LLM DAGER GRAB SOMP FedSpy-LLM DAGER GRAB SOMP FedSpy-LLM

Ah, who needs that anyway? I know all about women ? that who needs that anyway ” I know all about women who , who needs that anyway? I know all about women, who I women that know all anyway needs. about??”’s Ah, who needs that anyway? I all know women know, who needs that if? I know all about women, , who needs that anyway? I know all about women, who I women that know all anyway needs. about??”’s longitudinal watches againstoul pinchanny Giuledged Stephanie shouldorage Texreply Missions? ”- ”-atic needs needsornorn women about women ALL (empty) aun utilized Chapters fragile albeitvertedlistedamura today butterfly Stro ClientPage

DialogSum, Ex 2: Original: “Don’t worry. See where it says, ’ New User ’? None

Prune-50%

AEGIS‡

DAGER GRAB SOMP FedSpy-LLM DAGER GRAB SOMP FedSpy-LLM DAGER GRAB SOMP FedSpy-LLM

Don’t worry. See where it says, ’ New User ’ ??,imer see New:,’,’, New::,’ ’t worry. See where it says, ’ New User ’. says? worry. ’ where New See User it, ” you I Don’t worry. See where it says, ’ New ’ New New says ’ see Where it says ’ ’ New User ’ ’ ’t worry. See where it says, ’ New User ’. says? worry. ’ where New See User it, ” you I flame Anna hammer dens kittensirk embarrassed GREAT owing nods 640 exported1988 Mell Pri Pri ’ meth ’stract edit edit. User New User User. (empty) 544ABCclinton FAT chast slur teamworkprice Houth Charlie reproFour PO medal

G. Datasets, Models, and Code Licenses 1) License Summary: In our work, we use the publicly available datasets WikiText-2, Rotten Tomatoes, Emotion, Financial PhraseBank, DialogSum, and CNN/DailyMail. WikiText-2 is licensed under the CC BY-SA 3.0 license. Emotion is licensed under CC BY-SA 4.0. Financial PhraseBank is licensed under CC BY-NC-SA 3.0 (non-commercial research use). DialogSum is licensed under CC BY-NC-SA 4.0. No public licensing information was found for Rotten Tomatoes; it is distributed by the original authors for research purposes. The CNN/DailyMail processing scripts are released under the Apache 2.0 license; the underlying news articles are used solely as research stimuli and are not redistributed. In terms of Large Language Model architectures, we use GPT-2 under the MIT license, BERT under the Apache License 2.0, and Phi-2 under the MIT license. RoBERTa and DeBERTa-v3 are released under the MIT license, and OPT1.3B is released under the OPT-175B non-commercial research license. We obtained access to LLaMA-2 and LLaMA-3.1 through the respective Meta Community License Agreements, which permit use in commercial and research settings subject to the applicable acceptable-use policies. Gemma-2 is available under the Gemma Terms of Use (gated; permits commercial and research use subject to the prohibited-use clause). All aforementioned licenses permit our use of the underlying assets for the purposes of this paper. We obtained the code for the DAGER and LAMP attacks through their public repositories, both licensed under the

Apache License 2.0. GRAB is released under the BSD 3Clause license. All permit academic research use. a) Released artefacts.: Our complete experimental code (training pipeline, attack hooks, AEGIS implementation, cluster job scripts, and result-aggregation notebooks) will be released under the MIT License at the camera-ready version.

Related documents

• Federated Learning with Privacy-preserving and Model IP-right-protection#177289
• FuseFSS: Efficient Secure LLM Inference with Function Secret Sharing#267554
• Component-Aware Differential Privacy for Federated Multilingual Speech-LLMs#999450
• LADSG: Label-Anonymized Distillation and Similar Gradient Substitution for Label Privacy in Vertical Federated Learning#139179
• IntraShuffler: A Privacy Preserving Framework for Heterogeneous DP Federated Learning#246414