From Prompts to Responses: Dual-Sided Data Leakage and Defense in Split Large Language Models
Zixuan Gu 1 Xiaojun Ye 1 Yang Liu 2
1. Introduction
Abstract
arXiv:2606.14210v1 [cs.CR] 12 Jun 2026
Large language models (LLMs) are increasingly deployed in privacy-sensitive domains, where users must balance the risk of data exposure through external APIs against the high computational cost of local deployment. Split learning has therefore emerged as a promising paradigm for LLM fine-tuning and inference under limited local resources. However, it introduces new privacy risks. Prior work primarily studies leakage of private input prompts, typically via inversion attacks on intermediate representations, while the potential for sensitive information leakage through generative response outputs remains largely unexplored.
With the advancement of Large Language Models (LLMs), their applications have expanded into a growing number of fields. Privacy-sensitive domain users, however, face limitations in using external LLM APIs, while fully private deployments incur substantial computational costs. Split learning has then emerged as a privacy-preserving LLM finetuning and inference paradigm under resource constraints, termed as Split-LLM (Gu et al., 2025; Thapa et al., 2022; Lin et al., 2024; Shen et al., 2023). In a typical Split-LLM, a full LLM model is divided into three slices, with the head and tail residing with private data holders and the body hosted on a cloud server. Despite this, such split settings remain vulnerable to input data leakage from the model head (Gu et al., 2025; Chen et al., 2024; Lin et al., 2024; Shen et al., 2023), as well as label leakage for classification tasks at the tail(Zhu et al., 2019; Li et al., 2022; Zou et al., 2022). While some recent works begin to examine the output response leakage (Liu et al., 2025; Fu et al., 2022) in open-ended generation, a systematic understanding of how information can leak jointly from both model ends during generation remains largely unexplored. This dual-ended leakage is particularly critical in high-stakes domains such as finance, where both input prompts and output responses contain highly sensitive information, such as a company’s investment strategy or proprietary insights.
In this work, we unveil novel vulnerabilities of Split-LLM by presenting Patched Model Inversion with Dual-Sided Initialization(PIDI), a two-stage attack that simultaneously targets both private input prompts and output responses in Split-LLM settings. It combines dual-sided initialization with a patched inversion strategy to tackle long sequences, substantially outperforming prior inversion methods. To counter threats from both sides, we further propose the Adapterbased DualGuard with Mutual Information Defense(ADMI), which integrates an adapter-based local warmup strategy and mutual information regularization to provide a strong empirical privacy protection with minimal impact on task performance. Extensive experiments across diverse tasks and models demonstrate that ADMI effectively defends against PIDI and other state-ofthe-art inversion attacks. Our code is publicly available at https://github.com/FLAIR -THU/VFLAIR-LLM.
In this work, we study data leakage of both input prompts and generated responses in Split-LLM generation. Our main contributions are: • We identify new vulnerabilities in Split-LLM and propose Patched Model Inversion with Dual Branch Initialization(PIDI), a novel dual-sided attack that reconstructs both input prompts and end responses during open-ended generation by exploiting information leakage from both model ends.
1
School of Software, Tsinghua University, Beijing, China 2 the Hong Kong Polytechnic University, Hongkong, China. Correspondence to: Yang Liu <[email protected]>.
• We propose Adapter-based DualGuard with Mutual Information Defense(ADMI) to mitigate threats from both ends, which simultaneously guard attacks from both ends and achieves a superior privacy-utility tradeoff compared to other baselines.
Proceedings of the 43 rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026. Copyright 2026 by the author(s).
1
From Prompts to Responses: Dual-Sided Data Leakage and Defense in Split Large Language Models
2. Related Works
and subsequently freezes the model head during full training, which may constrain the extent of model adaptation and limit its protection ability.
2.1. Split Learning of Large Language Models(Split-LLM)
Recently, VFLAIR-LLM (Gu et al., 2025) provides an opensource benchmark and research framework for systematically evaluating privacy risks and defenses in Split-LLM settings. Our work is implemented on top of this library.
To enable the private deployment of large language models (LLMs) under constrained local resources and data privacy, Split Learning(SL)(Gupta & Raskar, 2018; Thapa et al., 2022; Gu et al., 2025) has emerged as a promising solution. In the Split-LLM paradigm, a model is partitioned into multiple segments that are distributed between the client and the server, reducing the client’s local computational burden and enhancing privacy by transmitting only intermediate activations or gradients rather than raw data. The "HeadBody-Tail (HBT)" Split-LLM(Lin et al., 2024; Gu et al., 2025) splits the LLM backbone into 3 slices, aiming to protect both private inputs and private outputs by isolating both ends of the model from the server and preventing direct exposure of sensitive data.
3. Problem Definition Data Party 𝑴𝒉
Data Party 𝑴𝒉
𝑴𝒕
𝑴𝒕
Allocate
𝑴𝒉
𝑴𝒕
𝑴𝒃 Partition
𝑴𝒇𝒖𝒍𝒍
2.2. Data Attacks and Defenses in Split-LLM
Input 𝑿
Response 𝒀
𝑴𝒉
𝑴𝒕
Split-LLM Finetune 𝑯 𝑴𝒃
𝑴𝒃
𝑻
𝑴𝒇𝒖𝒍𝒍
Reconstructed Data
Model Party
Model Party
Split-LLM Finetuning
Inference-Time Reconstruction Attack
Figure 1. Dual Data Reconstruction Attack Threat Model
Although transmitting only intermediate activations helps obscure private data, prior work shows that reconstruction attacks can still infer sensitive inputs from smashed representations. For example, (Pan et al., 2020; Song & Raghunathan, 2020; Qu et al., 2025) perform direct smashed data matching, while (Balunović et al., 2022; Deng et al., 2021) leverage backward gradient matching during training. (Chen et al., 2024) further combines forward and backward matching in the training phase with a novel SIP initialization. Beyond input leakage, (Zhu et al., 2019; Zou et al., 2022; Li et al., 2022) show that classification labels can be inferred from transmitted gradients. More recently, (Fu et al., 2022; Liu et al., 2025) further expand to generation tasks and demonstrate that private responses can be approximated by applying a pre-trained model tail to the model body’s output activations.
The Split-LLM Setting. We consider the typical headbody-tail(HBT) Split-LLM scenario(Gu et al., 2025; Chen et al., 2024), where two parties, a Model Party and a Data Party, collaborate to fine-tune an LLM model. The Model Party (e.g., an LLM provider with substantial computational resources) controls a full pre-trained LLM, denoted as Mfull , which is partitioned into three distinct segments: (1) M̄h : The embedding layer fe and the initial nh decoders/encoders layers ; (2) M̄b : The intermediate nb layers, constituting the majority of the model’s parameters; (3) M̄t : The final nt decoder/encoder layers and the output projection lm head flm . The Data Party (i.e., a private data holder with limited local resources) is allocated the lightweight M̄h and M̄t slices. The largest segment, M̄b , remains on the Model Party’s server. For an input sequence of n tokens X = [a1 , a2 , . . . , an ] and its generated response of m tokens Y = [an+1 , an+2 , . . . , an+m ], the auto-regressive generation requires m collaborative Split-LLM forward passes at inference time. The system first processes the full input to produce the first output token an+1 . In subsequent passes, the latest generated token is passed as input to the Split-LLM system due to the Key-Value (KV) Cache mechanism. By aggregating the transmitted activations across all m passes, the Model party can obtain the model head’s output activations H = [h1 , h2 , . . . , hL ] ∈ RL×d and the model body’s output activations T = [t1 , t2 , . . . , tL ] ∈ RL×d , where d is the model’s hidden dimension and L = n + m − 1.
To mitigate these threats, prior work injects noise via differential privacy or sparsification (Du et al., 2023; Chatzikokolakis et al., 2013a; Mai et al., 2023; Aji & Heafield, 2017), applies token-wise MLDP-based text perturbation (Yue et al., 2021; Chen et al., 2023; Tong et al., 2024), or introduces regularization-based objectives such as mutual information minimization(Zou et al., 2023; Duan et al., 2023), adversarial training(Pan et al., 2020), and prototype clustering (Zhou et al., 2023) to encourage privacy-preserving representations. These defenses primarily operate on the model head to prevent input prompt leakage, while providing limited protection against attacks that exploit the model tail. More recently, DualGuard (Liu et al., 2025) further adopts a local warm-up paradigm to induce rapid parameter shifts in both the model head and tail, aiming to jointly mitigate leakage from both sides. However, its warm-up stage replaces the model body with a lightweight projection layer
Normally, the data party first employs Split-LLM finetuning to adapt a pre-trained model M̄h/b/t using its private data. The fine-tuned model Mh/b/t is then deployed via the Split-LLM inference pipeline to generate private responses 2
From Prompts to Responses: Dual-Sided Data Leakage and Defense in Split Large Language Models
for data party’s private input prompts.
to their distinct characteristics. Specifically, we perform response and input initializations as follows:
Threat Model In this work, we assume the model party is an honest-but-curious attacker, which complies with the Split-LLM protocols and does not collude with external parties. Given a private input query X = [a1 , a2 , . . . , an ] and the Split-LLM generated response Y = [an+1 , an+2 , . . . , an+m ], the adversary is assumed to have access to the following information for conducting attacks during Split-LLM inference:
• Response Initialization via Model Completion (MC): Since the response sequence is closely tied to the model’s generative patterns, we initialize it using Model Completion (MC)(Fu et al., 2022; Liu et al., 2025). This method leverages the ’not-too-far’ property, where the parameters of the fine-tuned Mt remain relatively close to those of the original pre-trained M̄t . The attacker can simply feed the model body’s output T into the original M̄t to estimate the private response as Ŷ = M̄t (T ) = [a0n+1 , a0n+2 , . . . , a0L ], where a0i denotes the initialized token at position i.
• The original model segments M̄h/b/t along with the finetuned body Mb . • The intermediate activations H output by the model head and the corresponding outputs of the model body T.
• Input Initialization via SIP: In contrast, the input sequence is weakly correlated with the model’s generative process and is primarily encoded through the computational behavior of Mh . Hence, we initialize it using Semi-white-box Forward Inversion Paradigm(SIP)(Chen et al., 2024), which exploits the semantic properties of the hidden states H and the known transformation H = Mh (X). Using a small auxiliary dataset (50 samples in our experiments), we train a SIP model MSIP to directly estimate the private input from H as X̂ = MSIP (H) = [a01 , a02 , . . . , a0n ].
• (Optional) A small amount of auxiliary data drawn from a distribution similar to that of the Data Party’s private dataset, which can be used for training relevant attack models like in (Chen et al., 2024; Fu et al., 2022). Leveraging this information, the Model Party seeks to reconstruct both the Data Party’s private inputs X and generated responses Y at inference time during autoregressive generation. This setting differs from training-time attacks like(Chen et al., 2024; Balunović et al., 2022), as only forward computation is performed and no gradients are available for exploitation.
To this end, the complementary strengths of MC and SIP enable us to construct a robust joint initialization E0 = [a01 , a02 , . . . , a0L ]. However, these estimations remain coarse, as reflected by their limited attack performance in Figure 2. On the response side, discrepancies between Mt and barMt , together with token sampling during generation, introduce systematic deviations from the true private responses. On the input side, prediction error of the SIP model further incurs wrong reconstruction. Consequently, the initialized sequence still needs further refinement. In the next stage, we further introduce a subsequent Patched Model Inversion (PMI) method to jointly optimize and refine these estimations.
4. PIDI: Attacking Split-LLM Through Both Ends By exploiting the auto-regressive nature of text generation, where generated tokens are iteratively appended to the input sequence, we design a two-stage attack to reconstruct private inputs and responses using both the transmitted hidden states H and T , named Patched Model Inversion with DualSided Initialization(PIDI). As described in Figure 2, in the first "Dual-Sided Initialization(DSI)", we construct an initial estimate of the private input and responses by jointly applying model completion and SIP technique on the input branch and the response branch, respectively. In the subsequent "Patched Model Inversion(PMI)" stage, we further refine this prior estimation via a novel patched model inversion training method to produce optimized embedding vectors, from which the final reconstructed sequence is extracted.
4.2. Patched Model Inversion(PMI) In this phase, we propose a patched model inversion strategy, extending the traditional model inversion method to better handle long text input sequences in transformer-based model scenarios. Initially, the embedding estimations E0 ∈ RL×d are fed into the original model head M̄h to produce dummy hidden states Ĥ ∈ RL×d . Leveraging the ’not-too-far’ property between M̄h and its fine-tuned counterpart Mh , the attacker refines E0 by minimizing the inversion loss:
4.1. Dual-Sided Initialization(DSI) In this stage, we first initialize our estimations of both input and response sequences through separate attacks, due
Linv = ∥Mh (E) − H∥ 3
E1 = arg min Linv E0
(1)
From Prompts to Responses: Dual-Sided Data Leakage and Defense in Split Large Language Models
𝑯 ∈ 𝑹𝑳×𝒅 𝑬𝟎
Split in Patches
𝒂𝟎𝟏 … 𝒂𝟎𝒏 + 𝒂𝟎𝒏+𝟏 … 𝒂𝟎𝑳
freeze
ignored
Patch Size 𝒍
Full Sequence Length 𝑳
𝑯𝒑𝟏
𝑯𝒑𝟐
…
𝑯𝒑𝑵
𝑬𝒑𝟏
𝑬𝒑𝟐
…
𝑬𝒑𝑵
𝑬𝒑𝟏
𝑬𝒑𝟐
…
𝑬𝒑𝑵
Patch Inversion on 𝑬𝒑𝟏
𝑬′𝒑𝟏
𝑬𝒑𝟐
…
𝑬𝒑𝑵
Patch Inversion on 𝑬𝒑𝟐
𝑯𝒑𝒌 ∈ 𝑹𝒍×𝒅
Patch Inversion on 𝑬𝒑𝒌
𝑳𝒊𝒏𝒗
𝟎 𝟎 𝒂𝒏+𝟏 … 𝒂𝒏+𝒎
𝑴𝑺𝑰𝑷
ഥ𝒕 𝑴
𝑯 ∈ 𝑹𝑳×𝒅
𝑻 ∈ 𝑹𝑳×𝒅
𝑬′𝒑𝟏
𝑬′𝒑𝟐
…
𝑬𝒑𝑵
Patch Inversion on 𝑬𝒑𝑵
Input SIP Initialization
Response MC Initialization
𝑬′𝒑𝟏
𝑬′𝒑𝟐
…
𝑬′𝒑𝑵
Full Inversion
…
…
𝒂𝟏𝟎 … 𝒂𝒏𝟎
Dual-Sided Initialization(DSI)
trainable
𝒑𝒌 𝑯
𝑯𝒑𝒌
ഥ𝒉 𝑴 𝑬′𝒑𝟏
…
𝑬′𝒑(𝒌−𝟏)
𝒂𝟏𝟏 … 𝒂𝒏𝟏 + 𝒂𝟏𝒏+𝟏 … 𝒂𝟏𝑳
𝑬𝒑𝒌
𝑬𝟏
Patched Model Inversion(PMI)
Figure 2. The Two-Stage Attack Methodology of PIDI
The resulting E1 is then decoded to recover the private token sequence. However, directly optimizing embeddings of length L becomes increasingly difficult as L grows.
representations and final performance; (2) since the method does not change the system structure, the large Mb can pull parameters back during full training, leaving Mt vulnerable and the "not-too-far" property largely preserved.
To mitigate this issue, we split the full embedding E0 ∈ RL×d into N non-overlapping patches of length l, denoted as [Ep1 , Ep2 , . . . , EpN ] where each patch Epk ∈ Rl×d . The corresponding model head output H can also be partitioned as: H = [Hp1 , Hp2 ...HpN ], Hpk ∈ Rl×d , Ĥ = [Ĥp1 , Ĥp2 , . . . ĤpN ], Ĥpk ∈ Rl×d . Due to the causal structure of decoder-only large language models, the prefix embeddings [Ep1 , Ep2 , . . . , Epk ] and the corresponding [Ĥp1 , Ĥp2 , . . . Ĥpk ] and [Hp1 , Hp2 , . . . , Hpk ] can be treated as valid input–output pairs for the inversion problem, as the hidden states at these positions are independent of subsequent tokens. As described in Figure 2, we first conduct inversion training iteratively over the patches. At iteration k, only Epk is optimized by minimizing ∥Ĥpk − Hpk ∥ while previous patches remain frozen. After individual refinement, all patches are unfrozen for a final joint inversion stage to improve overall refinement.
To address these issues, we further propose ADMI (Adapterbased DualGuard with Mutual Information Defense): • In Section 5.1, we introduce the two defense regularizers used in ADMI: mutual information regularizer LMI , which protects Mh by hindering input leakage from H; and model distance regularizer LD , which breaks the "not-too-far" property to defend Mt against model completion attacks. These regularizers define the core defensive objectives that guide parameter updates during training. These regularizers define the core defensive objectives guiding parameter updates. • In Section 5.2, we describe the two-stage training of ADMI. First, the Adapter-based Local Warm-up introduces a novel adapter module into the system, enabling rapid parameter shifts guided by LD while preserving performance. Second, the Full Training phase further adapts the model to downstream tasks, where both LMI and LD are applied to enforce dual end defense throughout the pipeline.
Results in Table 1 demonstrate that PIDI reconstructs private inputs and responses with high fidelity, posing a significant threat to both model ends. It highlights the urgent need for effective defenses. We now present our defense design, which mitigates such reconstruction attacks while preserving model utility.
5.1. Dual Defense Regularizers Apart from the main task loss LT , we design two defense regularizers, LMI and LD , to protect the model head and tail respectively.
5. ADMI: Defending LLMs From Private Data Leakage At Both Ends To protect both the model head and tail, DualGuard (Liu et al., 2025) first establishes a "Local Warm-up + Full Training" paradigm. In the Local Warm-up, Mh and Mt are jointly trained via a projection layer with defense regularizers, aiming to break the “not-too-far” property and hinder model completion attacks. Full Training then continues with Mh frozen. However, this approach has two limitations: (1) replacing Mb with a simple projection layer can degrade
5.1.1. M UTUAL I NFORMATION R EGULARIZER LMI To prevent H from leaking information about the input X, we aim to minimize the mutual information I(X, H), thereby limiting task-irrelevant input information retained in H. This increase the difficulty of reconstructing X from H, hindering both the inversion training in PMI and the SIP initialization in DSI. 4
From Prompts to Responses: Dual-Sided Data Leakage and Defense in Split Large Language Models Ⅰ. Local Warm-up Phase
Model Party
Data Party
𝑻
×
𝑯 ഥ𝒕 𝑴
ℒ𝐃
𝑻′
𝑴𝒕
ℒ 𝐌𝐈
𝑴𝑽𝑰𝑩
ℒ lw = ℒ T + 𝑎𝑑𝑎 𝛽 ℒD
× 𝑿
𝒁
𝑽𝑰𝑩𝒆
𝑯′
𝑽𝑰𝑩𝒅
~𝑵(𝝁, 𝝈)
Mutual Information Regularizer
𝑴𝒉
𝑯
𝑨𝒆
𝑨𝒆 (𝑯)
𝑨𝒅
𝒀
ℒ𝐓 𝑻
Ⅱ. Full Training Phase
Model Party
Data Party
𝑴𝒃
𝑯′
𝑴𝒃
𝑯′
𝑿 𝑴𝒉
𝑴𝑽𝑰𝑩 𝑯
𝑻
ഥ𝒕 𝑴
ℒ𝐌𝐈
𝑨𝒆
ℒ ft = ℒ T + 𝛼ℒ MI +𝑎𝑑𝑎 𝛽 ℒ D
𝑨𝒆 (𝑯)
𝑨𝒅
𝑴𝒕
𝒀
𝑴𝒉
𝒀
Forward
𝒀
𝓛𝐃
Model Distance Regularizer
𝓛𝐃
𝑻′
𝑴𝒕
ℒ𝐓
Backward
ℒ
Loss Term
𝑴
Finetuned Model Slice
𝑴
Defense Model
ഥ 𝑴
Pretrained Model Slice
Figure 3. Defense Pipeline of ADMI.
distance loss LD is defined as:
Directly minimizing I(X, H) is intractable due to the discrete and non-differentiable nature of the token ids X. We therefore operate on the continuous embedding representation E = fe (X). Since the embedding layer fe is typically fixed during fine-tuning, it forms a deterministic function. The Markov chain X → E → H holds, and we come to I(X, H) ≤ I(E, H) by the data processing inequality. The objective can thus be reformulated as minimizing I(E, H).
LD =
5.2. Two-Stage Defense Training of ADMI 5.2.1. A DAPTER - BASED L OCAL WARMUP Excluding the central Mb from the local warm-up phase may significantly disrupt model training. To address this limitation, we integrate an encoder–decoder Adapter module into the SL-LLM framework, as illustrated in Figure 3. The workflow and training dynamics are detailed below: • During forward propagation, the adapter encoder Ae first processes the original model head output H into Ae (H). Subsequently, Mb receives the model head output H and generates T . This representation T is then passed to the Adapter decoder Ad , along with Ae as memory to produce a refined representation T ′ . Finally, T ′ is fed into Mt to get the final result Y .
Following the derivation provided in Section A, the mutual information regularization term LMI is defined as a tractable upper bound on I(E, H ′ ) as in Equation (2): LMI = UpperBound(I(E, H ′ )) N
d
(3)
To integrate this regularizer into the overall training objective, we weight it by a coefficient β.
As the exact computation of I(E, H) is infeasible, we adopt a variational upper bound based on the Variational Information Bottleneck (VIB) framework(Alemi et al., 2016; Zou et al., 2023). As illustrated in Figure 2, we insert a encoderdecoder stochastic bottleneck model MVIB between Mh and Mb , whose encoder projects H to Gaussian parameters (µ, σ) and samples a latent variable via the reparameterization trick as Z = µ + σ ⊙ ϵ, ϵ ∼ N (0, I). The decoder then maps Z to the perturbed representation H ′ , which is finally fed to Mb .
1 X1X 2 2 2 µij + σij − log(σij )−1 = N i=1 2 j=1
1 CrossEntropy(Mt (T ), Mt′ (T ))
• During backward propagation, the Data Party does not transmit any gradients to the Model Party. As a result, Mb remains frozen and only perform forward calculation, while the gradients of the main task loss LT flow through Ad and Ae back to Mh , enabling its updates.
(2)
where N is the batch size and d is the model’s hidden dimension. The final training objective incorporates this term with a weighting coefficient α, yielding the final mutual information regularization term as α · LMI .
The Model Distance Regularizer LD is added to the main task loss LT , with an additional β controlling its weight. Therefore, the overall training objective during Local Warmup can be expressed as:
5.1.2. M ODEL D ISTANCE R EGULARIZER LD To defend model completion attacks targeting Mt , we adopt the distance regularizer LD proposed in (Liu et al., 2025), as illustrated in Figure 2. The objective is to maximize the output divergence between the fine-tuned model Mt′ and its pre-trained counterpart Mt , thereby breaking the "nottoo-far" property to hinder model completion. The model
Llw = LT + ada(β) ∗ LD (4) In this way, the adapter module provides an alternative pathway for gradient flow, allowing us to leverage Mb ’s comprehension ability while keeping it frozen, thereby preserving the local warm-up scheme’s security properties. 5
From Prompts to Responses: Dual-Sided Data Leakage and Defense in Split Large Language Models Table 1. Attack Performance(APα=0.5 ) of the evaluated attack methods. Full results of APinput and APresponse are placed in Tables 5 and 6. Llama3.2-3B
Fin Llama3-8B
Qwen2.5-7B
Llama3.2-3B
Med Llama3-8B
Qwen2.5-7B
Llama3.2-3B
Dolly Llama3-8B
Qwen2.5-7B
0.868 0.828 0.43 0.665 0.691 0.391 0.4 0.006
0.883 0.716 0.423 0.610 0.697 0.347 0.418 0.312
0.892 0.77 0.442 0.546 0.77 0.546 0.442 0.224
0.901 0.925 0.373 0.851 0.686 0.58 0.236 0.005
0.881 0.707 0.264 0.546 0.707 0.537 0.261 0.277
0.801 0.741 0.424 0.706 0.706 0.667 0.245 0.054
0.876 0.801 0.473 0.483 0.707 0.341 0.323 0.004
0.934 0.775 0.454 0.323 0.691 0.307 0.391 0.002
0.985 0.814 0.412 0.667 0.74 0.45 0.405 0.029
PIDI(DSI+PMI) DSI+VMI MC+VMI BiSR(SIP+VMI) DSI SIP MC VMI 1.0
PIDI(DSI+PMI)
DSI+VMI
BiSR(SIP+VMI)
1.0
1.0
0.9 0.8
0.8
0.8
0.6 0.7
AP
AP
AP
0.6
0.4
0.6
0.4
PIDI(DSI+PMI)
DSI+VMI
BiSR(SIP+VMI)
0.5 0.2 0.2 0.4
500
600
700
800
900
PIDI(DSI+PMI) 600
1000
700
DSI+VMI
800
900
BiSR(SIP+VMI) 1000
600
700
800
Sequence Length
Sequence Length
(a) Llama3.2-3B
900
1000
Sequence Length
(b) Qwen2.5-7B
(c) Llama3-8B
Figure 4. AP Across Varying Sequence Length (Evaluated on Dolly Dataset)
0.6
0.4
0.4 0.2 0.0 0.0
0.8
0.4
0.2 0.1
0.2
0.3
MP
0.4
0.5
0.0
(a) Llama3.2-3B
PIDI
w/o defense ADMI DualGuard MID DPForward SP SanText CusText
AP
0.6
AP
0.6
0.8
PIDI
w/o defense ADMI DualGuard MID DPForward SP SanText CusText
AP
0.8
PIDI
w/o defense ADMI DualGuard MID DPForward SP SanText CusText
0.2 0.1
0.2
0.3
MP
0.4
(b) Qwen2.5-7B
0.5
0.0
0.1
0.2
0.3
MP
0.4
0.5
(c) Llama3-8B
Figure 5. Defense Performance(MP-AP) for defending PIDI attack on the Fin Dataset. Dot size represents the defense strength. Here AP denotes APα=0.5 .
Considering the definition in Equation (3),LD can be excessively large at early stages when Mt (T ) is close to Mt′ (T ). Therefore, we adopt an adaptive weight strategy ada(β) to balance it with the main task loss LT . Specifically, we compute the ℓ2 norms of their gradients w.r.t. input embeddings and define: ada(β) =
min (∥∇E LD ∥2 , 0.1 · ∥∇E LT ∥2 ) ∥∇E LD ∥2 + ϵ
to flow through the whole system. An additional Variational Information Bottleneck MVIB is inserted between Mh and Mb as detailed in Section 5.1.1. LMI generated from MVIB together with LD is added to main task loss LT , forming the overall objective as: Lft = LT + αLMI + ada(β) ∗ LD (6) Here, LMI enhances the security of Mh by mutual information minimization, thereby enabling safe training of Mh . Meanwhile, LD protects Mt from model completion attacks by discouraging the model from reverting to its pretrained parameter space.
(5)
where ϵ is a small constant for numerical stability. This approach bounds the influence of LD , ensuring stable training and preserving utility.
From an information-preservation perspective, since MVIB at the model head removes potentially sensitive semantics from H, it may also inadvertently erase useful promptrelated information. The adapter can compensate for this information loss by encoding the unperturbed information of H and leveraging it during decoding to enrich the perturbed representation T into T ′ .
5.2.2. F ULL T RAINING WITH D EFENSE R EGULARIZERS After performing the aforementioned Local Warm-up Training, we then come to the second phase: Full Training. In this phase, Data Party and Model Party perform gradient communication, enabling gradients of the main task loss LT 6
From Prompts to Responses: Dual-Sided Data Leakage and Defense in Split Large Language Models Table 2. Defense Performance Against PIDI Across Fin, Med, and Dolly Datasets. Here, AP denotes APα=0.5 and DCS denotes DCSα=0.5 . Full results are reported in Tables 7 to 9.
6. Experiment Settings To comprehensively evaluate our proposed PIDI attack and ADMI defense strategy, we designed the following experiments as detailed below. Full parameter settings and configurations are provided in Section B.
Llama3.2-3B MP AP DCS
Defense
MP
Llama3-8B AP DCS
MP
Qwen2.5-7B AP DCS
Fin Dataset
Models and Dataset Settings We conduct our experiments across 3 LLMs: Llama3.2-3B,Llama3-8B, Qwen2.57B and 3 datasets: Dolly(Conover et al., 2023)(General QA&Instruction following), MedicalMeadow WikiDoc PatientInfo(Han et al., 2023) (Medical QA), and Financial-qa10k(virattt, 2024) (Finance QA). In the following sections, we will refer to MedicalMeadow WikiDoc PatientInfo as Med and Financial-qa-10k as Fin.
w/o defense ADMI DualGuard MID DPForward SP SanText CuxText
0.56 0.516 0.459 0.51 0.549 0.554 0.385 0.507
0.868 0.017 0.296 0.167 0.721 0.617 0.746 0.837
/ 0.968 0.819 0.89 0.662 0.696 0.648 0.628
w/o defense ADMI DualGuard MID DPForward SP SanText CuxText
0.176 0.173 0.147 0.169 0.161 0.162 0.141 0.185
0.901 0.074 0.312 0.138 0.68 0.538 0.796 0.841
/ 0.95 0.819 0.911 0.675 0.724 0.64 0.627
w/o defense ADMI DualGuard MID DPForward SP SanText CuxText
0.22 0.155 0.146 0.149 0.16 0.157 0.111 0.167
0.876 0.011 0.5 0.062 0.321 0.444 0.753 0.878
/ 0.956 0.737 0.937 0.813 0.759 0.65 0.617
0.527 0.515 0.51 0.497 0.533 0.527 0.431 0.498
0.883 0.014 0.446 0.023 0.85 0.649 0.805 0.849
/ 0.987 0.76 0.974 0.625 0.685 0.636 0.625
0.55 0.527 0.542 0.515 0.533 0.544 0.399 0.464
0.892 0.004 0.572 0.29 0.711 0.756 0.767 0.82
/ 0.984 0.712 0.829 0.665 0.652 0.644 0.632
/ 0.995 0.758 0.976 0.626 0.678 0.61 0.611
0.184 0.182 0.178 0.191 0.138 0.168 0.149 0.097
0.801 0.007 0.503 0.211 0.729 0.686 0.73 0.627
/ 0.995 0.738 0.87 0.659 0.673 0.659 0.691
/ 0.989 0.735 0.983 0.699 0.7 0.629 0.601
0.183 0.163 0.171 0.171 0.162 0.171 0.118 0.133
0.985 0.27 0.613 0.139 0.727 0.49 0.845 0.579
/ 0.839 0.698 0.91 0.66 0.743 0.625 0.709
Med Dataset
Attack Settings Apart from our proposed PIDI, our evaluated baseline methods include: BiSR(Chen et al., 2024) (SIP initialization + vanilla model inversion, VMI)1 , Model Completion (MC)(Fu et al., 2022; Liu et al., 2025), which leverages the target model’s pretrained head to complete generated sequences, and Vanilla Model Inversion (VMI)(Fredrikson et al., 2015). Additionally, we assess the following ablation variants: DSI+VMI (our DSI initialization with vanilla inversion), MC+VMI (model completion initialization with vanilla inversion), and DSI (initialization only). Results are summarized in Table 1.
0.187 0.205 0.144 0.165 0.194 0.162 0.105 0.166
0.881 0.007 0.449 0.027 0.845 0.672 0.9 0.9
Dolly Dataset 0.176 0.205 0.163 0.154 0.171 0.173 0.128 0.176
0.934 0.016 0.511 0.012 0.608 0.607 0.834 0.937
wal, 2007) score of the models answer to the gold answer. Attack Performance (AP). We measure the attacks using the BLEU(Papineni et al., 2002) score between the inferred text and the ground-truth private text, defined as Attack Performance (AP). Since private data includes both the prompt input and the model-generated response, we separately define APinput as the AP on the input sequence and APresponse as that on the response sequence. We aggregate them into an overall attack performance metric using a weighted formulation:
Defense Settings Apart from our proposed ADMI defense, we evaluate the following methods as baselines: (1) Learning-based Defenses design learning strategies or defense regularizers to yield robust activations against reconstruction: DualGuard(Liu et al., 2025) applies a local warm-up strategy with several adversarial defense regularizers to guard both the model head and tail. MID(Zou et al., 2023; Gu et al., 2025) uses mutual information-based regularizers to prevent data leakage from forward activations. (2) Perturbation-based Defenses inject noise into forward calculation to obscure private information: DPForward(Du et al., 2023) and Sparsification(SP)(Aji & Heafield, 2017; Gu et al., 2025) inject differential privacy (DP) noise or apply sparsification onto the transmitted hidden states. SanText(Yue et al., 2021) and CusText(Chen et al., 2023) employ token-wise perturbation based on an MLDP(Chatzikokolakis et al., 2013b) mechanism. Each defense is evaluated under multiple strength levels to present its overall privacy-utility trade-off as detailed in Section B.
APα = α · APinput + (1 − α) · APresponse
(7)
where α ∈ [0, 1] controls the relative importance between input and response privacy. Defense Capability Score (DCS). To comprehensively evaluate the privacy-utility trade-off of different defenses, we adopt the Defense Capability Score (DCS) proposed in (Gu et al., 2025). Based on APα , we define DCSα,β as shown in Equation (8). A higher DCS value indicates a more favorable balance between privacy protection and model utility. Using larger α indicates placing greater emphasis on protecting input privacy over response privacy, while larger values of β indicate a stronger preference for model utility over privacy. Unless otherwise specified, we set α = 0.5, β = 0.5 throughout this paper.
Metrics To comprehensively evaluate our attacks and defenses, we follow the evaluation method in (Gu et al., 2025). Main Task Performance(MP) evaluate the model performance, which is measured by the METEOR(Lavie & Agar-
DCSα,β =
1 Since gradient is not available in open-end generation task, the gradient matching module in the original BiSR is not implemented
7
1 p 1 + (1 − β)(APα − AP∗α )2 + β(MP − MP∗ )2 (8)
From Prompts to Responses: Dual-Sided Data Leakage and Defense in Split Large Language Models APinput
Attack Time
0.90
16000 14000
0.85
AP
18000
12000
0.80
10000
0.75
8000
0.70
6000
0.65 4000 0.60
0
50
100
150
200
Patch Length
Figure 6. Changing trend of AP and Attack time with respect to Patch Length l (Evaluated with Llama3.2-3B on Fin Dataset)
19
ADMI_5
1
1
1
1
1
2
ADMI_10
2
2
2
2
3
4
ADMI_50
3
3
3
3
4
3
ADMI_1
4
4
4
5
5
6
MID_1e-06
6
5
5
4
2
1
MID_0.0001
7
7
6
6
6
5
14
DualGuard_50
5
6
7
8
9
13
13
MID_1e-08
8
8
8
7
7
9
12
MID_0.001
9
9
9
9
8
7
11
MID_0.01
10
10
10
10
13
11
10
DualGuard_10
12
11
11
11
12
12
9
DualGuard_5
14
13
12
12
10
10
8
DualGuard_1
16
14
13
13
11
8
7
SP_98
17
16
14
14
14
14
DPForward_1.5
11
12
15
15
15
15
DPForward_2
13
15
16
16
17
17
SP_95
18
18
17
17
16
16
DPForward_1
15
17
18
18
18
18
DPForward_0.5
19
19
19
19
19
19
α = 0.1
α = 0.3
α = 0.5
α = 0.7
α = 0.9
α = 1.0
18 17 16 15
Ranking Position
APresponse
Defense Methods
0.95
APα = 0.5
Attack Time (s)
1.00
6 5 4 3 2 1
Figure 7. DCS ranking with Different α values (Evaluated with Llama3-3B on Fin Dataset)
7. Results and Analysis
parably to ADMI under the BiSR attack, which targets only the model head. However, they fail to maintain robust performance under PIDI and MC, which additionally attack from the model tail. This observation highlights that ADMI’s adapter-based design confers a pronounced advantage in protecting the model tail. In comparison to these learning-based defenses, perturbation-based methods generally exhibit weaker performance. SP achieves slightly better results than DPForward, whereas token-wise perturbation approaches, such as SanText and CusText, yield the weakest results. This indicates that, in open-ended generation, simply perturbing input tokens at the model head is insufficient for comprehensive protection, as such strategies cannot mitigate attacks targeting the model tail.
7.1. Attack Performance Analysis As shown in Table 1, our proposed PIDI attack generally achieves the highest AP across various tasks and datasets, demonstrating strong attack effectiveness. The DSI initialization in DSI+VMI leverages both input and response features, yielding superior performance over other initializations such as MC+VMI and BiSR (SIP+VMI). Furthermore, as shown in Figure 4, traditional VMI-based methods(e.g. DSI+VMI, BiSR) suffer significant AP degradation as sequence length increases, whereas our patched inversion technique PMI maintains stable and high AP across varying lengths while its marginal benefit diminishes for short sequences in smaller models These results validate the effectiveness of both our DSI and PMI designs in PIDI.
To evaluate the overall defense performance fairly, we compute DCSα,β (Equation (8)) with β = 0.5 and varying α values. Higher α indicates attaching more importance to input privacy versus the response privacy. The resulting ranks are visualized in Figures 7, 9 and 10, where lower numbers (bluer cells) indicate better performance. ADMI consistently ranks among the highest, confirming that it consistently provides favorable defense performance across various preferences.
We also present an ablation study on the impact of patch length l, as shown in Figure 6. As l increases, APinput degrades significantly, while APresponse remains largely stable. Meanwhile, a smaller l leads to substantially longer attack times required. Considering the trade-off between reconstruction quality and attack efficiency, we select l = 50 in our experiments. 7.2. Defense Performance Analysis
For completeness, we present further ablation studies on both attack and defense settings, including split configurations and auxiliary data size, along with additional experimental results in Section C.
Considering the MP-AP trend depicted in Figure 5 and Figure 8, the performance dots of our proposed ADMI defense lie significantly towards the lower-right region of the figure, indicating a superior MP–AP trade-off. Notably, the adaptive weighting strategy allows ADMI to increase its defense strength while incurring minimal MP degradation, demonstrating its effectiveness in balancing privacy protection and utility retention.
8. Conclusions and Limitations In this work, we investigate the dual-sided privacy risks inherent in Split-LLMs by introducing a novel attack, Patched Model Inversion with Dual-Sided Initialization (PIDI), addressing the privacy of both the input prompts and modelgenerated responses. By effectively exploiting information leakage from both model ends and leveraging a patched inversion strategy, PIDI achieves substantially stronger attack performance than existing methods. To counter this threat, we further propose the Adapter-based DualGuard
Results in Table 2 further demonstrate that ADMI achieves minimal MP loss while maintaining the lowest AP among all baselines across various models and tasks. Its optimal DCS score confirms its overall effectiveness, highlighting a favorable privacy–utility trade-off. As shown in Figure 8, DualGuard and MID perform com8
From Prompts to Responses: Dual-Sided Data Leakage and Defense in Split Large Language Models
with Mutual Information Defense(ADMI), a dual-sided defense framework integrated with a novel adapter-based strategy. This formulation provides robust protection for both model ends while incurring minimal utility loss, thereby enabling a more favorable privacy–utility trade-off.
M. Lamp: Extracting text from gradients with language model priors, 2022. URL https://arxiv.org/ab s/2202.08827. Chatzikokolakis, K., Andrés, M. E., Bordenabe, N. E., and Palamidessi, C. Broadening the scope of differential privacy using metrics. In De Cristofaro, E. and Wright, M. (eds.), Privacy Enhancing Technologies, pp. 82–102, Berlin, Heidelberg, 2013a. Springer Berlin Heidelberg. ISBN 978-3-642-39077-7.
Despite these promising results, our work has several limitations. ADMI introduces additional computational overhead, which may limit its applicability in latency-sensitive or large-scale deployment scenarios. Moreover, while PMI is particularly effective in long-sequence attack settings, its advantage becomes less pronounced on short texts or smaller models, and it may incur higher attack latency.
Chatzikokolakis, K., Andrés, M. E., Bordenabe, N. E., and Palamidessi, C. Broadening the scope of differential privacy using metrics. In De Cristofaro, E. and Wright, M. (eds.), Privacy Enhancing Technologies, pp. 82–102, Berlin, Heidelberg, 2013b. Springer Berlin Heidelberg. ISBN 978-3-642-39077-7.
Impact Statement This work advances the understanding of privacy risks and defenses in Split-LLM. By identifying and formalizing dualsided data leakage from both input prompts and generative responses, our findings can help practitioners design more robust systems and make informed deployment decisions.
Chen, G., Qin, Z., Yang, M., Zhou, Y., Fan, T., Du, T., and Xu, Z. Unveiling the vulnerability of private fine-tuning in split-based frameworks for large language models: A bidirectionally enhanced attack, 2024. URL https: //arxiv.org/abs/2409.00960.
At the same time, our proposed PIDI attack methodology may be misused to extract sensitive information from inadequately protected Split-LLM systems. We mitigate this risk by presenting an effective defense framework ADMI, which substantially reduces vulnerability while preserving model utility, and by releasing our code to support reproducibility and responsible evaluation of privacy-preserving techniques.
Chen, S., Mo, F., Wang, Y., Chen, C., Nie, J.-Y., Wang, C., and Cui, J. A customized text sanitization mechanism with differential privacy. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 5747–5758, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-acl .355. URL https://aclanthology.org/2023. findings-acl.355/.
We believe this work contributes positively by promoting transparency around emerging privacy threats and by providing practical mechanisms to strengthen safeguards for real-world Split-LLM deployments.
Conover, M., Hayes, M., Mathur, A., Xie, J., Wan, J., Shah, S., Ghodsi, A., Wendell, P., Zaharia, M., and Xin, R. Free dolly: Introducing the world’s first truly open instructiontuned llm, 2023. URL https://www.databricks .com/blog/2023/04/12/dolly-first-ope n-commercially-viable-instruction-tun ed-llm.
Acknowledgements This work was supported by the Presidential Young Scholar Scheme(Project No. P0056638), RIFL(Project No. 4CG00), and the RIAIoT(Project No. P0059914) at The Hong Kong Polytechnic University.
Deng, J., Wang, Y., Li, J., Shang, C., Liu, H., Rajasekaran, S., and Ding, C. Tag: Gradient attack on transformerbased language models, 2021. URL https://arxiv. org/abs/2103.06819.
References Aji, A. F. and Heafield, K. Sparse communication for distributed gradient descent. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 440–445, 2017.
Du, M., Yue, X., Chow, S. S. M., Wang, T., Huang, C., and Sun, H. Dp-forward: Fine-tuning and inference on language models with differential privacy in forward pass. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, CCS ’23, pp. 2665–2679. ACM, November 2023. doi: 10.1145/3576 915.3616592. URL http://dx.doi.org/10.11 45/3576915.3616592.
Alemi, A. A., Fischer, I., Dillon, J. V., and Murphy, K. Deep variational information bottleneck. CoRR, abs/1612.00410, 2016. URL http://arxiv.org/ abs/1612.00410. Balunović, M., Dimitrov, D. I., Jovanović, N., and Vechev, 9
From Prompts to Responses: Dual-Sided Data Leakage and Defense in Split Large Language Models
Duan, L., Sun, J., Chen, Y., and Gorlatova, M. Privascissors: Enhance the privacy of collaborative inference through the lens of mutual information, 2023. URL https: //arxiv.org/abs/2306.07973.
Learning Representations, 2022. URL https://open review.net/forum?id=cOtBRgsf2fO. Lin, Z., Hu, X., Zhang, Y., Chen, Z., Fang, Z., Chen, X., Li, A., Vepakomma, P., and Gao, Y. Splitlora: A split parameter-efficient fine-tuning framework for large language models, 2024. URL https://arxiv.org/ abs/2407.00952.
Fredrikson, M., Jha, S., and Ristenpart, T. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, CCS ’15, pp. 1322–1333, New York, NY, USA, 2015. Association for Computing Machinery. ISBN 9781450338325. doi: 10.1145/2810103.2813677. URL https://doi.org/10.1145/2810103.2813 677.
Liu, Z., Wang, Y., Wang, R., and Wu, S. Dualguard: A parameter space transformation approach for bidirectional defense in split-based llm fine-tuning. In Annual Meeting of the Association for Computational Linguistics, 2025. URL https://api.semanticscholar.org/ CorpusID:280018451.
Fu, C., Zhang, X., Ji, S., Chen, J., Wu, J., Guo, S., Zhou, J., Liu, A. X., and Wang, T. Label inference attacks against vertical federated learning. In 31st USENIX Security Symposium (USENIX Security 22), pp. 1397–1414, Boston, MA, August 2022. USENIX Association. ISBN 978-1939133-31-1. URL https://www.usenix.org/c onference/usenixsecurity22/presentat ion/fu-chong.
Mai, P., Yan, R., Huang, Z., Yang, Y., and Pang, Y. Splitand-denoise: Protect large language model inference with local differential privacy. CoRR, abs/2310.09130, 2023. doi: 10.48550/ARXIV.2310.09130. URL https: //doi.org/10.48550/arXiv.2310.09130. Pan, X., Zhang, M., Ji, S., and Yang, M. Privacy risks of general-purpose language models. In 2020 IEEE Symposium on Security and Privacy (SP), pp. 1314–1331, 2020. doi: 10.1109/SP40000.2020.00095.
Gu, Z., Fan, Q., Sun, L., Liu, Y., and Ye, X. Vflair-llm: A comprehensive framework and benchmark for split learning of llms. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2, KDD ’25, pp. 5470–5481, New York, NY, USA, 2025. Association for Computing Machinery. ISBN 9798400714542. doi: 10.1145/3711896.3737411. URL https://doi.org/10.1145/3711896.3737 411.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Isabelle, P., Charniak, E., and Lin, D. (eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311–318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 10.3115/1073083.1073135. URL https://aclanthology.org/P02-1040/.
Gupta, O. and Raskar, R. Distributed learning of deep neural network over multiple agents, 2018. URL https: //arxiv.org/abs/1810.06060.
Qu, W., Zhou, Y., Wu, Y., Xiao, T., Yuan, B., Li, Y., and Zhang, J. Prompt inversion attack against collaborative inference of large language models, 2025. URL https: //arxiv.org/abs/2503.09022.
Han, T., Adams, L. C., Papaioannou, J.-M., Grundmann, P., Oberhauser, T., Löser, A., Truhn, D., and Bressem, K. K. Medalpaca–an open-source collection of medical conversational ai models and training data. arXiv preprint arXiv:2304.08247, 2023.
Sellam, T., Das, D., and Parikh, A. BLEURT: Learning robust metrics for text generation. In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J. (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7881–7892, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl- main.704. URL https: //aclanthology.org/2020.acl-main.704/.
Lavie, A. and Agarwal, A. METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments. In Callison-Burch, C., Koehn, P., Fordyce, C. S., and Monz, C. (eds.), Proceedings of the Second Workshop on Statistical Machine Translation, pp. 228–231, Prague, Czech Republic, June 2007. Association for Computational Linguistics. URL https://aclanthology.org/W07-0734/.
Shen, X., Liu, Y., Liu, H., Hong, J., Duan, B., Huang, Z., Mao, Y., Wu, Y., and Wu, D. A split-and-privatize framework for large language model fine-tuning, 2023. URL https://arxiv.org/abs/2312.15603.
Li, O., Sun, J., Yang, X., Gao, W., Zhang, H., Xie, J., Smith, V., and Wang, C. Label leakage and protection in two-party split learning. In International Conference on
Song, C. and Raghunathan, A. Information leakage in embedding models. Proceedings of the 2020 ACM SIGSAC 10
From Prompts to Responses: Dual-Sided Data Leakage and Defense in Split Large Language Models
Conference on Computer and Communications Security, 2020. URL https://api.semanticscholar. org/CorpusID:214743021. Thapa, C., Chamikara, M. A. P., Camtepe, S., and Sun, L. Splitfed: When federated learning meets split learning, 2022. URL https://arxiv.org/abs/2004.1 2088. Tong, M., Chen, K., Zhang, J., Qi, Y., Zhang, W., Yu, N., Zhang, T., and Zhang, Z. Inferdpt: Privacy-preserving inference for black-box large language model, 2024. URL https://arxiv.org/abs/2310.12214. virattt. financial-qa-10k. https://github.com/v irattt/financial-datasets, 2024. Accessed: 2025-01-05. Yue, X., Du, M., Wang, T., Li, Y., Sun, H., and Chow, S. S. M. Differential privacy for text analytics via natural text sanitization. In Zong, C., Xia, F., Li, W., and Navigli, R. (eds.), Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 3853–3866, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.findings-acl.337. URL https: //aclanthology.org/2021.findings-acl .337/. Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y. Bertscore: Evaluating text generation with bert, 2020. URL https://arxiv.org/abs/1904.09675. Zhou, X., Lu, Y., Ma, R., Gui, T., Wang, Y., Ding, Y., Zhang, Y., Zhang, Q., and Huang, X. TextObfuscator: Making pre-trained language model a privacy protector via obfuscating word representations. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 5459–5473, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.f indings-acl.337. URL https://aclanthology.o rg/2023.findings-acl.337. Zhu, L., Liu, Z., and Han, S. Deep leakage from gradients. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: //proceedings.neurips.cc/paper_files /paper/2019/file/60a6c4002cc7b29142d ef8871531281a-Paper.pdf. Zou, T., Liu, Y., Kang, Y., Liu, W., He, Y., Yi, Z., Yang, Q., and Zhang, Y.-Q. Defending batch-level label inference and replacement attacks in vertical federated learning. IEEE Transactions on Big Data, 2022. 11
Zou, T., Liu, Y., and Zhang, Y.-Q. Mutual information regularization for vertical federated learning, 2023. URL https://arxiv.org/abs/2301.01142.
From Prompts to Responses: Dual-Sided Data Leakage and Defense in Split Large Language Models
A. Detailed Derivation of the Regularizer LMI We first show that the mutual information term I(E; H ′ ) establishes a theoretical upper bound on information leakage through intermediate activations. Since T is deterministic given H ′ , we have I(X; H ′ , T ) = I(X; H ′ ) + I(X; T |H ′ ) = I(X; H ′ ). Let X̂ = A(H ′ , T ) be any attacker-reconstructed input (deterministic or randomized). By data processing inequality, I(X; X̂) ≤ I(X; H ′ , T ) = I(X; H ′ ) ≤ LM I . Thus, for any attacker reconstructed result X̂ = A(H ′ , T ), I(X; X̂) ≤ I(X; H ′ , T ) ≤ LM I . However, directly optimizing I(E; H ′ ) is intractable. To obtain a tractable training objective, we introduce a variational information bottleneck (VIB) model MVIB between Mh and Mb . This VIB model acts as a stochastic bottleneck and it takes an encoder-decoder structure: its encoder projects H to Gaussian parameters (µ, σ) and a latent variable Z is sampled as Z = µ + σ ⊙ ϵ, ϵ ∼ N (0, I). The decoder then maps Z to a perturbed representation H ′ . Given the Markov chain E → H → Z → H ′ , the Data Processing Inequality ensures I(E, H ′ ) ≤ I(H, Z). I(E, H ′ ) ≤ I(E, Z) ≤ I(H, Z)
(9)
We can further derive the expected KL divergence between the variational posterior qϕ (Z|H) and a chosen prior p(z) = N (0, I) as a variational upper bound surrogate of I(H, Z): I(H; Z) = Ep(H) [KL (qϕ (z|H) ∥ qϕ (z))] = Ep(H) [KL (qϕ (Z|H) ∥ p(Z))] − KL (qϕ (Z) ∥ p(Z))] ≤ Ep(H) [KL (qϕ (Z|H) ∥ p(Z))]
(10)
R where ϕ denotes the parameter of the VIB encoder and qϕ (z) = qϕ (z|H)p(H)dH is the marginal distribution of Z, also known as the aggregated posterior. Therefore we can further derive the upper bound of I(E, H ′ ) as: LMI = UpperBound(I(E, H ′ )) = EH∼p(H) [KL (qϕ (z|H) ∥ p(z))] N
≈
=
1 X KL N (µi , σ 2i ) ∥ N (0, I) N i=1 N X
(11)
d X
1 1 2 2 µ2ij + σij − log(σij )−1 N i=1 2 j=1
B. Detailed Experiment Settings Model and Dataset Configurations For all models, we set the partition settings as nh = 4 and nt = 4, leaving nb = 20, 20, 24 for Llama3.2-3B/Qwen2.5-7B/Llama3-8B, respectively. The Dolly(Conover et al., 2023) dataset is an open-source collection of instruction-following records, spanning multiple behavioral categories. The MedicalMeadow WikiDoc PatientInfo dataset(Med) is the Wikidoc Patient Information subset of the training set in (Han et al., 2023). It is a medical QA dataset with medical question-answer pairs extracted from WikiDoc, a collaborative platform for medical professionals to share and contribute to up-to-date medical knowledge. The financial-QA-10k dataset(Fin)(virattt, 2024) is a collection of 10,000 question-answer pairs derived from corporate 10-K filings, covering diverse topics in financial analysis, company operations, and corporate strategy. For all datasets, we split 5% of the full set as the test set and the remaining part as train set. Training Configurations We set learning rate to 2e-4 and batch size to 8 throughout the work. Convergence is marked with an early-stop strategy. Notably, the learning rate schedule is critical for the training stability of our ADMI defense as detailed in Table 4. During local warm-up stage, the randomly initialized adapters are trained with a larger learning rate to quickly align with the pretrained backbone, while Mh/t uses a smaller learning rate to induce a smooth parameter shift and prevent drastic representation drift. During full training, we increase the backbone learning rate to enable deeper task adaptation, and train the newly introduced MVIB with a larger learning rate for fast convergence and effective integration. Defense Settings In Figures 5 and 8, we evaluate each defense under a range of increasing defense strengths. The corresponding controlled hyper-parameter values, ordered from weakest to strongest, are summarized in Table 3. Parameters 12
From Prompts to Responses: Dual-Sided Data Leakage and Defense in Split Large Language Models Table 3. Summary of defenses hyper-parameters. Defense
Controlled Hyper-parameter Values
ADMI DualGuard MID TO DPForward SP SanText CusText
β = 1, 5, 10, 50 λ3 = 1, 5, 10, 50 λ = 10−6 , 10−4 , 10−3 , 0.01 ϵ = 10, 5, 2, 1 noise scale= 0.5, 1, 1.5, 2 r = 90.0%, 95.0%, 98.0%, 99.5% p = 0.02, 0.05, 0.08, 0.1 ϵ = 150, 100, 50, 30
Table 4. Learning rate configurations for different model components in PIDI. Model
Learning Rate
Stage Mh
Mt
Mh/t Ae /Ad MVIB
Llama3.2-3B
Local-Warmup 2e-5 / 2e-5 Full Training 2e-4 2e-4 2e-4
4e-4 2e-4
/ 4e-4
Qwen2.5-7B
Local-Warmup 2e-5 / 2e-4 Full Training 2e-4 4e-4 2e-4
2e-4 2e-4
/ 4e-4
Llama3-8B
Local-Warmup 2e-5 / 2e-4 Full Training 2e-4 4e-4 2e-4
2e-4 2e-4
/ 4e-4
used in Tables 2 and 7 to 9 are highlighted in bold. For ADMI, defense strength is controlled via the model distance regularizer weight β. In DualGuard, we fix λ1 = 30 and λ2 = 70, and vary λ3 (the anti-pretrained-tail loss weight, analogous to ADMI’s model distance regularizer) to adjust defense strength. MID uses the MI regularizer weight λ to tune defense strength. TO sets waway = 0.5, wclose = 0.1, and ncluster = 100, and varies ϵ to control defense strength. DPForward adjusts the noise scale injected into the embedding space. SP controls defense via the sparsification rate r, i.e., the fraction of sparsified tensor coordinates. SanText adjusts the fraction p of perturbed sensitive words sanitized, while CusText fixes top-k = 100 and varies ϵ for DP noise. For both ADMI and MID, the inserted MVIB encoder and decoder use a single-layer MLP with ReLU. For both ADMI and DualGuard, we set the local warmup epoch to 2.
Attack Settings We randomly select 200 samples from the test dataset to evaluate attack performance, on which the Split-LLM system performs open-end generation. For all vanilla model inversion training in our experiments (BiSR/VMI), we employ an early-stop strategy to determine convergence, with a maximum of 1000 inversion steps. For PIDI, each patched inversion also uses early stopping with a smaller maximum of 300 steps, while the final full inversion is limited to 500 steps. The inversion learning rate is set to 0.001 for all methods. The auxiliary dataset for SIP training consists of 50 samples randomly drawn from the training set. Table 5. Attack Performance(APinput ) of the evaluated attack methods
PIDI(DSI+PMI) DSI+VMI MC+VMI BiSR(SIP+VMI) DSI SIP MC VMI
Llama3.2-3B
Fin Llama3-8B
Qwen2.5-7B
Llama3.2-3B
Med Llama3-8B
Qwen2.5-7B
Llama3.2-3B
Dolly Llama3-8B
Qwen2.5-7B
0.844 0.753 0 0.75 0.582 0.582 0 0.005
0.873 0.579 0.009 0.726 0.557 0.557 0 0.27
0.877 0.678 0.022 0.678 0.678 0.678 0.022 0.035
0.951 0.94 0 0.937 0.902 0.902 0 0.005
0.931 0.893 0.004 0.893 0.893 0.893 0 0.214
0.945 0.93 0 0.93 0.921 0.921 0 0.004
0.857 0.699 0.002 0.673 0.622 0.623 0 0.006
0.953 0.67 0.083 0.603 0.6 0.6 0 0.002
0.996 0.757 0.005 0.789 0.673 0.673 0.004 0.002
Table 6. Attack Performance(APresponse ) of the evaluated attack methods
PIDI(DSI+PMI) DSI+VMI MC+VMI BiSR(SIP+VMI) DSI SIP MC VMI
Llama3.2-3B
Fin Llama3-8B
Qwen2.5-7B
Llama3.2-3B
Med Llama3-8B
Qwen2.5-7B
Llama3.2-3B
Dolly Llama3-8B
Qwen2.5-7B
0.891 0.904 0.86 0.58 0.8 0.199 0.8 0.008
0.893 0.854 0.836 0.495 0.836 0.138 0.836 0.354
0.907 0.863 0.863 0.414 0.863 0.414 0.863 0.414
0.851 0.911 0.746 0.764 0.471 0.259 0.471 0.006
0.831 0.522 0.525 0.198 0.522 0.181 0.522 0.339
0.658 0.551 0.847 0.482 0.49 0.412 0.49 0.103
0.895 0.903 0.944 0.292 0.792 0.06 0.646 0.003
0.916 0.881 0.825 0.043 0.783 0.014 0.783 0.002
0.975 0.87 0.819 0.544 0.807 0.227 0.807 0.055
13
From Prompts to Responses: Dual-Sided Data Leakage and Defense in Split Large Language Models BiSR
0.5 0.4
AP
0.3 0.2
0.1
0.1
0.0
0.5
0.4 0.3
0.1
0.2
0.3
MP
0.4
0.0
0.5
MC
w/o defense ADMI DualGuard MID DPForward SP SanText CusText
0.4 0.3
0.2
0.3
MP
0.4
0.0
0.5
0.2
0.3
MP
0.4
0.5
0.4
0.5
MC
w/o defense ADMI DualGuard MID DPForward SP SanText CusText
0.1
0.1 0.1
0.1
0.2
0.2
0.1
BiSR
w/o defense ADMI DualGuard MID DPForward SP SanText CusText
AP
MP
0.4
0.2
0.0 0.0
0.4
AP 0.3
AP
0.3
0.5
0.2
MC
w/o defense ADMI DualGuard MID DPForward SP SanText CusText
0.6
AP
0.4
BiSR
w/o defense ADMI DualGuard MID DPForward SP SanText CusText
0.3
AP
defense 0.8 w/o ADMI DualGuard 0.7 MID DPForward SP 0.6 SanText CusText 0.5 0.4 0.3 0.2 0.1 0.0 0.0 0.1 0.2
0.1
(a) Llama3.2-3B
0.2
0.3
MP
0.4
0.0
0.5
(b) Qwen2.5-7B
0.1
0.2
0.3
MP
(c) Llama3-8B
Figure 8. Defense Performance(MP-AP) for defending BiSR/MC attack on the Fin Dataset. Dot size represents the defense strength. Here AP denotes APα=0.5 .
Table 7. Full Defense Performance Against PIDI attack Across Fin, Med, and Dolly Datasets. Defense
MP
Llama3.2-3B APresponse APinput
Llama3-8B APresponse APinput
MP
MP
Qwen2.5-7B APresponse APinput
Fin Dataset w/o defense ADMI DualGuard MID DPForward SP SanText CuxText
0.56 0.516 0.459 0.51 0.549 0.554 0.385 0.507
0.891 0.009 0.592 0.308 0.78 0.763 0.693 0.832
0.844 0.025 0 0.026 0.662 0.471 0.8 0.842
0.527 0.515 0.51 0.497 0.533 0.527 0.431 0.498
w/o defense ADMI DualGuard MID DPForward SP SanText CuxText
0.176 0.173 0.147 0.169 0.161 0.162 0.141 0.185
0.851 0.124 0.623 0.229 0.506 0.401 0.695 0.746
0.951 0.024 0.001 0.046 0.854 0.676 0.896 0.935
0.187 0.205 0.144 0.165 0.194 0.162 0.105 0.166
w/o defense ADMI DualGuard MID DPForward SP SanText CuxText
0.22 0.155 0.146 0.149 0.16 0.157 0.111 0.167
0.895 0.002 0.986 0.099 0.256 0.46 0.728 0.874
0.857 0.02 0.013 0.026 0.385 0.429 0.779 0.882
0.893 0.017 0.86 0.033 0.888 0.748 0.754 0.833
0.873 0.012 0.032 0.013 0.812 0.55 0.856 0.864
0.55 0.527 0.542 0.515 0.533 0.544 0.399 0.464
0.907 0.003 0.906 0.571 0.855 0.904 0.705 0.772
0.877 0.005 0.238 0.01 0.568 0.607 0.83 0.867
0.931 0.007 0.3 0.011 0.846 0.69 0.91 0.958
0.184 0.182 0.178 0.191 0.138 0.168 0.149 0.097
0.658 0.001 0.873 0.411 0.585 0.572 0.562 0.302
0.945 0.012 0.133 0.011 0.874 0.8 0.897 0.952
0.953 0.016 0.036 0.012 0.565 0.583 0.905 0.97
0.183 0.163 0.171 0.171 0.162 0.171 0.118 0.133
0.975 0.489 0.938 0.272 0.857 0.662 0.761 0.33
0.996 0.051 0.288 0.005 0.598 0.318 0.928 0.827
Med Dataset 0.831 0.006 0.598 0.044 0.844 0.654 0.891 0.841
Dolly Dataset 0.176 0.205 0.163 0.154 0.171 0.173 0.128 0.176
14
0.916 0.015 0.986 0.011 0.651 0.63 0.763 0.904
From Prompts to Responses: Dual-Sided Data Leakage and Defense in Split Large Language Models
Table 8. Full Defense Performance Against BiSR attack Across Fin, Med, and Dolly Datasets. Defense
MP
Llama3.2-3B APresponse APinput
Llama3-8B APresponse APinput
MP
MP
Qwen2.5-7B APresponse APinput
Fin Dataset w/o defense ADMI DualGuard MID DPForward SP SanText CuxText
0.56 0.516 0.459 0.51 0.549 0.554 0.385 0.507
0.58 0.036 0 0.038 0.489 0.166 0.49 0.56
0.75 0.042 0 0.036 0.68 0.462 0.73 0.781
0.527 0.515 0.51 0.497 0.533 0.527 0.431 0.498
w/o defense ADMI DualGuard MID DPForward SP SanText CuxText
0.176 0.173 0.147 0.169 0.161 0.162 0.141 0.185
0.764 0.023 0 0.02 0.314 0.139 0.706 0.846
0.937 0.074 0 0.047 0.865 0.675 0.911 0.947
0.187 0.205 0.144 0.165 0.194 0.162 0.105 0.166
w/o defense ADMI DualGuard MID DPForward SP SanText CuxText
0.22 0.155 0.146 0.149 0.16 0.157 0.111 0.167
0.292 0.015 0 0.015 0.129 0.032 0.22 0.337
0.673 0.036 0 0.029 0.608 0.441 0.598 0.663
0.495 0.035 0.002 0.048 0.508 0.191 0.25 0.315
0.726 0.02 0 0.028 0.643 0.43 0.556 0.64
0.55 0.527 0.542 0.515 0.533 0.544 0.399 0.464
0.414 0.002 0 0.012 0.413 0.304 0.316 0.382
0.678 0.005 0 0.012 0.568 0.502 0.599 0.701
0.198 0.011 0.004 0.026 0.656 0.204 0.248 0.614
0.893 0.001 0 0.023 0.858 0.801 0.785 0.906
0.184 0.182 0.178 0.191 0.138 0.168 0.149 0.097
0.482 0.013 0 0.012 0.364 0.402 0.464 0.361
0.93 0.011 0 0.012 0.876 0.799 0.854 0.949
Dolly Dataset 0.176 0.043 0.205 0.038 0.163 0 0.154 0.01 0.171 0.053 0.173 0.013 0.128 0.046 0.176 0.146
0.603 0.017 0 0.024 0.601 0.497 0.548 0.608
0.183 0.163 0.171 0.171 0.162 0.171 0.118 0.133
0.544 0.017 0 0 0.341 0.146 0.487 0.214
0.789 0.052 0 0 0.648 0.328 0.777 0.786
Med Dataset
Table 9. Full Defense Performance Against Model Completion(MC) Attack Across Fin, Med, and Dolly Datasets. Defense
MP
Llama3.2-3B APresponse APinput
Llama3-8B APresponse APinput
MP
MP
Qwen2.5-7B APresponse APinput
Fin Dataset w/o defense ADMI DualGuard MID DPForward SP SanText CuxText
0.56 0.516 0.459 0.51 0.549 0.554 0.385 0.507
0.8 0 0.589 0.785 0.84 0.8 0.615 0.75
0 0 0 0 0 0 0.003 0.032
0.527 0.515 0.51 0.497 0.533 0.527 0.431 0.498
0.836 0 0.839 0.8 0.844 0.808 0.686 0.754
0 0 0.003 0 0 0 0 0.001
0.55 0.527 0.542 0.515 0.533 0.544 0.399 0.464
0.863 0 0.836 0.871 0.846 0.888 0.649 0.728
0.022 0 0 0 0 0 0.026 0.037
0 0 0.008 0.006 0 0 0.109 0
0.184 0.182 0.178 0.191 0.138 0.168 0.149 0.097
0.49 0 0.485 0.561 0.423 0.462 0.39 0.249
0 0 0.007 0 0 0 0.002 0
0 0 0.007 0.004 0 0 0 0
0.183 0.163 0.171 0.171 0.162 0.171 0.118 0.133
0.807 0.757 0.81 0.733 0.844 0.82 0.632 0.308
0.004 0 0.008 0 0.002 0.021 0.003 0.062
Med Dataset w/o defense ADMI DualGuard MID DPForward SP SanText CuxText
0.176 0.173 0.147 0.169 0.161 0.162 0.141 0.185
0.471 0.348 0.37 0.376 0.415 0.399 0.416 0.441
0 0.004 0.002 0 0 0 0.001 0.001
w/o defense ADMI DualGuard MID DPForward SP SanText CuxText
0.22 0.155 0.146 0.149 0.16 0.157 0.111 0.167
0.646 0.001 0.805 0.726 0.814 0.767 0.652 0.754
0 0.005 0.01 0 0.001 0.003 0.003 0.042
0.187 0.205 0.144 0.165 0.194 0.162 0.105 0.166
0.522 0 0.317 0.431 0.528 0.474 0.758 0.514
Dolly Dataset 0.176 0.205 0.163 0.154 0.171 0.173 0.128 0.176
15
0.783 0 0.758 0.656 0.704 0.779 0.657 0.768
From Prompts to Responses: Dual-Sided Data Leakage and Defense in Split Large Language Models 1
1
1
1
1
5
ADMI_10
2
2
2
2
2
6
ADMI_1
3
3
3
3
3
4
ADMI_5
4
4
4
4
4
7
MID_0.01
6
5
5
6
5
6
6
6
1
1
1
1
1
3
ADMI_1
2
2
2
2
2
4
ADMI_50
3
3
3
3
3
5
MID_0.001
4
4
4
4
5
7
MID_0.0001
5
5
5
5
4
6
8
15
DualGuard_5
10
7
6
6
6
1
13
DualGuard_1
12
9
7
7
7
2
12
DPForward_2
6
6
8
8
8
8
11
SanText_0.08
8
10
9
9
9
10
10
CuxText_30
7
8
10
12
13
15
9
SanText_0.05
11
12
11
10
10
9
8 7
18 17 16
11
5
18
ADMI_10
19
8
7
7
7
10
10
MID_0.0001
11
9
8
8
11
9
13
DualGuard_10
14
11
9
9
7
3
12
DualGuard_1
15
12
10
10
8
1
11
DualGuard_5
16
13
11
11
9
2
10
DualGuard_50
7
8
12
13
13
13
9
DPForward_2
17
16
13
12
12
12
8
CuxText_50
9
11
12
13
12
12
SanText_0.1
9
10
14
14
14
15
7
13
13
11
11
11
SanText_0.05
SanText_0.02
13
12
15
15
15
16
17
6
14
14
14
14
13
10
14
16
19
20
20
CuxText_100
14
CuxText_50 DPForward_1
19
18
17
16
15
14
CuxText_150
15
15
15
15
15
14
CuxText_100
13
17
18
20
19
19
DPForward_1
16
16
16
16
16
18
DPForward_0.5
18
19
19
18
18
18
DPForward_0.8
17
17
17
17
17
16
DPForward_0.8
20
20
20
17
17
16
DPForward_0.5
17
17
17
17
17
16
α = 0.1
α = 0.3
α = 0.5
α = 0.7
α = 0.9
α = 1.0
α = 0.1
α = 0.3
α = 0.5
α = 0.7
α = 0.9
α = 1.0
Defense Methods
MID_0.001
14
Ranking Position
Defense Methods
MID_1e-06
5
20
5 4 3 2 1
Figure 9. DCS ranking with Different α values (Evaluated with Qwen2.5-7B on Fin Dataset)
17 16 15 14
Ranking Position
ADMI_50
6 5 4 3 2 1
Figure 10. DCS ranking with Different α values (Evaluated with Llama3-8B on Fin Dataset)
Table 10. Split point ablation on Llama3.2-3B, Fin Dataset, under PIDI attack. model
defense
nh
nt
MP
APα=0.5
APresponse
APinput
Llama3.2-3B Llama3.2-3B Llama3.2-3B Llama3.2-3B
ADMI ADMI ADMI ADMI
3 4 5 6
3 4 5 6
0.474 0.516 0.505 0.496
0.009 0.017 0.004 0.010
0.009 0.009 0.001 0.005
0.010 0.025 0.006 0.014
Llama3.2-3B Llama3.2-3B Llama3.2-3B Llama3.2-3B
wo wo wo wo
3 4 5 6
3 4 5 6
0.560 0.560 0.560 0.560
0.950 0.868 0.784 0.726
0.946 0.891 0.842 0.828
0.954 0.844 0.726 0.623
C. Further Ablations C.1. Ablations on Split Point To further assess the impact of split configurations in Split-LLM systems, we additionally evaluate different split points with nh = nt ∈ {3, 4, 5, 6} on Llama3.2-3B over the Fin dataset under the PIDI attack. The results are reported in Table 10. Overall, the attack performance without defense gradually decreases as the split point moves deeper into the model, since larger model head and tail are harder to invert. In contrast, ADMI consistently maintains strong protection performance across all split settings, reducing AP to near-zero values while preserving stable utility. We also observe that, although deeper split points naturally lower AP even without defense, the remaining leakage is still substantial (e.g., APα=0.5 = 0.726 at the 6-6 split). This result further highlights the necessity of dedicated privacy-preserving mechanisms such as ADMI in practical Split-LLM deployments. C.2. Ablations on auxiliary dataset To evaluate the dependence of PIDI on attacker-side auxiliary data, we conduct additional ablation studies by varying the auxiliary dataset size used for SIP initialization as shown in Figure 11. We observe that reducing the auxiliary dataset size degrades the quality of the initial SIP reconstruction, especially in extremely low-resource settings. However, the final AP of PIDI remains relatively stable once a small amount of auxiliary data is available. This result suggests that the subsequent PMI refinement stage can effectively compensate for imperfect initialization. These findings demonstrate that PIDI is less sensitive to auxiliary data scarcity compared to SIP-only attacks, highlighting the synergistic interaction between initialization and iterative refinement in our framework. C.3. Computation overhead analysis We further analyze the computational overhead introduced by ADMI on the Dolly dataset(see Table 11). 16
From Prompts to Responses: Dual-Sided Data Leakage and Defense in Split Large Language Models Table 11. Computation overhead analysis on the Dolly dataset. Model
Method
Inference Throughput (token/s/GPU)
Training Speed (s/iter)
Warmup Training Speed (s/iter)
Total Training Time (s)
Qwen2.5-7B Qwen2.5-7B
w/o defense ADMI
24.892 20.418
0.915 1.381
/ 0.449
13888.013 21866.425
Llama3-8B Llama3-8B
w/o defense ADMI
23.354 17.486
0.996 1.449
/ 0.482
17182.315 21843.745
Overall, ADMI introduces moderate computational overhead due to the additional MID optimization and adapter-based warm-up stage. Compared with the undefended setting, inference throughput decreases by approximately 18% on Qwen2.57B and 25% on Llama3-8B. Training time also increases because of the additional optimization objectives and local warm-up process. We therefore consider computational efficiency an important limitation of the current design and a promising direction for future work, particularly in scenarios where both strong privacy protection and low-latency deployment are required. Table 12. Additional privacy leakage metrics (APα=0.5 ). model
dataset
method
EntityRecovery-F1
BERTScore-F1
BLEURT
BLEU(Origin)
Llama3.2-3B Llama3.2-3B Llama3.2-3B Llama3.2-3B Llama3.2-3B
Fin Fin Fin Fin Fin
w/o defense ADMI DualGuard MID random baseline
0.868 0.057 0.393 0.236 0.003
0.977 0.753 0.864 0.807 0.746
0.739 0.148 0.447 0.299 0.109
0.868 0.017 0.296 0.167 0.000
Llama3.2-3B Llama3.2-3B Llama3.2-3B Llama3.2-3B Llama3.2-3B
Med Med Med Med Med
w/o defense ADMI DualGuard MID random baseline
0.771 0.057 0.397 0.250 0.079
0.981 0.749 0.873 0.815 0.748
0.797 0.149 0.406 0.277 0.103
0.901 0.074 0.312 0.138 0.000
Qwen2.5-7B Qwen2.5-7B Qwen2.5-7B Qwen2.5-7B Qwen2.5-7B
Fin Fin Fin Fin Fin
w/o defense ADMI DualGuard MID random baseline
0.839 0.000 0.629 0.340 0.002
0.980 0.743 0.925 0.840 0.741
0.703 0.127 0.635 0.382 0.104
0.892 0.004 0.572 0.290 0.000
Qwen2.5-7B Qwen2.5-7B Qwen2.5-7B Qwen2.5-7B Qwen2.5-7B
Med Med Med Med Med
w/o defense ADMI DualGuard MID random baseline
0.674 0.056 0.688 0.309 0.081
0.976 0.733 0.923 0.827 0.743
0.711 0.126 0.575 0.275 0.099
0.801 0.007 0.503 0.211 0.000
D. Additional Results Due to space limit, we place the full experiment results on all datasets/models in this section(see Tables 5 to 9 and Figures 9 and 10). We further report additional semantic and entity-level evaluation metrics for both privacy leakage and model utility in Tables 12 and 13, including EntityRecovery-F1, BERTScore-F1 (Zhang et al., 2020), and BLEURT (Sellam et al., 2020), in addition to the BLEU/METEOR metrics adopted in the main paper for consistency with prior Split-LLM studies (Chen et al., 2024; Liu et al., 2025). EntityRecovery-F1 evaluates the recovery of sensitive entities from reconstructed text, while BERTScore and BLEURT measure semantic similarity. We additionally include a random baseline constructed from randomly sampled token sequences for reference. Overall, ADMI consistently achieves the lowest privacy leakage across lexical, semantic, and entity-level metrics, while maintaining competitive utility performance. These results demonstrate that the effectiveness of ADMI does not depend on a specific metric choice and generalizes beyond evaluation protocols.
17
From Prompts to Responses: Dual-Sided Data Leakage and Defense in Split Large Language Models
Table 13. Additional utility metrics (M P ).
model
dataset
method
BertScore-F1
BLEURT
METEOR(Origin)
Llama3.2-3B Llama3.2-3B Llama3.2-3B Llama3.2-3B
Fin Fin Fin Fin
w/o defense ADMI DualGuard MID
0.933 0.924 0.915 0.817
0.624 0.589 0.547 0.175
0.560 0.516 0.459 0.510
Llama3.2-3B Llama3.2-3B Llama3.2-3B Llama3.2-3B
Med Med Med Med
w/o defense ADMI DualGuard MID
0.867 0.863 0.856 0.863
0.352 0.340 0.330 0.339
0.176 0.173 0.147 0.169
Qwen2.5-7B Qwen2.5-7B Qwen2.5-7B Qwen2.5-7B
Fin Fin Fin Fin
w/o defense ADMI DualGuard MID
0.930 0.924 0.930 0.923
0.614 0.591 0.609 0.581
0.550 0.527 0.542 0.515
Qwen2.5-7B Qwen2.5-7B Qwen2.5-7B Qwen2.5-7B
Med Med Med Med
w/o defense ADMI DualGuard MID
0.869 0.867 0.867 0.865
0.350 0.345 0.345 0.348
0.184 0.182 0.178 0.191
0.9 0.8 0.8
AP Value
AP Value
0.6 0.7
0.4
0.6 0.2
APinput APresponse AP = 0.5 AP of SIP initialization on input
0.5
5 10
20
40
50
100
Auxiliary Data Size
APinput APresponse AP = 0.5 AP of SIP initialization on input
0.0
200
5 10
(a) Qwen2.5-7B
20
40
50
100
Auxiliary Data Size
(b) Llama3-8B
Figure 11. Impact of auxiliary dataset size on PIDI attack performance, evaluated on the Fin dataset.
18
200