Inverting the Hidden: Unveiling Multimodal Privacy Leakage in Collaborative LVLM Inference Shuaifan Jin†,≀ Yajie Zhou†,≀
Zhibo Wang†,≀,∗ Qiyuan Wang‡ Yiting Han†,≀ Yuanfan Zhang†,≀ Jiahui Hu§ Xiaoyi Pang♮
†
The State Key Laboratory of Blockchain and Data Security, Zhejiang University, China ≀ College of Computer Science and Technology, Zhejiang University, China ‡ Department of Statistics, Texas A&M University, USA § Nanchang University, China ♮ Hong Kong University of Science and Technology, China
arXiv:2608.01020v1 [cs.CR] 2 Aug 2026
{shuaifanjin,zhibowang}@zju.edu.cn [email protected] {yitinghan,yajiezhou,zhangyuanfan}@zju.edu.cn [email protected] [email protected]
Abstract—Collaborative inference deploys Large VisionLanguage Models (LVLMs) by partitioning computation between edge devices and the cloud. While withholding raw inputs supposedly ensures privacy, transmitting intermediate hidden states exposes a critical attack surface. However, it remains unclear whether deep-layer LVLM hidden states retain recoverable private information, given that visual content has been projected into the language embedding space. To address this concern, we theoretically analyze LVLM hidden-state recoverability and show that, under regularity assumptions and a positive semantic– nuisance margin, privacy-relevant visual semantics remain identifiable and stably recoverable. Motivated by this analysis, we propose RASR, a novel coarse-to-fine multimodal reconstruction attack. RASR obtains initial image and text reconstructions through modality-specific inverse paths that follow their respective forward processing pipelines in reverse, and then uses hidden-state consistency to refine both reconstructions. Evaluations on Qwen3-VL-8B-Instruct and LLaVA-1.5-7B across five datasets demonstrate that RASR reduces image reconstruction MSE by ∼50% compared to the strongest baselines, while achieving up to 99% token accuracy for text recovery. These results show that privacy-sensitive visual and textual information can be recovered even from deep-layer LVLM hidden states, exposing the privacy risks of collaborative inference. Index Terms—Collaborative Inference, Data Privacy, VisionLanguage Models
I. I NTRODUCTION Large Vision-Language Models (LVLMs) can jointly process visual and textual inputs, perceiving images and reasoning about them in natural language. This capability has made LVLMs widely used across a range of applications, including visual question answering, image captioning, and multimodal dialogue. However, the large parameter size and memory footprint of LVLMs make it difficult to run them directly on resource-constrained edge devices. A widely adopted solution is collaborative inference, in which the model is split into a front-end part that runs locally on the device and a back-end part that runs on a cloud server. The local device executes the first several layers and sends only the resulting intermediate ∗ Zhibo Wang is the corresponding author. Copyright may be transferred without notice, after which this version may no longer be accessible.
hidden states to the cloud, which completes the remaining computation. Because the raw image and text never leave the local device, this approach is commonly believed to enable efficient inference while preserving data privacy. In this paper, we question this presumed privacy guarantee. While raw inputs are withheld, prior research in unimodal contexts has demonstrated that transmitting intermediate representations does not inherently guarantee privacy. In language models, sensitive input text can be reconstructed from intermediate activations or gradients [1]–[3]. Similarly, in traditional vision models (e.g., CNNs and Vision Transformers), intermediate activations from shallow layers can be exploited to recover sensitive image content [4]–[6]. These vulnerabilities raise an alarming security question for multimodal ecosystems: Since collaborative LVLM inference exposes intermediate hidden states derived from both visual and textual inputs, can an adversary reconstruct the user’s original multimodal inputs from these hidden states? A recent survey on LVLM attacks also points out that the privacy landscape of LVLMs remains poorly understood [7]. Answering this question is far from a trivial extension of existing unimodal inversion attacks. Although recent work [2] has proved that text can be recovered from unimodal languagemodel hidden states by exploiting the discreteness of the text space, the recoverability of visual inputs from deep LVLM hidden states remains unclear, as visual reconstruction requires searching a high-dimensional continuous space without comparable discrete constraints. Moreover, the multimodal setting introduces further challenges. In LVLMs, the visual input is first encoded and projected into the language embedding space, and then processed together with the textual tokens through multiple Transformer layers. As a result, the exposed visual hidden states have undergone both cross-modal projection and deep transformation, and each textual hidden state is conditioned on the visual prefix through causal self-attention. These challenges led us to answer two fundamental subquestions for the original research question. First, do splitlayer hidden states retain privacy-relevant visual semantics? Second, if such information is retained, can an adversary
reconstruct the original visual and textual inputs from these LVLM hidden states, and how effective is such reconstruction? To address these questions, we first analyze whether privacy-relevant visual semantics remain recoverable after cross-modal alignment and LLM propagation. By modeling local image variations using semantic and nuisance factors, we derive a sufficient condition for local semantic recovery. Specifically, under local regularity assumptions and a positive semantic–nuisance separation margin, privacy-relevant visual semantics remain locally identifiable and stably recoverable. Empirical measurements further show that the evaluated semantics remain decodable and that the measured margins remain positive across model depth, providing support for our analysis. Motivated by this recoverability analysis, we propose RASR (Recoverability-Aware Symmetric Reconstruction), a coarseto-fine multimodal reconstruction attack for recovering images and text instructions from intercepted hidden states. RASR employs two modality-specific reconstruction branches whose architectures mirror the corresponding forward pathways of the target LVLM. In the coarse stage, the textual branch predicts all tokens in a single pass, and the visual branch progressively maps the visual hidden states back through the visual-embedding and visual-feature spaces to produce an initial image reconstruction. In the fine stage, RASR uses hidden-state consistency to selectively correct low-confidence textual tokens through position-wise search and iteratively refine the image by minimizing the discrepancy between the intercepted and regenerated hidden states. Our main contributions can be summarized as follows. • We provide the first theoretical analysis of hidden-state recoverability in collaborative LVLM inference. We derive an explicit sufficient condition under which privacyrelevant visual semantics remain locally identifiable and stably recoverable after cross-modal alignment and LLM propagation, and provide empirical evidence consistent with this condition on real LVLMs. • We propose RASR, a coarse-to-fine multimodal reconstruction attack that recovers images and text from an intercepted hidden-state sequence. RASR combines modality-specific symmetric reconstruction with hidden-state-consistencyguided refinement to progressively improve both visual and textual reconstructions. • Through extensive evaluations on leading LVLM architectures and diverse datasets, we demonstrate that RASR significantly outperforms existing baselines in reconstruction performance. Our findings bridge the gap between theoretical recoverability and practical attack efficacy, underscoring the urgent need for robust defenses in collaborative LVLM inference systems. II. R ELATED W ORK A. LVLMs and Collaborative Inference 1) Large Vision-Language Models: Large VisionLanguage Models bridge visual perception and language
reasoning [8], [9]. Unlike earlier dual-encoder models such as CLIP [10], current LVLMs [11]–[16] typically employ an LLM as the reasoning backbone and project visual features into its token space through an alignment module. However, their substantial computational and memory demands make end-to-end inference on resource-constrained devices challenging, motivating collaborative inference. 2) Collaborative Inference: In collaborative inference, model computation is partitioned across multiple devices. Neurosurgeon [17] studies layer-level partitioning of conventional DNNs between mobile devices and cloud servers. Recent systems extend this paradigm to large Transformer models: SplitLLM [18] distributes LLM layers between clients and servers, Model-Distributed Inference [19] partitions LLMs across edge devices, and Petals [20] distributes Transformer blocks across geographically separated nodes. Although these systems exchange intermediate activations or hidden states, they primarily focus on deployment efficiency rather than the privacy risks of the transmitted representations. B. Privacy Inversion Attacks 1) Vision Models: Prior studies show that intermediate CNN representations may retain sufficient information for reconstructing visual inputs [4], [21]–[23]. Recent attacks extend image inversion to Transformer-based vision models using gradients, embedding optimization, sparse representations, or learned inverse mappings [5], [6], [24], [25]. However, these studies focus on unimodal vision models and do not determine whether visual information remains recoverable after crossmodal alignment and propagation through an LLM backbone. 2) Language Models: Early language-model inversion attacks mainly recover specific attributes, patterns, or keywords [26], [27]. More recent methods reconstruct complete text from split-learning transmissions, model outputs, sentence embeddings, or hidden representations [1]–[3], [28]– [30]. Nevertheless, these methods are designed for unimodal language models and generally do not consider the visual context incorporated into LVLM hidden states. Recent cross-modal attacks remain limited to recovering visual attributes or training data [31], [32]. The reconstruction of both image and text inputs from intermediate hidden states in collaborative LVLM inference remains underexplored [7]. Our work addresses this gap by systematically investigating their recoverability from transmitted hidden states. III. P RELIMINARY This section first describes how multimodal inputs are processed and how intermediate hidden states are transmitted from edge devices to the cloud, and then specifies the adversary’s capabilities and objective. A. System Model 1) LVLM Architecture: In this paper, we consider a widely adopted LVLM architecture that employs an LLM as the central backbone [11]–[16]. It consists of a visual encoder fv , also referred to as the vision tower, a cross-modal aligner
Text instruction
Output head
···
Causal Attention
···
Transformer layer
···
Causal Attention
Visual token
Transformer layer
Embedding layer
··· Textual token
Visual feature
Cross-modal aligner
Visual encoder
Input image
B. Threat Model
Hidden Back-End Participant (Cloud) states
Front-End Participant (Edge)
Final response
𝑯
Fig. 1: Collaborative inference pipeline of LVLMs.
fa , and an LLM comprising a token embedding layer fe , a Transformer backbone ftr , and an output head. Formally, given an input image I, the visual encoder extracts features Z = fv (I), which the aligner maps to an ordered sequence of visual tokens V = fa (Z) ∈ Rn×d , where n is the number of visual tokens and d is the LLM hidden dimension. Let T = (t1 , . . . , tm ) denote the tokenized text instruction, with embeddings ET = fe (T) ∈ Rm×d . The concatenated multimodal sequence [V; ET ] is processed by the Transformer backbone, and the resulting contextualized representations are further processed by the output head to generate the response Y through autoregressive decoding. Special tokens, prompttemplate tokens, and padding positions are omitted for clarity. 2) Collaborative Inference: In this paper, we consider a two-participant abstraction consisting of a front-end participant and a back-end participant, which also captures a single partition boundary in a multi-participant deployment [17]– [20]. Specifically, the partition boundary is placed after the s-th layer of the Transformer backbone. The front-end participant hosts the visual encoder, the cross-modal aligner, the token embedding layer, and the first s Transformer layers, while the back-end participant hosts the remaining Transformer layers (s) (s) and the output head. Let Fpre and Fpost denote the Transformer sub-networks before and after the partition boundary, respectively. The Transformer backbone can therefore be de(s) (s) composed as ftr = Fpost ◦ Fpre . The front-end participant first constructs the unified multimodal sequence and then computes the split-layer hidden (s) states as H(s) = Φ≤s (I, T; θ) = Fpre [V; ET ] , where Φ≤s denotes the complete front-end mapping up to the partition boundary, and θ contains the parameters of the visual encoder, cross-modal aligner, token embedding layer, and first s Transformer layers. The transmitted sequence can be written (s) (s) (s) as H(s) = [Hv ; Ht ] ∈ R(n+m)×d , where Hv ∈ Rn×d (s) and Ht ∈ Rm×d denote the hidden states at the visual-token and textual-token positions, respectively. Here, the subscripts v and t refer only to the corresponding token positions in the original multimodal sequence, since both parts have already been contextualized by the Transformer layers. The hidden states H(s) are transmitted to the back-end participant, which processes them through the remaining Transformer layers and the output head to generate the final response Y. In this work, we focus on the privacy risks posed by the transmission of H(s) .
1) Adversary Scenario: In this paper, we consider an honest-but-curious back-end participant (e.g., a cloud server) that receives the transmitted hidden states during collaborative LVLM inference as the adversary. The adversary follows the prescribed inference protocol and does not modify the transmitted representations, interfere with model execution, or tamper with the final outputs. However, it attempts to infer private information from the intermediate representations exposed during collaborative inference. Specifically, the adversary can observe the transmitted hidden states H(s) , but cannot directly access the original image I or the text instruction T. 2) Adversary Knowledge and Capability: Following prior work [1], [2], [21], [30], we assume that the adversary knows the target LVLM architecture and collaborative inference configuration, including the partition boundary, hidden-state dimensions, and sequence structure. The adversary also has access to the parameters of the front-end mapping Φ≤s , enabling it to reproduce the forward computation and propagate gradients up to the partition boundary. Such access is feasible when the deployed LVLM weights are publicly available or the corresponding model checkpoint is accessible to participating parties. The adversary may use public auxiliary image-text data, but cannot access the victim’s original inputs or the frontend participant’s private runtime environment. 3) Adversary Objective: The adversary’s primary objective is to compromise user privacy by recovering sensitive visual and textual information from the transmitted hidden states. The adversary does not aim to disrupt the normal collaborative inference process, but instead observes the hidden states H(s) and reconstructs the victim’s private inputs: (Î, T̂) = Rψ H(s) , where Rψ denotes the adversary’s reconstruction mapping parameterized by ψ, and Î and T̂ denote the reconstructed image and text instruction, respectively. IV. R ECOVERABILITY OF LVLM H IDDEN S TATES This section gives a sufficient condition for locally recovering privacy-relevant visual semantics from exposed LVLM hidden states without requiring global or exact pixel-level invertibility. The condition is quantified by a quotient margin that separates semantic responses from nuisance variations and is estimated using Jacobian–vector products in Section VI-B. A. Quotient-Transverse Semantic Recoverability Fix a text instruction T and a split layer s. In a neighborhood of a natural image, let I = G(c, u),
hs (c, u) = vec(Φ≤s (G(c, u), T; θ)) .
(1)
where G is a local parameterization of natural images, c ∈ Rdc denotes selected perceptually relevant semantic coordinates, u ∈ Rdu denotes nuisance factors not specified by c, and hs (c, u) ∈ Rdh denotes the vectorized layer-s hidden state. Define ∂hs ∂hs , Ju = , Jc = ∂c ∂u (2) ⊥ † ⊥ Πu = Idh − Ju Ju , Ss = Πu Jc ,
where Idh is the dh -dimensional identity matrix, † denotes the Moore–Penrose pseudoinverse, and Π⊥ u is the orthogonal projector onto the complement of the nuisance-response space Im(Ju ). Thus, Ss measures the hidden-state response to semantic variations after removing components explainable by nuisance variations. At a reference point x0 = (c0 , u0 ), assume that: (C1) hs is twice continuously differentiable on an open neighborhood of x0 ; (C2) Ju has constant rank r on a neighborhood of x0 ; (C3) the quotient semantic margin γ0 := σmin (Ss (c0 , u0 )) is positive. Condition (C3) requires every nonzero semantic direction to induce a hidden-state variation that cannot be locally canceled by nuisance-induced variations. This condition depends only on the local geometry of the forward hidden-state mapping and does not presuppose the existence of an inverse decoder. Theorem 1 (Local semantic recovery from quotient transversality). Under Conditions (C1)–(C3), there exist a neighborhood W0 of x0 , a local hidden-state manifold M0 = hs (W0 ), and a continuously differentiable local semantic decoder ϕs : M0 → Rdc such that (c, u) ∈ W0 .
ϕs (hs (c, u)) = c,
(3)
Moreover, W0 can be chosen such that γ :=
inf (c,u)∈W0
σmin (Ss (c, u)) ≥
γ0 > 0. 2
(4)
For any (c, u) ∈ W0 and any tangent hidden-state perturbation e = Jc (c, u)∆c + Ju (c, u)∆u ∈ Ths (c,u) M0 ,
(5)
the semantic decoder satisfies ∥Dϕs (hs (c, u))e∥2 = ∥∆c∥2 ≤ γ −1 ∥e∥2 .
(6)
∥ϕs (h′ ) − ϕs (h)∥2 ≤ γ −1 ∥h′ − h∥2 + o(∥h′ − h∥2 ).
(7)
Proof. By Condition (C2), Ju† and hence Π⊥ u vary continuously in a sufficiently small neighborhood of x0 . Condition (C3) and the continuity of singular values then allow the neighborhood to be chosen such that σmin (Ss ) ≥ γ0 /2, which gives (4). For (∆c, ∆u) ∈ ker Dhs (c, u), =⇒
Ss ∆c = 0
=⇒
∆c = 0,
where the first implication follows by applying Π⊥ u , and the second follows because Ss has full column rank. Therefore, with πc (c, u) := c, we have ker Dhs (c, u) ⊆ ker Dπc (c, u).
(8)
Since Π⊥ u Jc lies in the orthogonal complement of Im(Ju ), rank Dhs = rank[Jc Ju ] = rank(Ju ) + rank(Π⊥ u Jc ) = r + dc .
Dϕs Jc = Idc ,
Dϕs Ju = 0.
(9)
Thus, for e = Jc ∆c + Ju ∆u, we have Dϕs e = ∆c. Moreover, ∥e∥2 ≥ ∥Π⊥ u e∥2 = ∥Ss ∆c∥2 ≥ γ∥∆c∥2 ,
(10)
which proves (6). Since M0 is a local C 2 embedded manifold, for h′ sufficiently close to h, there exists a C 1 curve in M0 joining h and h′ whose length is ∥h′ − h∥2 + o(∥h′ − h∥2 ) as h′ → h. Integrating the differential bound along this curve yields (7). B. Scope of the Result Theorem 1 establishes a locally stable semantic left inverse for the selected semantic coordinates c, rather than a full inverse for (c, u). This existence result motivates the learned reconstruction model introduced in Section V, while recovery beyond the selected semantic coordinates is evaluated empirically. Section VI-B examines finite-dimensional signatures of the sufficient condition, including semantic readability, non-degenerate semantic responses, and the quotient margin after removing measured nuisance directions. Because these quantities are estimated using selected semantic and nuisance directions at finitely many samples and split layers, the experiments provide evidence consistent with local recoverability rather than establishing a uniform guarantee. V. M ETHOD
Consequently, for h, h′ ∈ M0 with h′ → h,
Jc ∆c + Ju ∆u = 0
Hence, after shrinking W0 if necessary, hs has constant rank r + dc . The constant-rank theorem then allows W0 to be chosen such that M0 = hs (W0 ) is an embedded submanifold and each fiber of hs |W0 is connected. Together with (8), this implies that πc is constant along each such fiber. The local factorization theorem therefore yields a continuously differentiable map ϕs satisfying ϕs ◦hs = πc , which proves (3). Differentiating ϕs ◦ hs = πc gives
This section presents our inverse attack for revealing privacy risks in large vision-language models (LVLMs). After introducing the overall framework in section V-A, we detail the symmetric reconstruction architecture in section V-B and the consistency-guided refinement strategy in section V-C. A. Overview Building on the preceding recoverability analysis, we design an inverse attack to exploit the recoverable information preserved in split LVLM hidden states. The analysis shows that the split-layer mapping does not uniformly collapse visual information. Instead, privacy-relevant visual variations can remain distinguishable in the hidden-state space, suggesting that split LVLM hidden states may preserve recoverable visual semantics even though they are no longer raw inputs. Motivated by this observation, we construct modality-specific inverse mappings and further refine their outputs through hidden-state consistency to recover privacy-relevant input information. Fig. 2a illustrates the overall pipeline of our inverse attack. It follows a coarse-to-fine strategy comprising two
Consistency-Guided Refinement
𝑯
min 𝐷 𝐇
30
𝑯
Training phase
+
Transformer-based Reconstructor ℛ
Visual Encoder
Text
𝐈
Token IDs
Can I trust my therapist? 𝐓
Single-Pass Position-Wise Token Prediction
Pixel Update 𝐈ˆ
0
20
10
t-SN 0 −10 E 2 −20
0
10
20
NE t-S
1
trust
Can
I
my
counselor
<𝜏
−10
(b) Feature space (LLaVA).
Token Search 𝐓ˆ
𝐈ˆ
Textual tokens Visual tokens
?
10
𝑘
Visual Decoder ℛ
ℒ Training phase
Inverse Aligner ℛ
advisor
therapist
0 −10 −20
Can I trust ˆ my counselor? 𝐓
Image
Embedding layer
Text
Cross-modal Aligner
LVLM Prefix
𝐈ˆ , 𝐓ˆ
Token Distributions
10
−10 −20
𝐈ˆ
𝐈ˆ
Text
ℒ
Transformer-based Reconstructor ℛ
Image
LLM prefix
Image
Front-End Participant (Edge)
˜ 𝑯
20
t-SNE 3
𝑯
Textual tokens Visual tokens
,𝐇 t-SNE 3
Cloud
Symmetric Reconstruction LLM suffix
Can I trust ˆ my therapist? 𝐓
−20
−10
t-SN 0 E
1
10
20
−10
10
20
2 NE
0
t-S
(c) Feature space (Qwen).
(a) Overall pipeline of RASR with symmetric reconstruction and consistency-guided refinement.
Fig. 2: Overview of the proposed reconstruction pipeline and the feature-space distributions of visual and textual hidden states.
modules: Symmetric Reconstruction and Consistency-Guided Refinement. Given the intercepted hidden states, Symmetric Reconstruction first separates the visual and textual components and obtains initial image and text reconstructions through modality-specific inverse paths that approximately reverse the corresponding forward stages. Consistency-Guided Refinement then updates these reconstructions to reduce the discrepancy between their re-encoded hidden states and the intercepted target hidden states. B. Symmetric Reconstruction Architecture In collaborative LVLM inference, the visual input is encoded by the visual encoder, projected into the language embedding space by the cross-modal aligner, concatenated with textual tokens, and subsequently processed by the LLM backbone through causal self-attention. Consequently, the representations transmitted across the partition boundary are contextualized hidden states in a shared LLM hidden space rather than isolated modality-specific features. Although cross-modal interaction progressively transforms these representations, the visual-prefix layout and the positional correspondence of the original multimodal sequence remain unchanged. To explore whether modality-dependent structure is preserved after such propagation, we visualize the hidden states at visual and textual token positions using t-SNE. As shown in figs. 2b and 2c, the two modalities still form distinguishable distributions, suggesting that LLM contextualization does not completely eliminate their modality-dependent organization. Given the intercepted sequence H(s) = (h1 , . . . , hL ) and the known visual-prefix length b, we separate it into Hv = (h1 , . . . , hb ) and Ht = (hb+1 , . . . , hL ). After separation, the two modalities require different reconstruction strategies because visual hidden states are produced from images through multiple successive continuous transformations, whereas textual hidden states retain a position-wise correspondence with discrete input tokens. Thus, we process Hv and Ht using dedicated visual and textual branches that approximately reverse the major transformations along their respective forward paths.
Following this symmetric reconstruction design, we begin with visual reconstruction. Rather than mapping Hv directly to raw pixels with a single reconstruction network, we decompose the inverse mapping according to the major stages of the forward visual path. In the forward direction, the image is mapped to visual features by the visual encoder, projected into aligned visual tokens by the cross-modal aligner, and subsequently transformed into the intercepted visual hidden states by the LLM layers preceding the partition. Accordingly, we construct a hierarchical visual branch that approximately reverses this transformation sequence: I → Z → V → Hv ,
R
Ralign
R
vis Hv −−llm −→ V̂ −−−−→ Ẑ −−− → Î. (11)
Specifically, since Hv and V are both ordered token sequences with position-wise correspondence, a Transformer is naturally compatible with the required sequence-to-sequence mapping while capturing cross-token dependencies. We therefore implement Rllm as a Transformer-based reconstructor that maps Hv back to the aligned visual tokens V̂. Ralign then projects V̂ into the visual-feature space to obtain Ẑ. Finally, Rvis restores the spatial patch layout and progressively decodes Ẑ into the reconstructed image Î. With the hierarchical visual branch established, we optimize its three stages jointly rather than imposing separate reconstruction objectives on their intermediate outputs. Specifically, although structurally aligned with the corresponding forwardpath representations, V̂ and Ẑ serve only as latent transition states rather than explicit reconstruction targets. We do not supervise them directly because enforcing point-wise agreement with the original activations could unnecessarily restrict the latent solution space and hinder final image reconstruction. Accordingly, we define the optimization objective solely in the image space: 2 1 Lvisual = Î − I , (12) |Ω| 2 where I denotes the original image and |Ω| is the number of image elements.
For textual reconstruction, the intercepted textual hidden states preserve both their position-wise correspondence with the input tokens and the causal dependency structure induced by the decoder-only Transformer backbone Rtext . Therefore, we employ a structurally matched decoder-only Transformer parameterized by ψt to recover the aligned token sequence. After re-indexing the separated textual positions, let Ht = (ht1 , . . . , htm ) and T = (t1 , . . . , tm ) denote the textual hidden states and the corresponding ground-truth tokens, respectively. Under a causal attention mask, the output at position i depends only on the hidden-state prefix H≤i = (ht1 , . . . , hti ) and t predicts the aligned token ti . Accordingly, the optimization objective is the position-wise cross-entropy loss: m 1 X (i) log pψt ti | H≤i , (13) Ltextual = − t m i=1 (i)
where pψt (ti | H≤i t ) is the probability of ti at position i. Since predicted tokens are not fed back into the reconstructor, causal masking does not require sequential decoding, and all positions are recovered in a single forward pass. Overall, the visual and textual reconstruction branches are optimized using Lvisual and Ltextual , respectively, thereby learning modality-specific inverse mappings from the intercepted hidden states to the image and text spaces. Their outputs, denoted by (Î0 , T̂0 ), serve as the initial reconstructions for the subsequent consistency-guided refinement stage. C. Consistency-Guided Refinement Although the symmetric reconstruction stage provides initial reconstructions, its learned inverse mappings may not fully exploit the information preserved in the intercepted hidden states. Therefore, we refine these reconstructions by matching their regenerated hidden states to the intercepted ones: e (s) (I′ , T′ ), H(s) , H min D (14) ′ ′ I ,T
′
where I and T′ denote the candidate reconstructions, e (s) (I′ , T′ ) = Φ≤s (I′ , T′ ; θ) denotes their hidden states H regenerated by the frozen front-end mapping, and D(·, ·) measures the hidden-state discrepancy. For visual refinement, we fix T̂0 and start from Î0 . At itere (s) = H e (s) (Îk , T̂0 ) ation k = 0, . . . , K − 1, we regenerate H k and update the image by h i e (s) , H(s) , (15) Îk+1 = ΠN (Î0 ) Îk − ηk ∇Îk D H k where ηk is the step size and ΠN (Î0 ) denotes projection onto the neighborhood N (Î0 ). The resulting ÎK is used as the visual context for textual refinement. For textual refinement, we fix ÎK , initialize T̂ = T̂0 , and refine the low-confidence positions Popt = {i | qi < τ } from left to right, where qi is the highest token confidence at position i and τ is the threshold. For each position i, we initialize a continuous variable ei with its current token embedding and optimize it to minimize the hidden-state discrepancy, yielding e∗i . The candidate set Ci comprises the eight vocabulary tokens
whose embeddings are nearest to e∗i in Euclidean distance. Let T̂(i←v) denote the current reconstruction with its ith token hidden state at position i replaced by v, and e hti (v) the textual regenerated from ÎK , T̂(i←v) . We select (16) t̂i = arg min D e hti (v), hti , v∈Ci
where hti is the intercepted textual hidden state at position i. The selected token updates T̂ before the next position is processed, while positions outside Popt remain unchanged. VI. E XPERIMENTS To evaluate the recoverability of hidden states and the effectiveness of our reconstruction attack, we conduct comprehensive experiments to answer the following research questions: • RQ1: Do split-layer hidden states retain privacy-relevant visual semantics? • RQ2: How effectively can the proposed attack reconstruct visual and textual inputs from hidden states? A. Experimental Setup 1) Datasets and Pre-processing: For visual reconstruction, we use VQAv2 [33], CelebA [34], and Oxford Flowers102 [35], with 88752/1000, 36468/1000, and 6551/1000 training/evaluation samples, respectively. VQAv2 and CelebA follow their official training/validation and training/test splits, while Flowers-102 uses a class-stratified split. VQAv2 uses its associated questions, whereas CelebA and Flowers-102 use annotation-derived instructions and are evaluated only on image reconstruction. For textual reconstruction, we use 88752/120 VQAv2 samples, 21962/120 FEVER [36] samples, and 20117/120 MS MARCO [37] samples. Since FEVER and MS MARCO contain no images, their texts are paired with images from the corresponding VQAv2 training or evaluation pool. The selected VQAv2 training and evaluation subsets are disjoint at the image-ID level. All subsets and pairings are fixed using random seed 1024. Images are resized to 336 × 336 and normalized using the default parameters of each LVLM. Texts are formatted and tokenized using the corresponding chat template and tokenizer. 2) Models and Implementation Details: We evaluate Qwen3-VL-8B-Instruct (Qwen) and LLaVA-1.5-7B (LLaVA), with 36 and 32 Transformer layers, respectively. The models are partitioned after layers 23 and 20, roughly two-thirds of each backbone. All target-model parameters remain frozen throughout training and refinement. For each target model, both reconstruction branches use separately parameterized 1-layer causal Transformer blocks trained from scratch. The blocks follow the designs of Qwen31.7B and Llama-3.2-1B for Qwen and LLaVA, respectively, with a hidden size of 4,096. The visual branch further employs a 2-layer MLP projector, a model-specific inverse vision module, and a u-net style convolutional reconstruction backbone. The corresponding visual feature dimensions are 1,152 and 1,024, with patch sizes of 16×16 and 14×14, respectively. The textual branch projects the mapped hidden states onto
B. Empirical Validation of Hidden-State Recoverability (RQ1) Following Section IV, we examine whether privacy-relevant visual semantics remain (i) linearly decodable from hidden state, (ii) non-degenerate along semantic directions, and (iii) transverse to measured nuisance variations. We evaluate Qwen at layers 0, 6, 11, 17, 23, 29, and 35, and LLaVA at layers 0, 5, 10, 15, 20, 25, and 31. 1) Semantic Decodability: We mean-pool visual-token hidden states and fit independent ridge classifiers for ten CelebA attributes: Male, Smiling, Eyeglasses, Young, Black Hair, Blond Hair, Brown Hair, Mustache, No Beard, and Wearing Hat. Using disjoint sets of 5,000 training and 2,000 test images, the macro-average balanced accuracy ranges from 0.7708
TABLE I: Macro-average balanced accuracy of ten privacyrelevant CelebA attribute probes across model depths.
Layer
Layer
Layer
1.5506
1.0489
0.7317 0.6253
1.2857
5 10 15 20 25 31
0.8801
0.8033
0.7550
0
0.6620
0.7012
(d)
0
0.6586
10
0.5740
11.5142
37.9883 44.0236
Meani γi,̂ l
4.4106
9.3387
2.6093
2.1604
6 11 17 23 29 35
Attribute basis CLIP-PC basis
3.6655
0
2.2541
2.1093
(c)
0
1.9119
(b)
5 10 15 20 25 31
1.8706
10 0
1.8246
10
1
1.6435
Meani γi,̂ l
1.6858
2.6544 3.1359
1.1525
0.8271
Attribute basis CLIP-PC basis
1.4646
1.0381
0.7763
1.0005
0.9203
0.7839
1.0003
0
0.9127
10
0.7634
44.2189 54.2828
Mean raw σmin
5.3085
2.2443
6 11 17 23 29 35
4.1086
2.1857
0
2.4320
0
2.8513
1
Attribute basis CLIP-PC basis
10.8262
Attribute basis CLIP-PC basis
14.7312
2
1.9819
(a)
Layer 0 5 10 15 20 25 31 Bal. Acc. 0.8267 0.8067 0.8001 0.7853 0.7904 0.7798 0.7866
2.0124
10
LLaVA
1.9905
10
Qwen
1.8174
10
Evaluated layers Layer 0 6 11 17 23 29 35 Bal. Acc. 0.8213 0.8077 0.8039 0.8015 0.7971 0.8004 0.7708
2.3985 2.9139
Model
Mean raw σmin
the corresponding target-model vocabulary for position-wise prediction. The visual and textual reconstructors are trained for 3 and 2 epochs with batch sizes of 3 and 1, respectively. All reconstructors use AdamW with a learning rate of 3×10−5 , a linear schedule, and 100 warm-up steps. For consistency-guided refinement, the discrepancy D(·, ·) is defined as mean squared error. Image refinement runs for 100 iterations (K=100) using AdamW, with learning rates of 4.5×10−3 and 1.4×10−2 for Qwen and LLaVA, respectively. After each update, the image is projected to satisfy a normalized L2 (RMS) radius of 0.01 from the initial reconstruction and a maximum per-pixel deviation of 0.05, with pixel values clipped to [0, 1]. Textual refinement uses a confidence threshold of τ =0.85 and a candidate-set size of 8. All experiments are conducted on NVIDIA RTX 5880 Ada Generation GPUs. End-to-end latency is measured with a batch size of 1. 3) Methods for Comparison: We compare our method with five representative inversion baselines covering optimizationbased and learned reconstruction. For images, CLIPInversion [25] performs pixel optimization through representation matching, SMI [6] uses attention-guided sparse inversion, and INVERSE-TVM [5] learns module-wise inverse mappings. For text, PIA [30] combines embedding optimization with adaptive discretization, while SIPIT [2] sequentially recovers tokens through continuous optimization and vocabulary projection. We adapt each baseline to the separated visual or textual hidden states at the target partition layer while preserving its original inversion mechanism. All methods are evaluated using identical data splits, intercepted hidden states, and target-model access, with the same input resolution used for visual reconstruction. 4) Evaluation Metrics: For visual reconstruction, we report MSE, PSNR, and SSIM. MSE measures pixel-wise distortion, PSNR measures reconstruction fidelity, and SSIM evaluates structural similarity. Lower MSE and higher PSNR and SSIM indicate better performance. For textual reconstruction, we report Token Accuracy (Token Acc.), Exact Match Rate (EMR), and BERTScore F1 (BERT-F1), which measure position-wise token recovery, exact sequence recovery, and semantic similarity, respectively. Higher values indicate better performance. Notably, only tokens corresponding to the original textual content contribute to the textual evaluation.
Layer
Fig. 3: Exact-JVP results over 100 evaluation inputs per layer and semantic basis. Panels (a)–(b) report the mean σmin (Jbc,i,l ) for Qwen and LLaVA, while panels (c)–(d) report the corresponding mean quotient margin γ bi,l = σmin (Sbi,l ). Error bars denote 95% input-bootstrap confidence intervals.
to 0.8213 for Qwen and from 0.7798 to 0.8267 for LLaVA (Table I). Every attribute-layer probe at every evaluated layer achieves above 0.5 balanced accuracy, with minimum values of 0.5935 for Qwen and 0.5944 for LLaVA. The results indicate that privacy-relevant attributes remain linearly decodable from hidden states across different model depths. 2) Semantic Non-Degeneracy: Let zi be the vectorized projected visual-token representation of input i, and let Fl (z, T0 ) denote the vectorized layer-l hidden state under T0 = “Describe the image in detail.”. For a model-specific semantic basis Vc , we compute Jbc,i,l = Dz Fl (zi , T0 )Vc using exact JVPs via automatic differentiation. This quantity serves as a token-space proxy for Jc in Eq. (2), with Vc representing the selected semantic directions in the projected space. We use 192 images to construct two orthonormal semantic bases and a disjoint set of 100 images for evaluation. The first is a 9-dimensional CelebA basis formed from the mean differences between positive and negative examples of each attribute. We exclude Mustache because the basis set contains only one positive example. The second is a 16-dimensional CLIP-PC basis, where we map the leading principal components of normalized CLIP image embeddings into the centered projected-token space. Rank and σmin are computed separately for each input before aggregation, with numerical rank determined using the matrix-specific tolerance max{10−10 , 10−6 σmax }. All 2,800 semantic-response matrices Jbc,i,l are full column rank. As shown in Fig. 3(a)–(b), the mean σmin ranges are 1.9819–54.2828/1.8174–44.2189 for Qwen and 0.7839–3.1359/0.7634–2.6544 for LLaVA under the CelebA/CLIP-PC bases, respectively. These results indicate that the hidden state responds to every tested semantic variation along both semantic subspaces. 3) Semantic–Nuisance Transversality: From the same 192 basis images, we construct 960 unit-normalized projectedtoken secants using paired brightness (1.15/0.85), contrast
TABLE II: Quantitative comparison of image quality. Dataset
Qwen-VL
Method
LLaVA
TABLE III: Text reconstruction performance (%). Dataset
0.0195 0.0139 0.4685 0.6256
0.2531 0.1989 0.0335 0.0135
6.32 7.07 15.17 19.24
0.0200 0.0096 0.4761 0.6385
SMI CLIPInversion Flowers-102 INVERSE-TVM Ours
0.2129 0.1834 0.0465 0.0222
6.88 7.43 13.68 17.08
0.0231 0.0122 0.3390 0.4630
0.2041 0.2014 0.0489 0.0204
7.09 7.02 13.43 17.51
0.0220 0.0092 0.3429 0.4879
SMI CLIPInversion INVERSE-TVM Ours
0.2379 0.1687 0.0352 0.0213
6.50 7.81 15.07 17.39
0.0138 0.0130 0.3990 0.4797
0.2370 0.1861 0.0396 0.0249
6.54 7.37 14.51 16.61
0.0140 0.0099 0.3872 0.4504
VQAv2
(1.15/0.85), saturation (1.20/0.80), hue (+0.04/ − 0.04), and sharpness (1.30/0.70) transformations. Their leading 16 right singular vectors define a model-specific nuisance basis Vu shared across inputs and layers. In particular, Vc and Vu are orthonormalized separately. For each input i and layer l, we use automatic differentiation to compute Jbu,i,l = Dz Fl (zi , T0 )Vu , b ⊥ = Id − Jbu,i,l Jb† , Sbi,l = Π b ⊥ Jbc,i,l , and then obtain Π h u,i,l u,i,l u,i,l and γ bi,l = σmin (Sbi,l ). For each layer and semantic basis, we average γ bi,l across evaluation inputs separately. Across all 2,800 nuisance-projected semantic-response matrices Sbi,l , every matrix remains full column rank, with γ bi,l > 0. All 700 LLaVA nuisance-response matrices have rank 16. For Qwen, 693 matrices have rank 16 and the remaining seven have ranks between 11 and 15. Despite this lower nuisance rank, the corresponding projected semanticresponse matrices remain full column rank. Figure 3(c)–(d) shows that the layer-wise mean quotient margin is positive for both semantic bases at every evaluated layer. For Qwen, it ranges from 1.9119 to 44.0236 under the CelebA basis and from 1.6435 to 37.9883 under the CLIP-PC basis. For LLaVA, the corresponding ranges are 0.6620–2.9139 and 0.5740–2.3985. These positive margins indicate that the tested semantic variations remain distinguishable after removing the hidden-state responses associated with the measured nuisance variations. Since hidden-state scales differ across architectures, absolute margins are compared only within each model. To account for layer-dependent response scales, we additionally b ) σ (S compute ρbi,l = σ min(Jb i,l ) separately for each input before min c,i,l averaging. Across layers and semantic bases, the resulting mean ρbi,l ranges from 0.7952 to 0.9656 for Qwen and from 0.7010 to 0.9299 for LLaVA, showing that the weakest semantic response remains substantially preserved after nuisance projection. These pointwise results provide empirical evidence for local semantic recoverability within the evaluated subspaces, as the selected semantics remain linearly decodable and nondegenerate with positive quotient margins after removing measured nuisance responses. They do not establish neighborhoodwide positivity, global or exact pixel-level invertibility, or recoverability under unmeasured nuisance factors.
PIA SIPIT Ours
27.45 96.57 97.93
0.00 78.33 84.17
53.31 96.92 98.65
16.41 98.88 99.56
0.00 90.00 94.17
48.14 99.34 99.63
PIA MS MARCO SIPIT Ours
28.13 95.98 99.76
4.17 84.17 98.33
58.72 96.97 99.81
17.98 99.20 99.90
0.83 93.33 99.17
52.79 99.21 99.97
PIA SIPIT Ours
28.26 83.24 99.89
0.00 38.33 99.17
53.21 83.54 99.95
19.11 98.48 99.80
0.00 93.33 99.17
49.02 99.55 99.92
FEVER
VQAv2
QwenVL Original
SMI
CLIP INVERSE Ours (K=0) Inversion TVM
LLaVA Ours (K=100)
SMI
CLIP INVERSE Ours (K=0) Inversion TVM
Ours (K=100)
CelebA
6.28 7.57 15.58 19.23
Flowers102
0.2504 0.1780 0.0303 0.0133
LLaVA
Token Acc. ↑ EMR ↑ BERT-F1 ↑ Token Acc. ↑ EMR ↑ BERT-F1 ↑
VQAv2
SMI CLIPInversion INVERSE-TVM Ours
CelebA
Qwen
Method
MSE ↓ PSNR ↑ SSIM ↑ MSE ↓ PSNR ↑ SSIM ↑
Fig. 4: Visual comparison across different methods.
C. Effectiveness of Multimodal Reconstruction (RQ2) 1) Visual Reconstruction: To evaluate RASR across visual domains and LVLM architectures, we compare it with three baselines on CelebA, Flowers-102, and VQAv2, which cover faces, fine-grained flowers, and diverse real-world scenes, using Qwen and LLaVA. RASR consistently achieves the strongest quantitative performance. As shown in Table II, it obtains the lowest MSE and highest PSNR and SSIM in all six dataset–model combinations. Compared with INVERSETVM, the strongest baseline, RASR reduces MSE by 50.5% on average, improves PSNR by 3.27 dB, and yields a 30.5% average relative SSIM gain. Because aggregate metrics do not fully reflect input-specific content, we further examine representative reconstructions from all three datasets under both models. RASR also produces more faithful reconstructions. As shown in Fig. 4, CLIPInversion recovers limited meaningful structure, while INVERSE-TVM preserves coarse content but exhibits structural and color distortions. In contrast, RASR better retains spatial layout, color distribution, and input-specific appearance, particularly after consistencyguided refinement. These results show that exposed hidden states preserve recoverable visual information across the evaluated domains and architectures. 2) Textual Reconstruction: To evaluate textual leakage across task types and LVLM architectures, we reconstruct inputs from FEVER, MS MARCO, and VQAv2 using the intercepted hidden states of Qwen and LLaVA. These datasets cover fact verification, passage retrieval, and visual ques-
Layer Qwen
Reconstruction at different layers 0
6
11
17
23
29
35
PSNR ↑ 21.69 21.89 20.46 19.28 19.23 18.44 17.66 Sensitive Acc. ↑ 88.36% 89.04% 87.97% 86.83% 87.22% 86.23% 84.41% Token Acc. ↑ BERT-F1 ↑ Layer
1.0 0.5 0.2 0.1 0 25 50
0
5
10
15
20
25
31
100.00% 99.88% 100.00% 99.88% 99.56% 98.69% 96.82% 100.00% 99.95% 100.00% 99.92% 99.63% 98.44% 97.06%
100
150
0.20
LLaVA20
(b) Edge fidelity
0.10
0.00
200
0 25 50
100
150
200
Optimization steps
30
(c) Runtime
20 10 0 0 25 50
100
150
200
Fig. 5: Effect of the visual refinement budget K.
100.00% 99.57% 99.21% 97.64% 97.93% 96.07% 91.35% 100.00% 99.74% 99.63% 97.93% 98.65% 96.57% 93.24%
21.21 21.06 20.34 19.73 19.24 19.02 18.58 LLaVA PSNR ↑ Sensitive Acc. ↑ 89.54% 89.49% 89.12% 89.37% 89.26% 89.31% 89.11% Token Acc. ↑ BERT-F1 ↑
2.0
Optimization time (s/image)
Metric
(a) Hidden-state alignment
Edge PSNR gain (dB)
Model
Qwen23 Hidden loss (log scale)
TABLE IV: Reconstruction performance across partition layers on CelebA (visual) and FEVER (textual).
TABLE VI: Effect of reconstructed image guidance on textual refinement on VQAv2 (%). Qwen
Setting
LLaVA
Token Acc. EMR BERT-F1 Token Acc. EMR BERT-F1
TABLE V: Ablation of the visual reconstructor architecture on CelebA using Qwen at layer 23. Variant
Effective Params. (∆) MSE ↓ PSNR ↑ SSIM ↑
w/o Hierarchical Mapping 696.85M (+0.20%) 0.0154 w/o Inverse Patch Merger 690.15M (−0.76%) 0.0141 Full 695.47M (0.00%) 0.0133
18.56 18.97 19.23
0.6039 0.6178 0.6256
tion answering. RASR achieves the strongest reconstruction accuracy. As shown in Table III, it performs best on all three metrics across all six dataset–model combinations. Across these settings, RASR averages 99.47% Token Acc., 95.69% EMR, and 99.66% BERT-F1, exceeding SIPIT by 4.08, 16.12, and 3.73 percentage points, respectively. RASR is also substantially faster. Across the three datasets, it requires 0.193/0.267 s/token for Qwen/LLaVA, compared with 33.731/42.192 s/token for PIA and 60.553/160.997 s/token for SIPIT. These results show that split-layer hidden states enable accurate and efficient recovery of both exact textual content and semantic information. 3) Partition Depth: Since collaborative LVLM inference may use different partition depths, we evaluate RASR at seven layers per model on CelebA and FEVER for visual and textual recovery, respectively. On CelebA, we use the pretrained Anycost GANs attribute classifier [38] to measure ten-attribute macro accuracy (Sensitive Acc.), with 94.22% on the original images as the reference. Visual leakage persists across partition depths. As shown in Table IV, although image reconstruction quality generally declines with depth, Sensitive Acc. remains above 84% at the deepest evaluated layer for both models, indicating that privacy-relevant visual attributes remain recoverable. Textual leakage also persists across partition depths. Token Acc. exhibits an overall decline with depth but remains above 91% at the deepest evaluated layer for both models, showing that substantial input-text information persists even in deep-layer hidden states. D. Ablation and Parameter Analysis (RQ2) 1) Visual reconstructor ablation: We assess the contributions of hierarchical mapping and the inverse patch merger by separately removing each component on CelebA using Qwen at layer 23. As shown in Table V, the full reconstructor performs best across all three metrics. Its MSE rises from
w/o reconstructed image with reconstructed image
99.78 99.89
98.33 99.17
99.89 99.95
99.70 99.80
98.33 99.17
99.89 99.92
TABLE VII: Ablation study of the confidence threshold τ . Setting w/o refinement τ = 0.75 τ = 0.80 τ = 0.85 τ = 0.90 τ = 0.95
Token Acc. ↑ EMR ↑ BERT-F1 ↑ Time (ms/token) ↓ 96.16 99.44 99.44 99.47 99.45 99.45
73.33 95.28 95.28 95.69 95.69 95.83
96.92 99.63 99.63 99.66 99.66 99.64
11.25 201.38 214.70 229.74 268.91 315.32
0.0133 to 0.0154 without hierarchical mapping and to 0.0141 without the inverse patch merger. Notably, the former variant uses 0.20% more effective parameters, showing that the benefit of hierarchical mapping is not attributable to parameter count. 2) Visual refinement steps: We vary K on CelebA at the default partition layers to assess effectiveness and runtime. As shown in Fig. 5, relative to K = 0, K = 100 reduces hiddenstate MSE by 61.25% for Qwen and 54.30% for LLaVA, while improving Edge PSNR by only 0.168 and 0.024 dB, respectively. This limited gain may reflect the approximately many-to-one mapping of fine-grained image details into splitlayer hidden states. The constraint narrows the solution space around the initial reconstruction but cannot uniquely recover all edge details. Beyond K = 100, Edge PSNR changes little, LLaVA’s hidden-state MSE rises, and runtime increases. Therefore, we set K = 100 by default. 3) Visual guidance for textual refinement: We examine whether refined-image guidance improves text recovery on VQAv2, the only text benchmark with paired images. As shown in Table VI, it improves Token Acc. by 0.11/0.10 percentage points for Qwen/LLaVA and EMR by 0.84 percentage points for both, with slight BERT-F1 gains. Although modest given the high initial accuracy, these consistent gains indicate that refined visual context provides complementary information for text recovery. 4) Textual refinement threshold: We sweep τ across all six dataset–model combinations to balance refinement coverage and computational cost. As shown in Table VII, τ = 0.85 improves all three recovery metrics over no refinement, reaching 99.47% Token Acc. and 95.69% EMR at 229.74 ms/token. Higher thresholds increase runtime to 268.91–315.32 ms/token
without yielding consistent accuracy gains. Therefore, we use τ = 0.85 as the default threshold. VII. C ONCLUSION AND F UTURE W ORK This work addresses the underexplored risk of multimodal privacy leakage in collaborative LVLM inference. We first show that, under local regularity conditions and a positive semantic–nuisance separation margin, privacy-relevant visual semantics remain locally identifiable and stably recoverable from intermediate hidden states. Building on this analysis, we propose RASR, a coarse-to-fine attack that combines symmetric reconstruction with hidden-state-guided refinement to recover visual and textual inputs. Experiments on two LVLMs and five datasets show that RASR consistently outperforms existing attacks in both visual and textual reconstruction. By demonstrating the recoverability of both modalities, these results underscore the need for lightweight representationlevel defenses that suppress inversion-sensitive information while preserving inference utility, as well as selective privacypreserving computation that protects only high-risk hiddenstate components or layers to reduce computational and communication overhead. R EFERENCES [1] G. Chen, Z. Qin, M. Yang et al., “Unveiling the vulnerability of private fine-tuning in split-based frameworks for large language models: A bidirectionally enhanced attack,” in Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, 2024, pp. 2904–2918. [2] G. Nikolaou, T. Mencattini, D. Crisostomi et al., “Language models are injective and hence invertible,” arXiv preprint arXiv:2510.15511, 2025. [3] H. Zhao, Z. He, F. Yang et al., “Rep2text: Decoding full text from a single llm token representation,” arXiv preprint arXiv:2511.06571, 2025. [4] A. Dosovitskiy and T. Brox, “Inverting visual representations with convolutional networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 4829–4837. [5] J. Rathjens, S. Reyhanian, D. Kappel et al., “Understanding transformerbased vision models through inversion,” 2025. [Online]. Available: https://arxiv.org/abs/2412.06534 [6] Z. Hu, Y. Wei, L. Shen et al., “Sparse model inversion: efficient inversion of vision transformers for data-free applications,” in Proceedings of the 41st International Conference on Machine Learning, 2024, pp. 19 475– 19 490. [7] D. Liu, M. Yang, X. Qu et al., “A survey of attacks on large vision–language models: Resources, advances, and future trends,” IEEE Transactions on Neural Networks and Learning Systems, vol. 36, no. 11, pp. 19 525–19 545, 2025. [8] P. Xu, W. Shao, K. Zhang et al., “Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, no. 3, pp. 1877– 1893, 2025. [9] Z. Li, X. Wu, H. Du et al., “A survey of state of the art large vision language models: Benchmark evaluations and challenges,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2025, pp. 1587–1606. [10] A. Radford, J. W. Kim, C. Hallacy et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning. PmLR, 2021, pp. 8748–8763. [11] H. Liu, C. Li, Q. Wu et al., “Visual instruction tuning,” Advances in neural information processing systems, vol. 36, pp. 34 892–34 916, 2023. [12] H. Liu, C. Li, Y. Li et al., “Improved baselines with visual instruction tuning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024, pp. 26 296–26 306. [13] Y. Zhang, B. Li, h. Liu et al., “Llava-next: A strong zeroshot video understanding model,” April 2024. [Online]. Available: https://llava-vl.github.io/blog/2024-04-30-llava-next-video/
[14] J. Bai, S. Bai, S. Yang et al., “Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond,” arXiv preprint arXiv:2308.12966, 2023. [15] S. Bai, K. Chen, X. Liu et al., “Qwen2.5-vl technical report,” 2025. [Online]. Available: https://arxiv.org/abs/2502.13923 [16] S. Bai, Y. Cai, R. Chen et al., “Qwen3-vl technical report,” 2025. [Online]. Available: https://arxiv.org/abs/2511.21631 [17] Y. Kang, J. Hauswald, C. Gao et al., “Neurosurgeon: Collaborative intelligence between the cloud and mobile edge,” ACM SIGARCH Computer Architecture News, vol. 45, no. 1, pp. 615–629, 2017. [18] A. Mudvari, Y. Jiang, and L. Tassiulas, “Splitllm: Collaborative inference of llms for model placement and throughput optimization,” arXiv preprint arXiv:2410.10759, 2024. [19] D. Macario, H. Seferoglu, and E. Koyuncu, “Model-distributed inference for large language models at the edge,” in 2025 IEEE 31st International Symposium on Local and Metropolitan Area Networks (LANMAN), 2025, pp. 1–6. [20] A. Borzunov, D. Baranchuk, T. Dettmers et al., “Petals: Collaborative inference and fine-tuning of large models,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), D. Bollegala, R. Huang, and A. Ritter, Eds. Toronto, Canada: Association for Computational Linguistics, Jul. 2023, pp. 558–568. [Online]. Available: https: //aclanthology.org/2023.acl-demo.54/ [21] A. Mahendran and A. Vedaldi, “Understanding deep image representations by inverting them,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 5188–5196. [22] Y. Zhang, R. Jia, H. Pei et al., “The secret revealer: Generative modelinversion attacks against deep neural networks,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 253–261. [23] S. Jin, H. Wang, Z. Wang et al., “FaceObfuscator: Defending deep learning-based privacy attacks with gradient descent-resistant features in face recognition,” in 33rd USENIX Security Symposium (USENIX Security 24). Philadelphia, PA: USENIX Association, Aug. 2024, pp. 6849–6866. [Online]. Available: https://www.usenix.org/conference/ usenixsecurity24/presentation/jin-shuaifan [24] A. Hatamizadeh, H. Yin, H. R. Roth et al., “Gradvit: Gradient inversion of vision transformers,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 10 021–10 030. [25] H. Kazemi, A. Chegini, J. Geiping et al., “What do we learn from inverting clip models?” arXiv preprint arXiv:2403.02580, 2024. [26] C. Song and A. Raghunathan, “Information leakage in embedding models,” in Proceedings of the 2020 ACM SIGSAC conference on computer and communications security, 2020, pp. 377–390. [27] X. Pan, M. Zhang, S. Ji et al., “Privacy risks of general-purpose language models,” in 2020 IEEE Symposium on Security and Privacy (SP). IEEE, 2020, pp. 1314–1331. [28] C. Zhang, J. X. Morris, and V. Shmatikov, “Extracting prompts by inverting llm outputs,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 14 753– 14 777. [29] J. Morris, V. Kuleshov, V. Shmatikov et al., “Text embeddings reveal (almost) as much as text,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 12 448– 12 460. [30] W. Qu, Y. Zhou, Y. Wu et al., “Prompt inversion attack against collaborative inference of large language models,” in 2025 IEEE Symposium on Security and Privacy (SP). IEEE, 2025, pp. 1695–1712. [31] K. Xiu and S. Q. Zhang, “Caprecover: A cross-modality feature inversion attack framework on vision language models,” in Proceedings of the 33rd ACM International Conference on Multimedia, 2025, pp. 3808–3816. [32] N.-B. Nguyen, S.-T. Ho, K. J. Hao et al., “Do vision-language models leak what they learn? adaptive token-weighted model inversion attacks,” 2026. [Online]. Available: https://arxiv.org/abs/2508.04097 [33] Y. Goyal, T. Khot, D. Summers-Stay et al., “Making the v in vqa matter: Elevating the role of image understanding in visual question answering,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 6904–6913. [34] Z. Liu, P. Luo, X. Wang et al., “Deep learning face attributes in the wild,” in Proceedings of the IEEE international conference on computer vision, 2015, pp. 3730–3738.
[35] M.-E. Nilsback and A. Zisserman, “Automated flower classification over a large number of classes,” in 2008 Sixth Indian conference on computer vision, graphics & image processing. IEEE, 2008, pp. 722–729. [36] J. Thorne, A. Vlachos, C. Christodoulopoulos et al., “Fever: a largescale dataset for fact extraction and verification,” arXiv preprint arXiv:1803.05355, 2018. [37] T. Nguyen, M. Rosenberg, X. Song et al., “Ms marco: A humangenerated machine reading comprehension dataset,” 2016. [38] J. Lin, R. Zhang, F. Ganz et al., “Anycost gans for interactive image synthesis and editing,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021, pp. 14 986–14 996.