arXiv CS open access

Position Matters: Feature Inversion Attacks in ViT Split Inference with Token Reduction and Shuffling

2026 · arxiv_cs

arXiv CS · Papers · 2026

cryptography cybersecurity privacy security

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

Position Matters: Feature Inversion Attacks in ViT Split Inference with Token Reduction and Shuffling Stefano Leggio, Giulio Rossolini, Alessandro Biondi∗ Department of Excellence in Robotics & AI Scuola Superiore Sant’Anna Pisa, Italy

arXiv:2609.01232v1 [cs.CR] 1 Sep 2026

Abstract Vision Transformers (ViTs) are increasingly used in split-inference systems, where edge devices transmit intermediate token representations to a remote cloud. In this setting, token reduction lowers computation and communication costs, while token shuffling disrupts the spatial organization of the transmitted tokens, potentially limiting information leakage. However, their privacy benefits remain unclear against feature inversion attacks, which attempt to reconstruct the input from the transmitted embeddings. In this work, we show that, despite disrupting the spatial structure required by conventional reconstruction attacks, transmitted token embeddings retain substantial positional information. Based on this observation, we introduce the Spatially Aligned Reconstruction Attack (SARA), a unified pipeline that predicts token positions, restores their spatial layout, reconstructs missing embeddings using a feature-space masked autoencoder, and recovers the input image. Our results demonstrate that token shuffling provides only apparent privacy, as SARA largely reconstructs the original token organization. Token reduction offers stronger protection, but significant leakage persists when the retained tokens preserve sufficient semantic and positional information. Finally, we introduce a lightweight edge-side defense that removes positional embeddings and progressively adapts the edge-side transformer blocks through knowledge distillation. It substantially reduces attack performance against SARA, while preserving downstream task accuracy and requiring no changes to the cloud-side model.

CCS Concepts • Computing methodologies → Machine learning; • Security and privacy → Privacy protections.

Keywords Split Inference, Vision Transformers, Feature Inversion Attacks, Privacy-Preserving Machine Learning

Introduction

Vision Transformers (ViTs) [6] have become a reference architecture for computer vision, achieving strong performance across a wide range of tasks [22, 27, 33]. ViTs process an image by splitting it into a sequence of patch tokens, each encoding the visual content of a local image region together with information about its spatial position. This sequence-based representation makes intermediate embeddings naturally well suited to token-level manipulations, often without modifying the model parameters or retraining the downstream network. For example, token reduction methods [2, 18, 26, 28] ∗ Contact: stefano.leggio@santannapisa.it

drop or merge redundant and less informative tokens, thereby reducing the computational cost of subsequent Transformer blocks while preserving a favorable trade-off with task performance. In contrast, token shuffling methods [21, 37, 40] permute the token sequence, disrupting the explicit correspondence between individual tokens and the spatial layout of the input image. Although these operations are primarily introduced for efficiency or representation manipulation, they may also provide an apparent privacy benefit by limiting the amount of information exposed in intermediate features and/or concealing the correspondence between transmitted tokens and their original image locations. The combination of efficiency and potential privacy benefits makes these operations particularly appealing in split inference scenarios [12, 15, 30], where a neural network is partitioned between an edge device holding the private input and a remote cloud. In privacy-sensitive applications, the edge device processes the input through the initial portion of the model and transmits the resulting intermediate representation to the cloud, which completes the inference. Token manipulations can therefore be naturally applied on the edge side before transmission, reducing client-side computation and communication costs while potentially obscuring the spatial correspondence between the transmitted tokens and the original image regions. In this setting, it is reasonable to consider an honest-but-curious threat model [11, 12, 23], in which the cloud correctly performs the assigned inference computation but may also attempt to reconstruct the private input from the received features, as illustrated in Figure 1. However, the privacy provided by these token manipulations remains insufficiently investigated, despite being of crucial importance in split-inference scenarios. Conventional feature inversion attacks typically assume that intermediate representations preserve a fixed spatial organization [9, 11, 12, 17, 19, 38]. Consequently, their failure after token reduction or token shuffling may simply result from the violation of this assumption, rather than from a genuine reduction in information leakage. Indeed, the transmitted token embeddings may still retain sufficient semantic and positional information to recover their original spatial organization and reconstruct the private input. This motivates the need for reconstruction attacks specifically designed to operate on manipulated token representations. Paper contributions. In this work, we provide a systematic privacy-oriented analysis of token reduction and token shuffling in ViT-based split inference. We first show that intermediate token embeddings retain substantial positional information, even after passing through multiple ViT blocks, which enables a positionaware attacker to infer their original spatial location. Building on this observation, we introduce the Spatially Aligned Reconstruction

Cloud

Edge Smashed data Input image

Positional encoding

ViT transformer blocks

...

Head

...

Reconstructed image Patch embedding

Dropping

Merging

Attacker

Shuff ling

Prediction

Figure 1: Illustration of the threat model. The edge device transmits the smashed data ℎ to the cloud, where an honest-but-curious attacker ˜ attempts to output a reconstruction of the original input image 𝑥.

2.1

Attack (SARA), a unified reconstruction pipeline that predicts the original positions of the transmitted tokens, restores their spatial arrangement, reconstructs missing token embeddings through a feature-space masked autoencoder, and finally recovers the input image. This pipeline enables a consistent evaluation of information leakage under both token reduction and token shuffling. Experimental results show that token shuffling alone provides only a false sense of privacy, as its apparent benefits can be greatly disrupted by SARA. Token reduction offers stronger protection by discarding part of the representation, but introduces a privacy– utility trade-off due to the resulting degradation in task accuracy. Nevertheless, its privacy benefits remain limited when the retained tokens preserve sufficient semantic and positional information for SARA to infer the missing content. Motivated by these findings, we finally introduce a lightweight edge-side defense, specifically tailored to split inference settings, where adapting the cloud-side model should be avoided to accommodate multiple edge devices. The proposed mechanism provides a simple yet effective baseline that attenuates positional cues before transmission, and substantially reduces reconstruction leakage against ad-hoc attacks, while maintaining a favorable trade-off with task performance. Paper Structure. The remainder of the paper is organized as follows. Section 2 introduces the threat model for ViT-based split inference and reviews the token manipulation strategies considered in this work. Section 3 presents the proposed SARA attack pipeline and provides a detailed description of its components. Section 4 introduces a simple yet effective edge-side defense. Section 5 first evaluates the effectiveness of SARA against the considered token manipulation strategies and then assesses the benefits of the proposed defense. Finally, Sections 6 and 7 discuss the connections with the related literature and present the conclusions and limitations, respectively.

ViT tokenization and token operations

Let 𝑥 ∈ [0, 1]𝐶 ×𝐻 ×𝑊 be an input image. A ViT 𝑓 partitions 𝑥 into 𝑁 0 = 𝐻𝑊 non-overlapping image patches of size 𝑝 × 𝑝. Each patch 𝑝2 is then linearly projected into a 𝑑-dimensional embedding and combined with a learnable positional embedding (PE). The patch input sequence to the first transformer block is therefore ℎ (0) = ℎ (0),1 ; . . . ; ℎ (0),𝑁0 ∈ R𝑁0 ×𝑑 ,

(1)

where ℎ (0),𝑖 ∈ R𝑑 denotes the embedding associated with the 𝑖-th image patch, including its positional embedding. Note that, from the sequence ℎ (0) , we intentionally omit the classification token to denote only the patch-token representations.1 From a layer-level perspective, let the ViT consist of 𝐿 blocks {𝐵 1, . . . , 𝐵𝐿 }. In the original ViT architecture, without additional token-manipulation operations, each block preserves the sequence length such that the patch-token representation produced after block 𝐵 ℓ is ℎ (ℓ ) ∈ R𝑁ℓ ×𝑑 , with 𝑁 ℓ = 𝑁 0 for every ℓ ∈ {1, . . . , 𝐿}. Token-reduction methods modify ViT inference by progressively decreasing the number of tokens processed by subsequent transformer blocks. In particular, operations such as token dropping [28] and token merging [2] remove or aggregate selected tokens, respectively. Consequently, the sequence length becomes 𝑁𝑖 ≤ 𝑁 𝑗 for 𝑖 > 𝑗. For simplicity, we first consider a fixed token reduction policy, where 𝑟 denotes the number of patch tokens removed from the sequence, either by dropping or merging, after each transformer block output. Assuming that the same reduction amount 𝑟 is applied after each of the first ℓ transformer blocks, the number of patch tokens available in ℎ (ℓ ) is 𝑁 ℓ = max{0, 𝑁 0 − ℓ𝑟 }, where 𝑁 0 denotes the initial number of patch tokens. Once 𝑁 ℓ = 0, only the class token remains, and no further token reduction is applied in the subsequent blocks. Importantly, increasing 𝑟 improves computational and communication efficiency by reducing the number of processed and transmitted tokens. However, stronger reduction may improve

Preliminaries and Threat Model 1 The classification token is still processed by all ViT layers and is included in the

This section formalizes the split-inference setting under consideration, the token-manipulation operations applied at the edge, and the capabilities and objective of the adversary.

legitimate inference pipeline. We omit it from the reconstruction formulation because it does not have a direct correspondence with a spatial image patch and is not explicitly used by the considered reconstruction attack and pipelines. 2

privacy at the cost of downstream accuracy. To capture this tradeoff, in Section 5 we introduce a unified metric by jointly accounting for task utility and privacy leakage. In contrast, token shuffling does not change the number of transmitted tokens. Instead, it permutes their sequence order at the output of a selected block ℓ. Thus, when shuffling is applied without token reduction, it holds 𝑁 ℓ = 𝑁 0 , while the representation transmitted after block ℓ is an ordered permutation of the original token sequence.

2.2

3.1

ViT Split Inference

remaining 𝐿 −𝑘 blocks and returns the predicted logits 𝑓c(𝑘 ) ℎ (𝑘 ) . Consequently, when token reduction or shuffling is applied on the edge-side, the corresponding operations are included in 𝑓e(𝑘 ) , while the learned ViT parameters remain unchanged. The cloudside model therefore receives the manipulated token sequence and continues the inference procedure from the selected split point. As discussed in Section 1, these techniques are particularly relevant to split inference because they can improve computational and communication efficiency and, in the case of token shuffling, conceal the original spatial arrangement of the transmitted tokens.

Threat Model

We consider a honest-but-curious cloud that correctly executes the cloud-side portion of the ViT but attempts to infer private information about the edge’s input. For an input image 𝑥, the adversary observes the intermediate representation ℎ (𝑘 ) = 𝑓e(𝑘 ) (𝑥), with 𝑁𝑘 tokens embedding, transmitted at split point 𝑘. Depending on the edge-side configuration, ℎ (𝑘 ) may contain a reduced token sequence and/or a shuffled token sequence. Note that, in this threat model, the adversary cannot directly observe private edge-side information associated with a specific input, such as the original image, the spatial provenance of the retained or merged tokens, or the permutation used to shuffle the transmitted sequence. The adversary is passive and does not alter the inference protocol, the transmitted representation, or the legitimate prediction. Its objective is to reconstruct the private input from the observed smashed data. Following these assumptions, the attacker trains a reconstruction model 𝑔𝜃 : R𝑁𝑘 ×𝑑 → [0, 1]𝐶 ×𝐻 ×𝑊 using an auxiliary dataset n o𝑀 Daux = 𝑥𝑖 , ℎ𝑖(𝑘 ) , where parameters are learned by solving 𝑖=1

𝜃𝑔∗ = arg min 𝜃𝑔

𝑀 1 ∑︁ Lrec 𝑔𝜃 ℎ𝑖(𝑘 ) , 𝑥𝑖 , 𝑀 𝑖=1

Motivation and Intuition

Standard reconstruction attacks on intermediate representations typically train a convolutional decoder to recover the input image directly from the transmitted data ℎ (𝑘 ) [9, 11, 12, 17, 19, 38]. These approaches generally assume that the intermediate representation preserves a fixed spatial organization that can be mapped back to the original image via classic operations. However, token reduction and shuffling violate this assumption by altering the number or ordering of the transmitted tokens, potentially creating a misleading impression of improved privacy when evaluated against convolutional-based reconstructions. To illustrate this limitation, we evaluate a standard reconstruction decoder under two configurations of ViT-B/16 (setup details in Section 5.1): an aligned setting, in which tokens preserve their original spatial order, and a shuffled setting, in which their order is randomly permuted. As reported in the table in Figure 2 (top), shuffling reduces the average Structural Similarity Index Measure (SSIM) [36] from 0.700 to 0.246. As expected, a consistent drop across all split points confirms that conventional reconstruction decoders strongly depend on the original token arrangement. However, shuffling only changes the order of the transmitted tokens; it does not directly remove the semantic or positional information encoded in their representations. This observation naturally raises the question of whether shuffling operations can be inverted by inferring the original spatial position of each token. In principle, recovering the corresponding patch index would allow the tokens to be rearranged before reconstruction, thereby restoring the spatial structure and recovering the same reconstruction quality as in the aligned setting. To investigate this possibility, we train different token-position predictors to classify each intermediate token according to its original position in the input patch grid. As shown in Figure 2 (bottom), positional information remains highly accessible throughout the ViT. Even a linear predictor achieves nearly perfect accuracy in the early layers, although its performance decreases substantially for deeper representations. In contrast, the Transformer-based predictor maintains high accuracy through the layers.

We consider a split point 𝑘, with 1 ≤ 𝑘 < 𝐿, such that the edge executes the first 𝑘 transformer blocks and produces the intermediate representation ℎ (𝑘 ) = 𝑓e(𝑘 ) (𝑥), while the cloud executes the

2.3

Attack Pipeline

This section describes the Spatially Aligned Reconstruction Attack (SARA). We first motivate the idea by highlighting the importance of recovering positional information. We then present the steps as a unified approach for reconstructing images from features that have undergone token reduction or shuffling on the edge side.

(2)

3.2

Lrec denotes the reconstruction loss, e.g., a pixel-level mean square error (MSE). After training of 𝑔, given a smashed representation

Spatially Aligned Reconstruction Attack

We propose the Spatially Aligned Reconstruction Attack (SARA), a three-stage pipeline designed to improve the reconstruction of smashed representations ℎ (𝑘 ) affected by token reduction or shuffling. The overall pipeline is illustrated in Figure 3 and consists of three stages, which are described more technically as follows: (i) a Token Position Predictor, which estimates the original spatial position of each transmitted token and constructs a spatially aligned, full-length masked representation; (ii) a Masked Autoencoder, which

of an unseen private input, the attacker produces 𝑥˜ = 𝑔 ℎ (𝑘 ) . ˜ We assess privacy leakage through the similarity between 𝑥 and 𝑥, measured using pixel-level and perceptual reconstruction metrics (see Section 5.1). Higher reconstruction quality indicates that more information about the private input can be recovered from the transmitted token representation. 3

Split point

Avg.

Aligned Shuffled

0.92 0.18

0.89 0.21

0.84 0.22

0.79 0.24

0.74 0.25

0.69 0.26

0.65 0.27

0.61 0.27

0.56 0.27

0.53 0.27

0.49 0.27

0.70 0.25

Original

Aligned

where 𝜏 is a confidence threshold. If multiple tokens are assigned to the same position, we retain the one with the highest confidence. The spatially aligned representation is therefore constructed as

Shuffled

Accuracy

ℎ˜ 𝑗 =

100 80 60 40 20 0

Linear

5 6 7 8 ViT Layer

MLP

9 10 11

Transformer-based

imputes the representations associated with missing token positions; and (iii) a Decoder, which maps the restored smashed representation back to the input image through a convolutional reconstruction network. 3.2.1 Token Position Prediction and Placement. Given an unordered set of smashed tokens, the Token Position Predictor (TPP) estimates the original image-patch position of each token and rearranges the tokens accordingly to recover their spatial organization. For simplicity, we omit the split-point superscript and denote the smashed representation at a split point 𝑘 as ℎ = {ℎ 1, . . . , ℎ 𝑁𝑘 }, ℎ𝑖 ∈ R𝑑 , where 𝑁𝑘 is the number of tokens transmitted after token manipulation, 𝑁 0 is the number of image patches in the original token sequence, and 𝑑 is the token dimensionality. The TPP is implemented as a position classifier 𝐶 pos that assigns each transmitted token a probability distribution over the 𝑁 0 possible patch positions: Í𝑁 0 𝑃 = 𝐶 pos (ℎ) ∈ [0, 1] 𝑁𝑘 ×𝑁0 , 𝑗=1 𝑃𝑖,𝑗 = 1, where 𝑃𝑖,𝑗 denotes the predicted probability that token ℎ𝑖 originated from the 𝑗-th image patch. For each token, we define its predicted position and the corresponding confidence as max

𝑃𝑖,𝑗 ,

𝑐𝑖 =

max

𝑗 ∈ {1,...,𝑁 0 }

𝑃𝑖,𝑗 .

LMAE =

𝑖 ∗𝑗 = arg max 𝑐𝑖 , 𝑖 ∈ I𝑗

(5)

1 ∥M (ℎ mask ) − ℎ full ∥ 2𝐹 , 𝑁0 𝑑

(6)

where ∥ · ∥ 𝐹 denotes the Frobenius norm. By learning the dependencies among the visible token embeddings, the model infers the representations associated with missing spatial positions. When ℎ˜ contains no missing positions, the masked-autoencoder stage is ˜ bypassed, and we directly set ℎ rec = ℎ. 3.2.3 Image reconstruction. The completed smashed representation ℎ𝑟𝑒𝑐 is finally passed to an image reconstruction decoder, denoted by Dec, which reconstructs an approximation of the original input image 𝑥˜ = Dec(ℎ𝑟𝑒𝑐 ). In particular, the reconstructed patch embeddings in ℎ rec are first linearly projected and reshaped into a spatial grid. A convolutional decoder composed of upsampling blocks then progressively restores the original image resolution. As commonly done in reconstruction attacks [9, 11, 12, 17, 19, 38], the decoder is trained on an auxiliary image dataset available to the attacker and optimized by minimizing the mean squared error between the reconstructed image and the corresponding groundtruth input. Note that we train the decoder independently of the

(3)

The transmitted tokens are then placed into a full-length representation ℎ˜ ∈ R𝑁0 ×𝑑 . For each position 𝑗, we first identify the set of sufficiently confident tokens assigned to that position: I𝑗 = {𝑖 | 𝑝ˆ𝑖 = 𝑗 ∧ 𝑐𝑖 ≥ 𝜏 } ,

otherwise,

3.2.2 Reconstruction of Missing Tokens. The spatially aligned representation produced by the previous step, ℎ˜ ∈ R𝑁0 ×𝑑 , may still contain missing token embeddings, represented by zero vectors at their corresponding positions. Whenever such missing positions remain, we adopt the principle of masked auto-encoders (MAEs) [10] to recover the unavailable representations. In particular, a feature-space masked autoencoder, denoted by M, maps this incomplete representation to a complete reconstructed ˜ ∈ R𝑁0 ×𝑑 . representation: ℎ rec = M (ℎ) To do this, the model is trained The model is trained to recover the complete smashed representation from a partially observed version of it; a subset of token embeddings in a full smashed representation, denoted for simplicity as ℎ full ∈ R𝑁0 ×𝑑 , collected by the attacker from the auxiliary dataset, is randomly selected and replaced with zero vectors, producing a masked representation ˜ enℎ mask ∈ R𝑁0 ×𝑑 that emulates the incomplete representation (ℎ) countered at inference time. The model is then optimized using the MSE between the reconstructed and complete representations:

decoder on ImageNet when the transmitted ViT-B tokens retain their original spatial order (Aligned) or are randomly permuted (Shuffled); (center) illustration of the original input and reconstructions at split point 4 with a convolutional decoder; (bottom) Top-1 token-position accuracy across ViT layers for different predictors.

𝑗 ∈ {1,...,𝑁 0 }

if I𝑗 ≠ ∅,

where ℎ void = 0 ∈ R𝑑 denotes the void value. This operation restores the spatial indexing required by the subsequent reconstruction stages. Positions associated with removed tokens, low-confidence predictions, or unresolved assignments remain marked by the void value and are subsequently processed by the masked autoencoder. For training, the attacker uses an auxiliary image dataset to extract the corresponding smashed representations and simulate randomly shuffling across their token order. The TPP is then optimized using a cross-entropy loss, where the ground-truth label of each token corresponds to its original patch index.

Figure 2: (Top) SSIM achieved by a conventional convolutional

𝑝ˆ𝑖 = arg

  ℎ𝑖 ∗𝑗 ,   ℎ void, 

(4) 4

Smashed data TPP

MAE

Missing tokens?

YES

... NO

Token position prediction and placement

Reconstruction of missing tokens

Image reconstruction

Figure 3: Overview of the Spatially Aligned Reconstruction Attack (SARA). The pipeline consists of three stages: (1) token position prediction and spatial alignment, (2) missing-token reconstruction via a Masked Autoencoder, and (3) image reconstruction with a convolutional decoder.

Algorithm 1 Progressive Positionless Edge Finetuning

embeddings stage is removed, as shown in the following. Omitting the class token for simplicity, the initial teacher and student representations are therefore given by

Require: Pretrained edge 𝑓e(𝑘 ) , cloud model 𝑓c(𝑘 ) , unlabeled adaptation dataset DA , split point 𝑘, number of epochs 𝑁 epoch (𝑘 ) ← 𝑓e(𝑘 ) 2: Remove its positional embeddings 3: for 𝑗 = 1 to 𝑘 do 4: for 𝑒 = 1 to 𝑁 epoch do 5: for all mini-batches B ⊂ DA do 6: LKD ← 𝐷 KL 𝑓c(𝑘 ) ( 𝑓¯e(𝑘 ) (B)) 𝑓c(𝑘 ) (𝑓e(𝑘 ) (B)) 7: Update the 𝑗-th block of 𝑓¯e(𝑘 ) to minimize LKD 8: end for 9: end for 10: end for (𝑘 ) 11: return finetuned student edge 𝑓¯e

teacher : ℎ (0) ← 𝐸 patch (𝑥) + 𝐸 pos,

1: Initialize the student edge 𝑓¯e

where 𝐸 patch denotes the patch-embedding function and 𝐸 pos denotes the positional embeddings. Although directly removing the positional embeddings eliminates the primary source of positional information, it may substantially alter the intermediate representations and degrade the predictive performance of the model. We therefore recover task performance by progressively adapting the edge-side transformer blocks through knowledge distillation using the logit Kullback– Leibler divergence loss 𝐷 KL , which encourages the student edge to preserve features that allow achieving an output behavior of the original model. In particular, as shown by the outer loop in Algorithm 1, at the finetuning stage 𝑗 ∈ {1, . . . , 𝑘 }, only the parameters of the 𝑗-th student block are optimized. Previously adapted blocks remain fixed, whereas subsequent edge-side blocks retain their original pretrained parameters. The entire cloud-side network also remains fixed throughout the entire procedure, although gradients are backpropagated through it to optimize the edge-side blocks. Notably, this progressive block-wise adaptation provides a more stable optimization of the task objective than jointly fine-tuning the entire edge-side model. In the ablation study presented in Sections 5.3.3 and 5.3.4, we further validate this design choice and investigate the potential benefits and limitations of incorporating a min–max optimization strategy into the formulation.

MAE stage, allowing us to better isolate the benefits of the preceding stages and provide a fairer comparison with conventional reconstruction approaches based solely on a decoder.

student : ℎ (0) ← 𝐸 patch (𝑥),

Proposed Defense

To mitigate the high effectiveness of SARA, as demonstrated by the experimental results in Section 5, we introduce a simple yet effective edge-side defense that limits the attacker’s ability to infer the positional information retained in the smashed representations. The proposed defense pursues two objectives. First, it aims to attenuate the positional information encoded in the smashed representations, thereby reducing the effectiveness of the TPP stage in SARA. Second, it aims to preserve the task-level behavior of the original model without modifying the cloud-side ViT part. To achieve these objectives, we adopt a knowledge-distillation approach in which a student version of the original ViT is constructed by removing the positional embeddings from the edge-side model. Its transformer blocks are then progressively adapted using the original pretrained ViT as the teacher. The overall procedure is summarized in Algorithm 1 and described in detail below. We use the original pretrained model as a frozen teacher, composed of the edge-side component 𝑓e(𝑘 ) and the cloud-side component 𝑓c(𝑘 ) . The student differs from the teacher only in its edge-side component, denoted by 𝑓¯e(𝑘 ) . Before fine-tuning, 𝑓¯e(𝑘 ) is initialized with the same pretrained parameters as 𝑓e(𝑘 ) , but the positional

Experimental Results

In this section, we first evaluate the effectiveness of the proposed attack and then assess the benefits of introducing the proposed defense against both token reduction and shuffling. Before presenting the experimental results, we describe the experimental setup.

5.1

Experimental Settings

Models and datasets. We conduct all experiments on ImageNet1K [4] using ViT-B/16 [6] and MAE-B/16 [10]. Although the two models share the same ViT architecture, they differ in their pretraining strategies: ViT-B is trained using supervised classification, whereas MAE-B relies on self-supervised masked autoencoding and requires downstream finetuning. We therefore evaluate the pretrained ViT-B directly on ImageNet, while adapting MAE-B by fine-tuning its classification head and final Transformer block 5

Table 1: Architectural setup of SARA’s components.

for 30 epochs on the ImageNet-1K training set, with all preceding blocks kept frozen. The latter setup also allows us to explore a splitinference scenario in which fixed client representations can support different server-side tasks. Furthermore, as shown in the following experiments, MAE-B is more vulnerable to reconstruction attacks than the supervised ViT-B, even at deeper split points, likely due to its reconstruction-oriented pretraining. Token operations and metrics. We consider token shuffling and two representative token reduction methods. Specifically, we evaluate ToMe [2], which progressively merges 𝑟 tokens based on their similarity, and a random token dropping strategy [18, 26, 28], in which a fixed number 𝑟 of tokens is discarded after each ViT block. To evaluate image reconstruction quality, we employ several well-established vision metrics, namely the Structural Similarity Index Measure (SSIM) [36], Peak Signal-to-Noise Ratio (PSNR), and Feature Similarity Index Measure (FSIM) [41]. Higher SSIM, PSNR, and FSIM values indicate better reconstruction quality and, consequently, greater information leakage and weaker privacy protection. Finally, particularly for token reduction methods, increasing the reduction parameter may improve privacy while causing a substantial degradation in classification accuracy. To jointly evaluate privacy and utility, we introduce the Privacy–Utility Reconstruction Index (PURI). Utility is measured as the classification accuracy relaAcc tive to the unmodified baseline: 𝑈 = Acc . Privacy is quantified as base the relative degradation in reconstruction quality compared with the baseline attack, which applies no token operations and uses a convolutional decoder at the considered split point. Accordingly, the privacy score is defined as 1 SSIM PSNR FSIM 𝑃 = max 0, 1 − + + . (7) 3 SSIMbase PSNRbase FSIMbase

𝑈 ·𝑃 , 𝜆 · 𝑃 + (1 − 𝜆) · 𝑈

Stage

Configuration

TPP

Projection Encoder Head

Linear 768 → 384 Transformer Encoder ×2, 𝑑 = 384 Linear 384 → 196 + Softmax

MAE

Input encoding Encoder Head

MAE positional embedding, 𝑑 = 768 Transformer Encoder ×4, 𝑑 = 768 LayerNorm + Linear 768 → 768

Decoder

Reshape Upsampling Output

196 × 768 → 768 × 14 × 14 ConvT: 768 → 256 → 128 → 64 → 32 Conv 32 → 3

are detailed in Table 1. Their configurations were selected based on the best performance observed in preliminary experiments; for example, the TPP architecture was chosen according to the results reported in Figure 2). Regarding the confidence threshold 𝜏 of the TPP introduced in Eq. 4, in both the attack and defense preliminary evaluation, we did not observe substantial differences in reconstruction performance when varying 𝜏. Therefore, for simplicity, all reported experiments use 𝜏 = 0.0, and so every token is assigned to its most likely predicted position. Regarding the setup used for the proposed defense, each transformer block is fine-tuned using the ImageNet-1K training set for 50 epochs with a learning rate of 10−4 and temperature 4 in the 𝐷 KL . Both attack and defense performance are evaluated on the complete ImageNet-1K validation set. Every component of the SARA pipeline is trained independently for each split point. When a defense is applied, all components of SARA are retrained referring to the defended model.

Accordingly, 𝑃 = 0 indicates no privacy improvement over the baseline, whereas larger values indicate a stronger degradation in reconstruction quality. Finally, PURI combines utility and privacy through a weighted harmonic mean: PURI𝜆 =

Component

5.2

Attack results

In this first part of the experiments, we evaluate SARA under token shuffling and token reduction strategies. The main objective is to assess whether our attack remains capable of reconstructing the original input despite these token operations.

(8)

5.2.1 Shuffling. As illustrated in the motivation presented in Section 3, inputs subjected to token shuffling could not be reconstructed using the convolutional decoder. In contrast, as shown in Fig. 4, SARA can fully reconstruct the input image when token shuffling is applied to both ViT and MAE models, achieving reconstruction quality comparable to that of the baseline across multiple split points. This result is expected because token shuffling alters only the order of the tokens, while their original spatial positions can be reliably inferred even at very deep split points. We also observe that the reconstruction quality of MAE-B/16 remains nearly constant as the split point moves to deeper blocks. This finding reinforces the observation made in the experimental setup regarding the importance of accounting for potential privacy leakage in models pretrained with reconstruction-based objectives.

where 𝜆 ∈ [0, 1] controls the relative importance assigned to utility and privacy. We set 𝜆 = 0.7, thereby assigning greater importance to utility preservation, which reflects the practical requirement that a privacy-preserving method should maintain high classification accuracy while reducing reconstruction leakage. Attack and Defense Setup. Regarding the setup of SARA, each component is trained considering Daux a randomly selected 25% subset of the ImageNet-1K training set to limit computational cost. The TPP and image decoder are trained for 20 epochs, whereas the MAE is trained for 10 epochs. We use a learning rate of 10−4 for the TPP and MAE and 10−3 for the image decoder. Consistent with the discussion in Section 3.2, the TPP is trained on randomly shuffled smashed representations generated using different random seeds. The MAE is trained on randomly masked smashed representations to reconstruct the missing tokens, whereas the decoder is trained on the original, unshuffled and unreduced smashed representations, following the same setup as the baseline convolutional decoder. The architectures of all SARA components

Takeaway. Token shuffling alone does not defend against SARA, since the original token order remains recoverable even at deep ViT split points. 6

Baseline

Shuffling + SARA

Shuffling + decoder

k=2

k=4

k=6

k=8

0.5

0.8

0.5

0.8

0.2 0.0

Split block (k)

(a) ViT-B/16

0.2

0.1

0.0

0.0 2

PURI

0.4

0.3

0.3 0.2 0.1 0.0

Split block (k)

(b) MAE-B/16

(a) ViT-B/16 (Dropping) 0.5

shuffling applied (reconstruction upper bound). Shuffling + decoder applies the convolutional decoder directly to the shuffled tokens. Shuffling + SARA is our attack against shuffled tokens.

0.4

PURI

Figure 4: SSIM under token shuffling. Baseline is a client with no

(b) MAE-B/16 (Dropping) 0.5 0.4

0.3 0.2 0.1

0.3 0.2 0.1 0.0

0.0 20

Tokens reduced per block (r)

5.2.2 Token reductions. We first evaluate how effectively SARA reconstructs inputs affected by ToMe and random token dropping under different layer-wise reduction amounts, denoted by 𝑟 . Figure 5 reports the PURI scores, defined in Equation 8, for reduction amounts ranging from 5 to 90. For each curve, the value of 𝑟 that maximizes the PURI score is highlighted. Figure 5 reveals a clear trend. At low reduction amounts, the transmitted representation remains close to the original one and also, especially for shallow split points, retains substantial information about the input. This helps preserve accuracy but provides limited privacy, resulting in a low PURI score. As the reduction amount increases, privacy improves, while accuracy gradually decreases. This degradation is generally less pronounced at shallower split points, where token reduction is applied across fewer transformer blocks. Importantly, optimal values of 𝑟 at each split point identify the operating points that provide the best trade-off between downstream accuracy and privacy against reconstruction attacks. We therefore use the optimal configuration as representative reduction settings to analyze reconstruction quality and downstream task performance in greater detail in Figure 6. In this analysis, the first row shows the effect of token reduction considering of each split point on downstream classification accuracy. As expected, selecting optimal setups for PURI allow to have a good balance of accuracy performance, even when addressing deeper split points (where optimal 𝑟 is clearly lower than shallow ones, as shown in Figure 5), where in all the case stay close to the baseline value. The second row reports instead the reconstruction quality achieved by the SARA attack, measured using SSIM. For ViT-B/16, the attack results in only moderate reconstruction degradation, particularly at the final split points of the network. In contrast, for MAE-B/16, reconstruction quality remains remarkably stable across the network depth, with consistently high SSIM values comparable to those obtained at the second split point of ViT-B/16. A graphical illustration across all token operations addressed is shown in Figure 7. Regarding the comparison between the two token-reduction approaches, the considered metrics do not reveal a clear distinction, likely because they are agnostic to the most semantically relevant regions of the image.

Tokens reduced per block (r)

PURI

0.4

0.6

PURI

SSIM

0.4 0.6

k = 10

Tokens reduced per block (r)

(d) MAE-B/16 (ToMe)

Figure 5: PURI as a function of the reduction amounts 𝑟 for Random Dropping (top) and ToMe (bottom), on ViT-B/16 and MAE-B/16. Baseline

Dropping

ToMe

0.8

Accuracy

0.6 0.6 0.4

0.2

0.2 0.0

0.4

0.0

Split block (k)

(a) Accuracy - ViT-B/16

0.8

0.6

SSIM

(b) Accuracy - MAE-B/16

0.8

0.4 0.2 0.0

Split block (k)

0.6 0.4 0.2 0.0

Split block (k)

(d) SSIM - MAE-B/16

Figure 6: Classification accuracy (top row) and reconstruction quality (bottom row) at optimal 𝑟 , on ViT-B/16 and MAE-B/16. The dash line refers to the accuracy without reductions.

Takeaway. Token reduction substantially degrades SARA’s reconstructions only at the later split points of ViT-B/16. In contrast, for MAE-B/16, the attack continues to produce reconstructions with an SSIM above 0.5 across all split depths. 7

Shuffle

ToMe

Dropping

k=2

ViT-B/16

Baseline

k=4

k=6

PURI

0.4

MAE-B/16

0.4

0.2

0.0

0.0 20

(a) ViT-B/16 (Dropping)

PURI

0.4

0.0 20

Tokens reduced per block (r) 0.8

0.6

SSIM

0.8

0.4 0.2 0.0

Tokens reduced per block (r)

(d) MAE-B/16 (ToMe)

Figure 9: PURI as a function of the reduction amounts 𝑟 for Random

0.4

Dropping (top) and ToMe (bottom), on defended models.

0.2

0.0

Split block (k)

(a) ViT-B/16

Split block (k)

5.3.2 Token reductions. For the token-reduction analysis, we first identify the optimal 𝑟 at each split point according to the PURI metric, following the procedure described in the previous experiments. The results are reported in Figure 9, where, compared with the undefended model in Figure 5, the optimal value of 𝑟 is consistently lower, and the PURI curves are already close to their maximum at the smallest tested reduction amounts. This indicates that attenuating positional information substantially improves privacy against SARA even before applying aggressive token reduction. Figure 10 reports the classification accuracy and reconstruction quality obtained at the optimal 𝑟 of each split point considered. The top panel also shows the accuracy loss introduced by the defense itself, with the dotted line indicating the performance of the original undefended model. This degradation is limited and is mainly attributable to the removal of positional embeddings from the client-side model. Considering the accuracy drop induced by token reductions, the overall accuracy degradation is smaller than without the defense, which is mainly because the optimal reduction amounts are lower, allowing more tokens to be retained while preserving a favorable privacy–utility trade-off. As shown in the bottom panel of Figure 10, the reconstruction quality remains below an SSIM of 0.4 across all split points for both ViT-B/16 and MAE-B/16. The increasing SSIM trend previously observed for MAE-B/16 under the shuffling-only setting remains visible but is substantially attenuated. This suggests that combining positional-information attenuation with token reduction limits the spatial information available in deeper MAE-B/16 representations, thereby reducing the effectiveness of the reconstruction attack. Illustrations of the defense benefits are shown in Figure 11.

(b) MAE-B/16

Figure 8: Reconstruction quality (SSIM) under token shuffling for models adapted using the proposed defense.

5.3

0.4

0.2

0.0

Shuffling + SARA

0.6

0.2

Baseline (no shuffling)

(b) MAE-B/16 (Dropping)

0.6

at split point 𝑘 = 4 for ViT-B/16 and MAE-B/16. For ToMe and Dropping, we used the optimal 𝑟 that maximizes PURI.

Tokens reduced per block (r)

Figure 7: Reconstructed images from intermediate representations

k = 10

0.6

0.2

SSIM

k=8

Defense results

In this part of the experimental evaluation, we assess the effectiveness of the proposed defense against the SARA attack. 5.3.1 Shuffling. We first investigate whether combining shuffling with our defense can be effective to mitigate SARA. Figure 8 reports the reconstruction quality achieved by the attacker under this setting. For ViT-B/16, the proposed defense consistently reduces the attacker’s reconstruction quality across all split points, with the SSIM remaining below 0.4. This represents a substantial improvement over the baseline, indicating that attenuating the positional encoding significantly enhances the effectiveness of shuffling. For MAE-B/16, the defense also degrades reconstruction quality, although the attack becomes more effective at deeper split points. This seemingly counterintuitive trend reflects the MAE training objective, which encourages the model to preserve spatial relationships for masked-patch reconstruction. Consequently, unlike ViT-B/16, whose deeper features become increasingly classification-specific, MAE-B/16 retains stronger spatial structure that the attacker can exploit even after positional information has been attenuated. We further study this problem and ad-hoc alternatives for the defense proposed in Section 4. 8

Baseline (No reductions)

Dropping

Table 2: PURI scores on token shuffling, with a simple PE-removal

ToMe

finetuning → and progressive PE-removal finetuning (Alg.1). 0.8

(a) ViT-B/16

Accuracy

0.6 0.6 0.4

0.4

0.2

0.2 0.0

0.0

0.8

0.6

0.4

0.2

0.0

𝑘

PURI

2 4 6 8 10

0.705 → 0.715 0.601 → 0.672 0.573 → 0.623 0.539 → 0.579 0.488 → 0.518

2 4 6 8 10

0.715 → 0.690 0.605 → 0.685 0.542 → 0.641 0.568 → 0.638 0.621 → 0.627

Table 3: Each entry reports Defense → Defense+Adversarial. Lower SSIM and higher accuracy are better; the best values are in bold. ViT-B/16

Split block (k)

PURI

0.4

0.2

𝑘

(b) Accuracy - MAE-B/16

SSIM

(a) Accuracy - ViT-B/16

Split block (k)

(b) MAE-B/16

Split block (k)

(d) SSIM - MAE-B/16

MAE-B/16

𝒌

SSIM

Acc.

SSIM

Acc.

2 4 6 8

0.370 → 0.290 0.385 → 0.280 0.378 → 0.282 0.355 → 0.211

78.3% → 71.9% 80.9% → 72.2% 81.4% → 70.5% 81.3% → 68.4%

0.384 → 0.355 0.419 → 0.373 0.498 → 0.390 0.501 → 0.377

62.6% → 63.7% 66.6% → 67.4% 70.7% → 70.1% 72.0% → 71.0%

Figure 10: Accuracy (top row) and reconstruction quality (bottom row) at optimal 𝑟 , on defended ViT-B/16 and MAE-B/16. The dash line refers to the accuracy without defense and reductions.

Shuffle

ToMe

Dropping

ViT-B/16

Baseline

more stable results and a better privacy–utility trade-off than directly finetuning the entire edge-side model, even when both approaches use the same total number of training epochs. By updating one transformer block at a time, the optimization better preserves compatibility with the fixed cloud-side network and limits abrupt changes in the intermediate representations, resulting in improved performance across most split points.

MAE-B/16

5.3.4 On Deep Layers of MAE. The previous analysis highlighted a limitation of the proposed defense for MAE-B/16. Under token shuffling, the reconstruction quality may increase at deeper split points because MAE representations retain spatial relationships even after removing PE. To address this limitation, we investigate an adversarial variant of the defense in Algorithm 1. In addition to progressively adapting the edge-side transformer blocks without positional embeddings, we explicitly discourage the resulting smashed representations from revealing token positions. To this end, we introduce an auxiliary TPP, denoted by 𝐶 pos , which is trained to recover the original patch indices, while the student client is trained to hinder its predictions. Let 𝜃¯e and 𝜃 pos denote the parameters of the student edge 𝑓¯𝑒 and the auxiliary TPP, respectively. We replace the distillation-only objective in Algorithm 1 with the following adversarial objective, while the pipeline and progressive finetuning are the same:

Figure 11: Reconstructed images from intermediate representations at split point 𝑘 = 4 for defended ViT-B/16 and MAE-B/16. Reconstructions for ToMe and Dropping are shown using the optimal 𝑟 .

Takeaway. The proposed defense introduces only a small accuracy drop relative to the original model. Under token shuffling, the defense substantially reduces the attacker’s reconstruction capability across all split points, although for MAEB/16 it slightly increases at deeper splits. When combined with token reduction, it consistently degrades reconstruction across all split points for both ViT-B/16 and MAE-B/16.

min max E B ⊂ DA LKD (B) − 𝜆pos LTPP (B) , 𝜃¯e

(9)

𝜃 pos

where LKD preserves compatibility with the fixed server-side model (same as used in line 6 of Alg.1), LTPP is the cross-entropy loss for token-position classification, and 𝜆pos controls the trade-off between task preservation and positional-information suppression. As shown in Table 3, adversarial training effectively addresses this issue by stabilizing the attacker’s reconstruction capability across all ViT-MAE split points. In the proposed defense without

5.3.3 Benefits of progressive finetuning. As shown in Table 2, the progressive finetuning strategy in Algorithm 1 generally yields 9

Defense

costly or unstable adversarial training. In contrast, our proposed defense operates only on the client side without introducing unstable learning trends and leaves the cloud-side model unchanged.

Defense + Adversarial

k=6

Baseline

Token-based privacy mechanisms. Transformer architectures enable lightweight manipulations of intermediate representations at the token level. Token shuffling alters the order of transmitted tokens by exploiting the permutation-related properties of Transformer blocks [37] and has been investigated as a privacyenhancing mechanism to hinder input reconstruction while preserving inference performance [40]. Token reduction instead discards or merges tokens and has been primarily studied as an efficiency mechanism, with only limited analyses of its privacy implications [9]. Related works have also explored the removal or anonymization of selected image regions [1] and the application of established defenses, such as differential privacy, to intermediate representations [34]. Unlike cryptographic or learned defenses, token shuffling and reduction can potentially be applied without retraining or modifying the cloud-side model. However, their low overhead does not necessarily guarantee privacy.

Figure 12: Reconstructed image in MAE-B/16 under shuffling. Comparison with baseline, defense w/o and w/ adversarial training.

adversarial training, the reconstruction quality under token shuffling, measured by SSIM, increases from 0.384 at split point 𝑘 = 2 to 0.501 at split point 𝑘 = 8, corresponding to an increase of 0.117. In contrast, with adversarial training, the SSIM increases only from 0.355 to 0.377. As a qualitative reference, we select split point 𝑘 = 6 to visualize the effect of adversarial training on the attacker’s reconstruction quality. As shown in Fig. 12, the reconstruction obtained against the adversarially trained defense is noticeably less recognizable than that obtained with the standard defense. Despite these benefits, the adversarial formulation can introduce a non-negligible accuracy drop, particularly for ViT-B/16, due to the instability of the min–max optimization. We also omit the results for 𝑘 = 10 from Table 3, as training at this last split point exhibited stability issues. For this reason, we retain the simpler progressive finetuning strategy (Alg. 1) as our primary defense. A more robust integration of an adversarial regularization is left for future work.

Conclusions and limitations

In this work, we presented a comprehensive analysis of token reduction and token shuffling from a privacy perspective. We also proposed SARA, a novel attack pipeline for evaluating the privacy of these approaches. Our analysis showed that token reduction provides a certain degree of feature obfuscation. However, for token shuffling and particularly in the case of MAE-B/16, SARA was still able to produce high-quality reconstructions, highlighting the limitations of these techniques as standalone privacy-preserving mechanisms. Motivated by these findings, we proposed a defense mechanism that attenuates positional information through a progressive fine-tuning strategy aimed at removing the positional encoding on the client side. Our experimental results demonstrate that this defense substantially improves the obfuscation of the intermediate features, reducing the effectiveness of FIAs. Despite the promising results, we acknowledge several limitations. First, we evaluate utility only in terms of downstream classification accuracy, although the proposed fine-tuning strategy is task-agnostic. Extending the evaluation to other computer vision tasks, such as semantic segmentation, is therefore left for future work. Second, investigating whether SARA and the proposed defense can be effectively adapted to other domains, particularly language models, represents another promising direction. Finally, more sophisticated token-reordering strategies could be explored to achieve a better privacy-utility trade-off and to assess their robustness against adaptive reconstruction attacks.

Related work

Feature inversion attacks and defenses. Although split inference keeps the raw input on the edge device, the intermediate representations transmitted to the cloud may still leak substantial information about the private input. This leakage can be exploited through model inversion attacks (MIAs) [39], more specifically referred to as feature inversion attacks (FIAs) in split-inference settings [11, 12, 19, 29, 32]. These attacks typically train a reconstruction model to map intermediate features back to the input space, often relying on convolutional decoders that exploit local spatial relationships and assume a fixed correspondence between feature locations and image regions. This assumption, however, is violated when ViT tokens are shuffled or reduced, making conventional FIAs poorly suited to evaluating manipulated token representations. Existing defenses can broadly be grouped into cryptographic, data-modification, and learned approaches [17, 38]. Cryptographic methods, including homomorphic encryption [7], secure multiparty computation [25], and function secret sharing [3], provide strong guarantees but generally incur substantial computational and communication overhead [13, 16, 20]. Trusted Execution Environments offer a hardware-assisted alternative [31, 35]. Data-modification methods protect transmitted representations through mechanisms such as differential privacy and quantization [8, 23], often introducing a privacy–utility trade-off. Learned approaches instead optimize intermediate representations to preserve task-relevant information while suppressing sensitive content [5, 14, 24], but may require

References [1] Nazia Aslam, Abhisek Ray, Joakim Bruslund Haurum, Lukas Esterle, and Kamal Nasrollahi. 2026. From Pixels to Privacy: Temporally Consistent Video Anonymization via Token Pruning for Privacy Preserving Action Recognition. arXiv:2603.26336 [cs.CV] [2] Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. 2023. Token Merging: Your ViT But Faster. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 23217–23226. [3] Elette Boyle, Niv Gilboa, and Yuval Ishai. 2015. Function Secret Sharing. In Advances in Cryptology – EUROCRYPT 2015 (Lecture Notes in Computer Science, Vol. 9057). Springer, 337–367. doi:10.1007/978-3-662-46803-6_12 10

[23] Yuzhe Luo, Zhen Zhang, Jiacheng Li, Yu Wang, and Yao Chen. 2024. PrivacyPreserving Compression for Efficient Collaborative Inference. In Proceedings of the IEEE 30th International Conference on Parallel and Distributed Systems (ICPADS). IEEE, 142–151. doi:10.1109/ICPADS63350.2024.00028 [24] Fatemehsadat Mireshghallah, Mohammadkazem Taram, Ali Jalali, Ahmed Taha Elthakeb, Dean M. Tullsen, and Hadi Esmaeilzadeh. 2021. Not All Features Are Equal: Discovering Essential Features for Preserving Prediction Privacy. In Proceedings of the Web Conference 2021 (WWW). Association for Computing Machinery, 669–680. doi:10.1145/3442381.3449965 [25] Payman Mohassel and Yupeng Zhang. 2017. SecureML: A System for Scalable Privacy-Preserving Machine Learning. In Proceedings of the IEEE Symposium on Security and Privacy (SP). IEEE, San Jose, CA, USA, 19–38. doi:10.1109/SP.2017.12 [26] Lorenzo Papa, Paolo Russo, Irene Amerini, and Luping Zhou. 2024. A Survey on Efficient Vision Transformers: Algorithms, Techniques, and Performance Benchmarking. IEEE Transactions on Pattern Analysis and Machine Intelligence 46, 12 (Dec. 2024), 7682–7700. doi:10.1109/TPAMI.2024.3392941 [27] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning. PmLR, 8748–8763. [28] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. 2021. DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification. In Advances in Neural Information Processing Systems (NeurIPS). [29] Maria Rigaki and Sebastian Garcia. 2024. A Survey of Privacy Attacks in Machine Learning. Comput. Surveys 56, 4, Article 101 (2024). doi:10.1145/3624010 [30] Giulio Rossolini, Fabio Brau, Alessandro Biondi, Battista Biggio, and Giorgio Buttazzo. 2025. Exploiting edge features for transferable adversarial attacks in distributed machine learning. Internet of Things 34 (2025), 101795. [31] Mohamed Sabt, Mohammed Achemlal, and Abdelmadjid Bouabdallah. 2015. Trusted Execution Environment: What It Is, and What It Is Not. In Proceedings of the IEEE Trustcom/BigDataSE/ISPA, Vol. 1. IEEE, 57–64. [32] Ahmed Salem, Giovanni Cherubin, David Evans, Boris Köpf, Andrew Paverd, Anshuman Suri, Shruti Tople, and Santiago Zanella-Béguelin. 2023. SoK: Let the Privacy Games Begin! A Unified Treatment of Data Inference Privacy in Machine Learning. arXiv preprint arXiv:2212.10986 (2023). [33] Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. 2025. Dinov3. arXiv preprint arXiv:2508.10104 (2025). [34] Praneeth Vepakomma, Otkrist Gupta, Tristan Swedish, and Ramesh Raskar. 2018. Split learning for health: Distributed deep learning without sharing raw patient data. arXiv:1812.00564 [cs.LG] https://arxiv.org/abs/1812.00564 [35] Yulong Wang and Ahmed Habib. 2025. Protect Data Confidentiality for OnDevice Machine Learning Through Split Inference. In Proceedings of the 10th International Conference on Fog and Mobile Edge Computing (FMEC). IEEE, 290– 297. doi:10.1109/FMEC65595.2025.11119383 [36] Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. 2004. Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE Transactions on Image Processing 13, 4 (April 2004), 600–612. [37] Hengyuan Xu, Liyao Xiang, Hangyu Ye, Dixi Yao, Pengzhi Chu, and Baochun Li. 2024. Permutation Equivariance of Transformers and Its Applications. arXiv preprint arXiv:2304.07735 (2024). arXiv:2304.07735 [cs.CR] [38] Mengda Yang, Ziang Li, Juan Wang, Hongxin Hu, Ao Ren, Xiaoyang Xu, and Wenzhe Yi. 2022. Measuring Data Reconstruction Defenses in Collaborative Inference Systems. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 35. Article 934. [39] Ziqi Yang, Bin Shao, Bohan Xuan, Ee-Chien Chang, and Fan Zhang. 2020. Defending Model Inversion and Membership Inference Attacks via Prediction Purification. arXiv preprint arXiv:2005.03915 (2020). [40] Dixi Yao, Liyao Xiang, Hengyuan Xu, Hangyu Ye, and Yingqi Chen. 2022. PrivacyPreserving Split Learning via Patch Shuffling over Transformers. In Proceedings of the IEEE International Conference on Data Mining (ICDM). IEEE, Orlando, FL, USA, 638–647. doi:10.1109/ICDM54844.2022.00074 [41] Lin Zhang, Lei Zhang, Xuanqin Mou, and David Zhang. 2011. FSIM: A Feature Similarity Index for Image Quality Assessment. IEEE Transactions on Image Processing 20, 8 (Aug. 2011), 2378–2386. doi:10.1109/TIP.2011.2109730

[4] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A Large-Scale Hierarchical Image Database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 248–255. doi:10.1109/CVPR.2009.5206848 [5] Shiwei Ding, Lan Zhang, Miao Pan, and Xiaoyong Yuan. 2024. PATROL: PrivacyOriented Pruning for Collaborative Inference Against Model Inversion Attacks. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE, 4704–4713. doi:10.1109/WACV57701.2024.00465 [6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An Image Is Worth 16 × 16 Words: Transformers for Image Recognition at Scale. In Proc. of the International Conference on Learning Representations (ICLR). [7] Nathan Dowlin, Ran Gilad-Bachrach, Kim Laine, Kristin Lauter, Michael Naehrig, and John Wernsing. 2016. CryptoNets: Applying Neural Networks to Encrypted Data with High Throughput and Accuracy. In Proceedings of the 33rd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 48). PMLR, 201–210. [8] Cynthia Dwork. 2006. Differential Privacy. In Automata, Languages and Programming, Michele Bugliesi, Bart Preneel, Vladimiro Sassone, and Ingo Wegener (Eds.). Lecture Notes in Computer Science, Vol. 4052. Springer, Berlin, Heidelberg, 1–12. doi:10.1007/11787006_1 [9] Omar Erak, Omar Alhussein, Hatem Abou-Zeid, Mehdi Bennis, and Sami Muhaidat. 2025. Adaptive Token Merging for Efficient Transformer Semantic Communication at the Edge. arXiv:2509.09955 [10] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2022. Masked Autoencoders Are Scalable Vision Learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 16079– 16088. doi:10.1109/CVPR52688.2022.01553 [11] Zecheng He, Tianwei Zhang, and Ruby B. Lee. 2019. Model Inversion Attacks Against Collaborative Inference. In Proceedings of the 35th Annual Computer Security Applications Conference (ACSAC). Association for Computing Machinery, 148–162. doi:10.1145/3359789.3359824 [12] Zecheng He, Tianwei Zhang, and Ruby B. Lee. 2021. Attacking and Protecting Data Privacy in Edge–Cloud Collaborative Inference Systems. IEEE Internet of Things Journal 8, 12 (2021), 9706–9716. doi:10.1109/JIOT.2020.3022358 [13] Israt Jarin and Birhanu Eshete. 2021. PRICURE: Privacy-Preserving Collaborative Inference in a Multi-Party Setting. In Proceedings of the 2021 ACM Workshop on Security and Privacy Analytics. Association for Computing Machinery, 25–35. doi:10.1145/3445970.3451156 [14] Jonghu Jeong, Minyong Cho, Philipp Benz, and Tae-hoon Kim. 2023. Noisy Adversarial Representation Learning for Effective and Efficient Image Obfuscation. In Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence (Proceedings of Machine Learning Research, Vol. 216), Robin J. Evans and Ilya Shpitser (Eds.). PMLR, 953–962. [15] Yiping Kang, Johann Hauswald, Cao Gao, Austin Rovinski, Trevor Mudge, Jason Mars, and Lingjia Tang. 2017. Neurosurgeon: Collaborative Intelligence Between the Cloud and Mobile Edge. In Proceedings of the Twenty-Second International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). Association for Computing Machinery, New York, NY, USA, 615–629. doi:10.1145/3037697.3037698 [16] Tanveer Khan, Mindaugas Budzys, and Antonis Michalas. 2024. Make Split, Not Hijack: Preventing Feature-Space Hijacking Attacks in Split Learning. In Proceedings of the 29th ACM Symposium on Access Control Models and Technologies (SACMAT). Association for Computing Machinery, 19–30. doi:10.1145/3649158. 3657039 [17] Tanveer Khan and Antonis Michalas. 2026. Oops!... They Stole It Again: Attacks on Split Learning. In Proceedings of the 18th ACM Workshop on Artificial Intelligence and Security (AISec ’25). Association for Computing Machinery, 123–135. doi:10.1145/3733799.3762972 [18] Minchul Kim, Shangqian Gao, Yen-Chang Hsu, Yilin Shen, and Hongxia Jin. 2024. Token Fusion: Bridging the Gap between Token Pruning and Token Merging. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). doi:10.1109/WACV57701.2024.00141 [19] Wa-Kin Lei, Jun-Cheng Chen, and Shang-Tse Chen. 2025. DRAG: Data Reconstruction Attack Using Guided Diffusion. In Proceedings of the 42nd International Conference on Machine Learning (ICML). Article 1325. [20] Dong Li, Anupam Chattopadhyay, Qianyu Li, Qingguo Lü, Jiahui Wu, Tao Xiang, and Xiaofeng Liao. 2026. CryptDNN: A Fast Privacy-Preserving Deep Neural Network Inference Architecture Based on Cloud–Edge–Client Collaboration. IEEE Transactions on Network Science and Engineering 13 (2026), 2726–2740. doi:10. 1109/TNSE.2025.3620373 [21] Zhengyi Li, Yakai Wang, Kang Yang, Yu Yu, Jiaping Gui, Yu Feng, Ning Liu, Minyi Guo, and Jingwen Leng. 2026. On the (In-)Security of the Shuffling Defense in the Transformer Secure Inference. arXiv preprint arXiv:2605.04901 (2026). [22] Yang Liu, Yao Zhang, Yixin Wang, Feng Hou, Jin Yuan, Jiang Tian, Yang Zhang, Zhongchao Shi, Jianping Fan, and Zhiqiang He. 2022. A Survey of Visual Transformers. arXiv preprint arXiv:2111.06091 (2022).

Declaration on the Use of Generative AI During the preparation of this manuscript, the authors used generative AI tools (ChatGPT-5.3 Instant and Claude Opus 4.8) exclusively to assist with grammar correction, language refinement and code generation. 11

Baseline

Shuffle

ToMe

Dropping

Table 4: Optimal reduction parameter 𝑟 and corresponding number

k=2

of transmitted tokens for the two architectures at 𝜆 = 0.7. Each entry reports 𝑟 /𝑇 , where𝑇 is the number of transmitted tokens. (a) Results without defense. (b) Results with proposed defense.

k=6

(a) Without Defense.

k = 10

ViT-B/16

(a) ViT-B/16 Shuffle

ToMe

Dropping

ToMe

Dropping

ToMe

2 4 6 8 10

70 / 57 40 / 37 25 / 47 20 / 37 15 / 47

65 / 67 40 / 37 30 / 17 20 / 37 20 / 1

75 / 47 40 / 37 25 / 47 20 / 37 15 / 47

65 / 67 35 / 57 25 / 47 20 / 37 15 / 47

Dropping

(b) With defense.

k=2

Baseline

k = 10

k=6

ViT-B/16

shuffling), shuffling, and token reductions, across different split points 𝑘, without our defense. Dropping

k = 10

k=6

k=2

ToMe

A.2 ToMe

Dropping

k = 10

k=6

k=2

Shuffle

ToMe

Dropping

ToMe

2 4 6 8 10

5 / 187 10 / 157 15 / 107 10 / 117 5 / 147

25 / 147 20 / 117 15 / 107 15 / 77 10 / 97

20 / 157 10 / 157 10 / 137 10 / 117 10 / 97

20 / 157 15 / 137 15 / 107 10 / 117 10 / 97

(b) MAE-B/16

Figure 14: Reconstructed images with SARA for baseline (noshuffling), shuffling, and token reductions, across different split points 𝑘, with our defense.

Additional Illustrations.

In addition to the examples presented in the main body, we provide further reconstructions in Figures 14 and 13 obtained at the PURIoptimal values of 𝑟 for both the original and defended models across different split points. Importantly, these reconstructions correspond specifically to the PURI-optimal reduction amounts. Using a lower 𝑟 would allow SARA to recover higher-quality images, approaching the reconstruction quality observed under token shuffling. Conversely, a higher 𝑟 would make reconstruction more difficult but would also cause a greater degradation in task accuracy.

(a) ViT-B/16 Baseline

Dropping

Table 4 reports the optimal values of 𝑟 , within the considered range, for the two token-reduction techniques at different split points. We also report the corresponding number of transmitted tokens. For a fixed split point, increasing 𝑟 reduces the number of tokens transmitted from the edge to the cloud, thereby lowering communication and cloud-side computational costs, but potentially degrading task accuracy. As shown in Table 4, the PURI-optimal value of 𝑟 changes substantially after fine-tuning with the proposed defense. This does not imply that users should simply adopt a smaller 𝑟 to improve privacy. Rather, the defense shifts the privacy-utility trade-off, allowing lower reduction amounts, and thus better accuracy, while still achieving favorable privacy scores. This behavior is also evident from the comparison between the PURI curves obtained with the proposed defense and those reported in the original attack analysis.

Figure 13: Reconstructed images with SARA for baseline (no-

Shuffle

MAE-B/16

𝑘

A Appendix A.1 Optimal ratios with PURI

(b) MAE-B/16

Baseline

MAE-B/16

𝑘

Position Matters: Feature Inversion Attacks in ViT Split Inference with Token Reduction and Shuffling

Analyze & explore related research in Conceptio

Related documents