Privacy-Preserving Object Detection for Vision Transformer-Based Models

Privacy-Preserving Object Detection for Vision Transformer-Based Models 1st Homare Sueyoshi

2nd Kiyoshi Nishikawa

3rd Hitoshi Kiya

Tokyo Metropolitan University 6-6, Asahigaoka, Hino, Tokyo, Japan sueyoshi-homare@ed.tmu.ac.jp

Tokyo Metropolitan University 6-6, Asahigaoka, Hino, Tokyo, Japan kiyoshi@tmu.ac.jp

Tokyo Metropolitan University 6-6, Asahigaoka, Hino, Tokyo, Japan kiya@tmu.ac.jp

arXiv:2608.20712v1 [cs.CR] 21 Aug 2026

Abstract—We propose a novel object detection method that enables us to protect sensitive visual information of test images. Previous studies considering visual information protection focus on image classification tasks. This paper proposes an object detection method using perceptual encryption for the first time. The proposed method can achieve almost the same accuracy as that of models without any protection by utilizing the embedding structure of the Vision Transformer (ViT) and a domain adaptation technique with keys. In experiments, the effectiveness of the proposed method is verified in terms of accuracy and visual protection under the use of ViTdet, which is a ViT-based object detection model. Index Terms—Object Detection, Vision Transformer, Image Encryption, Privacy Preserving

I. I NTRODUCTION

The vision transformer (ViT) [1] trained with perceptual encryption has achieved great success in various applications such as privacy-preserving learning [2]–[4], access control, model protection [5], and adversarial defense [6], [7]. However, previous studies focus on image classification tasks [8]. Accordingly, in this paper, we propose a novel object detection method using perceptional encryption for the first time. In the proposed method, the ViTdet [9] model, which employs the ViT, is used as a base model for object detection. Perceptional encryption and a domain adaptation technique with secret keys studied for image classification are expanded for object detection. In experiments, the proposed method is demonstrated to be effective in object detection in terms of detection accuracy and visual protection. II. R ELATED WORK

A. ViTDeT Our proposed method is designed to be applicable to models based on the Vision Transformer (ViT) architecture. In this study, we utilize ViTDeT as a representative example of ViTbased Object Detection models. A architecture of the ViTDeT is shown in Figure 1. Unlike traditional object detectors that rely on hierarchical backbones and Feature Pyramid Networks (FPNs). ViTDet adopts a plain ViT backbone. As shown in the ”Input x” and ”Linear Projection” sections of Figure 1. Figure 1 shows the input image divided into patches x1p , . . . , x9p . These patches are then linearly projected into an embedding space using a patch embedding E, allowing the

Fig. 1: Architecture of ViTDeT

model to process image patches similarly to word tokens in natural language processing. To preserve spatial information, learnable position embeddings Epos are added to the patch embeddings. A learnable class token xclass is also prepended to the sequence. Consequently, the input z0 fed into the Transformer Encoder is formulated as follows. z0 = [xclass ; x1p E; . . . ; xN p E] + Epos

(1)

The backbone processes this sequence using Transformer blocks. While standard ViTs use global attention, ViTDet incorporates Window Attention (computing attention only within local windows) to reduce computational costs for highresolution inputs, supplemented by a few Global Attention blocks to enable information propagation across windows. Finally, a simple feature pyramid is constructed from the single-scale feature map output to perform object detection. In this paper, specifically within the structure described in Equation 1, we focus on the patch embedding matrix E. III. P ROPOSED M ETHOD A. Overview of propoesd method The framework of the proposed method is summarized as below (see Fig. 2). ⃝ 1 A model creator generates a random permutation matrix Eenc using a secret key k.

Fig. 3: Procedure of image encryption

Fig. 2: Framework of proposed method

⃝ 2 Encrypt patch embedding E to get an encrypted model by using Eenc in a model trained with plain images. ⃝ 3 Upload the encrypted model to a cloud server. ⃝ 4 A user receives Eenc from the model creator and encrypt a test image using Eenc . ⃝ 5 Upload the encrypted test image to the cloud server. ⃝ 6 The client receives an inference result from the cloud server. In the above framework, the cloud server does not have plain images and the secret key. B. Generation of Random Permutation Matrix The generation process of the random permutation matrix is described as follows: 1) A vector is given by: l = [l(1), l(2), . . . , l(i), . . . , l(L)],

(2)

where each element l(i) satisfies: l(i) ∈ {1, 2, . . . , L} i ̸= j ⇒ l(i) ̸= l(j),

∀i, j ∈ {1, 2, . . . , L}.

L = p2 c is the number of pixels in a block. p is defined as the patch size, and c is defined as the number of channels in the image. 2) A random permutation matrix Eenc ∈ RL×L is defined as:   k(1,1) k(1,2) · · · k(1,L)  k(2,1) k(2,2) · · · k(2,L)    (3) Eenc =  . .. ..  . . . .  . . . .  k(L,1) k(L,2) · · · k(L,L)

where the components k(i,j) are determined by: ( 0 if l(i) ̸= j k(i,j) = . 1 if l(i) = j Since Eenc is an orthogonal matrix, the following property holds: −1 ⊤ Eenc = Eenc .

(4)

As a concrete example, if L = 4 and the generated vector is l = [3, 4, 2, 1], the random permutation matrix Eenc is given by:   0 0 1 0 0 0 0 1  (5) Eenc =  0 1 0 0 . 1 0 0 0 C. Image Encryption The image encryption procedure is shown in the Figure 3. Encrypted images are used for model testing. In the proposed method, an encryption method is carried out in a similar manner as in conventional methods [10]. An input image x is divided into N non-overlapping blocks B = {B1 , B2 , . . . , BN } H of size p × p. N = W p × p is the number of blocks. We assume that width (W ) and height (H) are divisible by p. The block size p is set to be consistent with the patch size p of the ViT. Each block Bi is flattened into a vector bi . Using Eenc ∈ RL×L , each image block is encrypted as follows: ⊤ b̂i = bi Eenc .

(6)

The encrypted vectors b̂i are subsequently reshaped into encrypted blocks B̂i , which are then concatenated to construct the encrypted image x̂. Fig. 4 shows an example of encrypted images.

Consequently, the input sequence ẑ0 for the encrypted model is given by: c1 Ê; . . . ; xc N Ê] + E ẑ0 = [xclass ; x pos p p = [xclass ; x1p E; . . . ; xN p E] + Epos

(10)

= z0

Original image

This result demonstrates that ẑ0 = z0 , meaning that the model encryption effectively cancels out the influence of the image encryption, allowing the Transformer to process the data without any influence of encryption on object detection accuracy.

Zoomed-in view

(a) Non-encrypted images

IV. E XPERIMENTS A. Setup

Encrypted image

Zoomed-in view

(b) Encrypted images (block size: 16×16) Fig. 4: Example of encrypted images (image size: 1024×1024)

D. Model Encryption The trained model is encrypted by using the equation Ê = Eenc E.

(7)

where E ∈ RD×L and D denotes the embedding dimension of the Vision Transformer model (see Fig. 2). Accrordingly, the inference is carried out in the encrypted domain. E. Properties of proposed method We analyze the impact of encryption on object detection accuracy. Let E be the original patch embedding projection matrix and xip be the i-th patch of the input image. We introduce an encryption matrix Eenc , which satisfies the or−1 thogonality condition E⊤ enc = Eenc . The encrypted embedding matrix Ê and the encrypted i-th ci are defined as follows: patch x p Ê = Eenc E,

ci = xi E⊤ . x p p enc

(8)

By substituting these definitions, the projection of the encrypted patch can be derived as: ci Ê = (xi E⊤ )(Eenc E) = xi (E⊤ Eenc )E = xi E, x p p enc p enc p where we utilize the property that E⊤ enc Eenc = I.

(9)

In experiments, an object detection task was performed by using the COCO dataset [11] as test images, which consists of 80 object categories. The performance of trained models was also evaluated with the “val2017” subset as validation images from the COCO dataset. Furthermore, we extended our evaluation to the LVIS dataset [12]. While LVIS shares the exact same images as those of the COCO dataset, it is characterized by different data splits and significantly richer annotations. Unlike the 80 categories in COCO, LVIS features a massive vocabulary of 1,203 categories. To comprehensively assess detection performance, we adopted ViTdet as a vision transformer-based model with the Mask R-CNN framework. For evaluation metrics, we employed the standard Average Precision (AP) scores, which range from 0 to 100, where a higher value indicates superior performance. To evaluate performance across different object scales, we utilized APs , APm , and APl for small (area < 32 × 32), medium (32 × 32 ≤ area ≤ 96 × 96), and large (area > 96 × 96) objects. B. Experimental Results Table I presents the evaluation results of object detection accuracy (AP) under various conditions. In this evaluation, ViTdet models with three different backbones (ViT-B, ViT-L, and ViT-H) were utilized. These models correspond to the three models called “model final 61ccd1.pkl”, “model final 6146ed.pkl”, and “model final 7224f1.pkl” published on the official ViTdet GitHub repository. “Baseline” represents the results obtained using both non-encrypted models and images. “Without model encryption” shows results when only the images were encrypted and the models were not encrypted. In the table, “Proposed” denotes the proposed method utilizing both encrypted model and images. As shown in Table I, it was confirmed that the proposed method achieved almost the same as that of the non-encrypted models. Theoretically, the proposed block-wise encryption can cancel out the influence of the image encryption at the embedding layer, but the accuracy was slightly different from that of “Baseline” due to the influence of JPEG compression used in the COCO dataset.

TABLE I: Comparison of object detection accuracy (AP) using Mask R-CNN on COCO dataset.

TABLE III: Comparison of object detection accuracy (AP) using Cascade Mask R-CNN on COCO dataset.

Backbone

Method

AP

APs

APm

APl

Backbone

Method

AP

APs

APm

APl

ViT-B

Baseline Without model encryption Proposed

51.412 0.035 50.118

35.645 0.018 32.882

55.031 0.055 53.875

66.273 0.156 64.717

ViT-B

Baseline Without model encryption Proposed

53.781 0.085 52.520

36.146 0.013 34.307

57.487 0.116 56.028

68.660 0.335 67.181

ViT-L

Baseline Without model encryption Proposed

55.450 0.804 54.534

39.342 0.129 38.762

59.699 0.648 58.634

69.569 1.776 68.337

ViT-L

Baseline Without model encryption Proposed

57.470 0.739 56.671

40.699 0.047 40.326

61.592 0.525 60.651

72.414 1.704 71.172

ViT-H

Baseline Without model encryption Proposed

56.594 0.406 55.843

40.456 0.066 40.895

60.783 0.733 59.932

71.301 0.725 69.997

ViT-H

Baseline Without model encryption Proposed

58.650 0.399 57.586

41.985 0.071 41.347

62.684 0.484 61.868

73.835 0.603 72.262

TABLE II: Comparison of object detection accuracy (AP) using Mask R-CNN on LVIS dataset.

TABLE IV: Comparison of object detection accuracy (AP) using Cascade Mask R-CNN on LVIS dataset.

Backbone

Method

AP

APs

APm

APl

Backbone

Method

AP

APs

APm

APl

ViT-B

Baseline Without model encryption Proposed

39.883 0.030 38.585

30.731 0.009 29.339

48.922 0.026 47.601

56.971 0.080 55.375

ViT-B

Baseline Without model encryption Proposed

42.770 0.074 41.570

32.840 0.005 31.500

52.097 0.027 51.168

61.230 0.260 59.783

ViT-L

Baseline Without model encryption Proposed

45.924 0.286 44.546

36.345 0.048 35.738

55.264 0.284 53.989

63.392 1.002 61.469

ViT-L

Baseline Without model encryption Proposed

49.017 0.214 47.900

38.651 0.024 37.950

59.417 0.133 57.720

67.392 0.732 65.849

ViT-H

Baseline Without model encryption Proposed

49.123 0.227 47.811

39.369 0.070 38.917

59.184 0.230 57.320

65.992 0.611 64.993

ViT-H

Baseline Without model encryption Proposed

51.175 0.131 50.583

41.012 0.027 40.755

61.324 0.159 60.442

68.627 0.383 67.860

V. C ONCLUSION We proposed a privacy-preserving method for object detection that utilizes the embedding structure of Vision Transformers (ViTs). Using a ViTdet as a representative example, we demonstrated that applying block-wise encryption effectively protects visual information while maintaining high detection accuracy equivalent to that of non-encrypted models. Furthermore, the significant decrease in accuracy observed when only the images were encrypted demonstrates the necessity of encrypting the model’s embedding layer. ACKNOWLEDGEMENTS This work was supported in part by JSPS KAKENHI (Grant Number 25K07750). R EFERENCES [1] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” in Proceedings of the International Conference on Learning Representations (ICLR), 2021. [2] H. Kiya, A. MaungMaung, Y. Kinoshita, S. Imaizumi, and S. Shiota, “An overview of compressible and learnable image transformation with secret key and its applications,” arXiv preprint arXiv:2201.11006, 2022. [3] M. Hirose, S. Imaizumi, and H. Kiya, “Learnable image encryption without key management for privacy-preserving vision transformer,” IEEE Access, vol. 13, pp. 201351–201362, 2025. [4] W. Sirichotedumrong, T. Chuman, S. Imaizumi, and H. Kiya, “Grayscale-based block scrambling image encryption for social networking services,” 2018 IEEE international conference on multimedia and expo (ICME), pp. 1–6, 2018. [5] A. MaungMaung and H. Kiya, “A protection method of trained CNN model with a secret key from unauthorized access,” APSIPA Transactions on Signal and Information Processing, vol. 10, pp. e10, 2021.

[6] M. AprilPyone and H. Kiya, “Block-wise image transformation with secret key for adversarially robust defense,” IEEE Transactions on Information Forensics and Security, vol. 16, pp. 2709–2723, 2021. [7] R. Iijima, S. Shiota, and H. Kiya, “A random ensemble of encrypted vision transformers for adversarially robust defense,” IEEE Access, vol. 12, pp. 69206–69216, 2024. [8] M. AprilPyone and H. Kiya, “Privacy-Preserving Image Classification Using an Isotropic Network,” IEEE MultiMedia, vol. 29, no. 2, pp. 23– 33, Apr.–Jun. 2022. [9] Y. Li, H. Mao, R. Girshick, and K. He, “Exploring Plain Vision Transformer Backbones for Object Detection,” arXiv preprint arXiv:2203.16527, 2022. [10] H. Kiya, T. Nagamori, S. Imaizumi, and S. Shiota, “Privacy-Preserving Semantic Segmentation Using Vision Transformer,” Journal of Imaging, vol. 8, no. 9, article 233, 2022. [11] T.-Y. Lin, M. Maire, S. Belongie, L. Bourdev, R. Girshick, J. Hays, P. Perona, D. Ramanan, C.L. Zitnick, and P. Dollár, “Microsoft COCO: Common Objects in Context,” arXiv preprint arXiv:1405.0312, 2015. [12] A. Gupta, P. Dollár, and R. Girshick, “LVIS: A Dataset for Large Vocabulary Instance Segmentation,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.

Related documents

• Privacy-Preserving Object Detection for Vision Transformer-Based Models#659269
• Privacy-Preserving Object Detection for Vision Transformer-Based Models#657133
• On the Robustness of Vision-Language Models in Zero-shot Privacy Classification#635659
• On the Robustness of Vision-Language Models in Zero-shot Privacy Classification#633310
• PROTECTING PRIVACY#2109