LORENC: LOW-RANK ENCRYPTION FOR SECURING FOUNDATION MODELS AND LORA ADAPTERS Beomjin Ahn1 ∗ , Jungmin Kwon3 † , Chanyong Jung4 § , Jaewook Chung2
arXiv:2605.13163v1 [cs.CR] 13 May 2026
1
Samsung Research, 2 Samsung Electronics, 3 Amazon Web Services, 4 University of Michigan ABSTRACT
1. INTRODUCTION
Foundation models and low-rank adapters enable efficient ondevice generative AI but raise risks such as intellectual property leakage and model recovery attacks. Existing defenses are often impractical because they require retraining or access to the original dataset. We propose LoREnc, a training-free framework that secures both FMs and adapters via spectral truncation and compensation. LoREnc suppresses dominant low-rank components of FM weights, compensates for the missing information in authorized adapters, and further applies orthogonal reparameterization to obscure structural fingerprints of the protected adapter. Unauthorized users produce structurally collapsed outputs, while authorized users recover exact performance. Experiments demonstrate that LoREnc provides strong protection against model recovery with under 1% computational overhead.
Foundation models (FMs) can be adapted to many downstream tasks, improving the practical usability of large-scale models. Parameter-Efficient Fine-Tuning (PEFT) methods are widely adopted for this purpose [1], and LoRA [2] is a de facto standard due to its simplicity and broad tooling support. However, releasing FMs also introduces risks: weights can enable unauthorized inference or partial recovery of proprietary models, making exposure especially harmful. Existing protection mechanisms offer limited practical guarantees in this setting. Passive approaches focus on ownership verification rather than preventing unauthorized use. More recent methods attempt to prevent extraction or misuse by modifying or hiding deployed weights, but typically require expensive retraining or still assume reversible parameters are deployed to edge devices. Full-model encryption is also impractical in this setting: runtime decryption of an entire FM requires loading the plaintext model into device memory at inference time, negating the efficiency constraints that define edge deployment.
Index Terms— Generative AI, Foundation Models, LoRA, Parameter-Efficient Fine-Tuning * Corresponding author. E-mail: [email protected] † This work was completed before the author joined Amazon. §Work done while at Samsung Research.
Offline Protection Phase FM
TSVD
Edge Device Inference Phase Truncated FM
－
Low-rank Component
To address these limitations, we propose LoREnc (LowRank Encryption), a training-free framework that jointly protects FMs and their LoRA adapters (Figure 1). Unlike conven-
Encrypted LoRA
LoRA Reparam.
FM Inference
Downstream Task (Unauth. User)
SVD
LoRA
＋
Compensated LoRA
Restoration Key
Downstream Task (Auth. User)
Fig. 1: Overview of the LoREnc pipeline. The framework protects FMs by relocating dominant spectral components to LoRA adapters, preventing unauthorized use (visualized as structural collapse) while enabling numerically exact recovery.
tional cryptographic methods that secure data confidentiality at the bit level, LoREnc can be interpreted as operating in the spirit of perceptual encryption [3], where unauthorized access leads to severe semantic degradation of model outputs, and the protection is realized directly in the model’s weight space. Inspired by the Eckart–Young theorem [4], LoREnc mathematically suppresses the dominant low-rank components of FM weights to structurally degrade unauthorized inference outputs. Conversely, it compensates for these components in authorized adapters to enable theoretically exact recovery of original performance. Unlike prior approaches, LoREnc operates purely on post-training weights without accessing the original dataset, thereby ensuring data-independence suitable for privacy-sensitive on-device deployment. Specifically, we propose a training-free spectral truncation and compensation mechanism that preserves authorized performance while inducing structural collapse for unauthorized users. We further introduce a secure adapter encoding scheme robust against reuse and recovery attacks. Extensive experiments, including on-device benchmarks, confirm that LoREnc achieves strong protection with under 1% overhead.
2. RELATED WORK
3. PROBLEM DEFINITION AND THREAT MODEL Our objective is to protect the deployed FM weights against unauthorized reuse while preserving the functionality of authorized downstream tasks using LoRA adapters. To this end, we consider a training-free protection setting in which subsets of model parameters are secured and distributed with LoRA adapters, thereby allowing only authorized users to recover the intended behavior. 3.1. Threat Model and Assumptions Unlike server-side deployments, on-device models reside in user-controlled environments where physical memory inspection and static weight analysis are readily available. We assume restoration keys are protected in a hardware-backed environment such as a Trusted Execution Environment (TEE), while the deployed artifacts (encrypted FM weights and encrypted adapters) are accessible to an unauthorized party. The adversary then attempts restoration via ML-level weight-extraction methods such as Spectral DeTuning (SDT) [10] or limited finetuning. LoREnc targets practical empirical resistance against such ML-level extraction, rather than formal cryptographic unrecoverability; physical side-channel attacks and direct key leakage are outside the scope of this work.
2.1. Vulnerabilities in Edge Deployment 3.2. Design Requirements Deploying deep learning models on edge devices exposes model weights to adversaries with physical or software-level access, making unauthorized reuse, extraction, and model stealing practical at scale [5, 6, 7, 8, 9]. Moreover, PEFT and lightweight adapters such as LoRA [2] simplify edge deployment, but can also facilitate attacks by providing structured update signals. For example, Spectral DeTuning [10] shows that collecting merged FM and adapter weights can recover pre-trained parameters via iterative low-rank factorization.
We define six design requirements for practical FM protection, summarized in Table 1, which serve as evaluation criteria throughout this paper. The first five requirements are adopted from prior work [15], and we introduce data-independence to reflect a realistic situation in which collecting training data and retraining models become impractical. 4. LORENC: LOW-RANK ENCRYPTION 4.1. Spectral Truncation
2.2. Model Protection and Encryption Model protection approaches can be broadly categorized into passive and active methods. Unlike passive techniques such as watermarking and fingerprinting [11, 12], active methods restrict the model’s functionality. Representative active methods hide important layers in secure storage (e.g., SOTER [13], ShadowNet [14]), obfuscate weights (e.g., NNSplitter [15], GroupCover [16]), or decompose parameters (e.g., SLIP [17]) to prevent unauthorized inference or weight extraction. While these provide stronger protection by modifying deployed parameters, they typically rely on retraining or iterative optimization (e.g., NNSplitter), or expose transformed weights via interactive secure-resource protocols at inference time (e.g., SLIP). In contrast, LoREnc is fully on-device, training-free, and data-independent.
Let W ∈ Rm×n denote the weight matrix of an FM layer. Our objective is to construct a truncated weight W̃ that conceals the principal knowledge of W while enabling theoretically exact downstream recovery. We decompose the weight as W = W̃ + L, where L is the low-rank component (serving as the spectral key) extracted via truncated SVD. Low-rank Component Extraction To maximally suppress the semantic information of W , we utilize the Eckart–Young theorem [4], which states that the leading singular components capture the dominant energy of a matrix. Consequently, removing L effectively eliminates the model’s ability to form coherent structures, leaving only high-frequency residuals that lack semantic meaning. In the supplementary material, we further prove that this truncation maximizes the Frobeniusnorm distance between the original and truncated weights. We compute the low-rank component via TSVD as:
Table 1: Summary of design requirements for practical foundation-model (FM) protection. Requirement Effectiveness Integrity Stealthiness Efficiency Resilience Data-independence
Description Unauthorized foundation inference should yield semantically meaningless outputs. Authorized downstream inference should exactly match baseline performance. Protected weights should not appear structurally distinct from ordinary adapters. Authorized inference should incur minimal computational and memory overhead. The FM should remain unrecoverable under model separation and restoration attacks. No training data should be required for encryption or downstream authorization.
Table 2: Visualization of text-to-image results with SD 1.5. The first row shows the baseline results; the remaining rows depict outputs from LoREnc. (Prompt: “A trio of dogs sitting in their owner’s lap in a red convertible.”)
Model Authorization
Foundation
Downstream Task 1
Task 2
Task 3
Task 4
Task 5
Baseline
✗ LoREnc
✓
L = UF M ΣF M VFTM = TSVD∆r (W ),
(1)
where TSVD∆r (·) denotes the rank-∆r truncated SVD operator. Here, UF M ∈ Rm×∆r , ΣF M ∈ R∆r×∆r , and VF M ∈ Rn×∆r . The hyperparameter ∆r specifies the number of truncated singular components and thus controls the strength of the perceptual encryption. Increasing ∆r generally improves security, but comes with a trade-off of higher overhead. Since L is never deployed to the edge device, reconstruction of W from W̃ alone is infeasible. Spectral Compensation via LoRA To preserve downstream functionality, we require the compensated adapters to satisfy W̃ +B̃k Ãk = W +Bk Ak , which yields the condition B̃k Ãk = L + Bk Ak . To guarantee exact compensation of L, we employ a temporary rank expansion via concatenation: 1/2
B̃k = [ Bk , UF M ΣF M ],
1/2
Ãk = [ Ak , ΣF M VFTM ],
(2)
where B̃k ∈ Rm×(r+∆r) and Ãk ∈ R(r+∆r)×n . This construction ensures exact downstream recovery while effectively fusing the low-rank component into the LoRA adapters, satisfying the integrity requirement.
4.2. LoRA Adapter Encryption Since the compensated adapters (B̃k , Ãk ) explicitly contain the spectral key L, unauthorized access could compromise both the adapter and the foundation model. We therefore introduce an explicit LoRA adapter encryption stage to protect LoRA modules against unauthorized access. LoRA Restoration Keys We apply SVD to the adapter T weights B̃k Ãk = ULo ΣLo VLo and split it as 1/2
[KB̃k , B̃k∗ ] = ULo ΣLo ,
1/2
T . [KÃk , Ã∗k ] = ΣLo VLo
(3)
Here, B̃k∗ ∈ Rm×r and Ã∗k ∈ Rr×n denote the encrypted LoRA adapter weights, while KB̃k and KÃk form the LoRA restoration keys. Beyond encrypting the adapter contents, this step also reduces the LoRA rank from r + ∆r back to r, which helps conceal whether LoREnc has been applied. Orthogonal LoRA Reparameterization Finally, we apply a reparameterization B̃k′ = B̃k∗ Mk , Ã′k = MkT Ã∗k , with a random orthogonal matrix Mk ∈ Rr×r . This induces an isometric rotation in the parameter space, creating infinite equivalent factorizations for the same product. Without this repa-
Table 3: Performance test of LoREnc on SD 1.5 with COCO Captions [18]. LPIPS is computed between images generated by the baseline model and the LoREnc-protected model, and ∆CLIP is computed as CLIPLoREnc − CLIPBaseline for each task. Lower ∆CLIP values indicate more severe performance degradation. For readability, 0.000 values are visually emphasized. Authorization ∆CLIP score
✗ ✓
LPIPS
−0.148
Task 1 −0.155
Task 2 −0.148
Downstream Task 3 −0.143
Task 4 −0.144
Task 5 −0.149 (±0.048)
Foundation (±0.046)
(±0.052)
(±0.048)
(±0.044)
(±0.049)
−0.148
0.000
0.000
0.000
0.000
0.000
(±0.046)
(±0.000)
(±0.000)
(±0.000)
(±0.001)
(±0.000)
✗
0.827
0.870
0.857
0.869
0.860
0.844
(±0.080)
(±0.074)
(±0.073)
(±0.084)
(±0.084)
(±0.085)
✓
0.827
0.000
0.000
0.000
0.000
0.000
(±0.080)
(±0.000)
(±0.000)
(±0.000)
(±0.001)
(±0.000)
Table 4: Efficacy on autoregressive FMs evaluated on WikiText-2 [19] under authorized access. (Left) Increase in perplexity (∆PPL) after applying LoREnc. (Right) Example outputs from the protected FMs (input: “Kirby ’s Block Ball is”).
∆PPL
Model GPT-2 Llama 3
Foundation 120.0 8793
Downstream 0.000 0.000
Example output of the autoregressive FM (GPT-2) Baseline Kirby ’s Block Ball is a special item that can be used ... LoREnc Kirby ’s Block Ball is a ” a ” a ” a ” a ” a ” a ” a ” a ” ...
rameterization, the encrypted adapters would retain the strict orthogonality inherent to SVD, making them distinguishable from standard Gaussian-initialized weights. This structural fingerprint would allow adversaries to easily detect the presence of the protection, thereby compromising the stealthiness requirement against simple structural inspection. We note that adaptive detectors specifically designed for protected adapters may still distinguish them, which is outside the scope of this work.
our experiments address: efficacy of authorized recovery vs. unauthorized degradation (Q1), resilience to fine-tuning attacks (Q2), robustness to Spectral DeTuning [10] (Q3), and edge-device efficiency (Q4). Unless otherwise specified, we set ∆r = 4, as it offers a practical trade-off between effectiveness and computational overhead. Additional details are provided in the supplementary material.
4.3. Authorized Downstream Inference
We compare three cases: (i) the original model without LoREnc, (ii) LoREnc-applied model under unauthorized access, and (iii) LoREnc-applied model with valid keys. Table 3 reports CLIP [21] and LPIPS [22] scores on SD 1.5 [20]. With LoREnc, foundation-only inference is severely degraded, demonstrating strong effectiveness against unauthorized access. Conversely, authorized users recover baseline outputs up to negligible floating-point errors, confirming the integrity of the downstream tasks. Table 2 shows structurally collapsed unauthorized outputs and indistinguishable authorized outputs. Similar trends hold for autoregressive models (Table 4). These results suggest that this spectral degradation is modalityagnostic. LoREnc successfully induces high perplexity on these models (GPT-2 [23], Llama 3 [24]), confirming that our method is applicable beyond computer vision.
An authorized user retrieves W̃ , B̃k′ , and Ã′k from storage and obtains the restoration keys KB̃k and KÃk from a secure environment. The decrypted weight is obtained as W̃ + B̃k′ Ã′k + KB̃k KÃk = W + Bk Ak .
(4)
This reconstruction occurs on-the-fly during the forward pass, requiring no additional memory storage for the restored FM weights. Notably, even authorized users cannot directly access the original FM weight W , as the low-rank component L is never deployed to the device. 5. EXPERIMENTS We evaluate LoREnc across diverse generative architectures. To ensure a direct comparison with the state-of-the-art weightrecovery method, Spectral DeTuning [10], we primarily utilize Stable Diffusion v1.5 (SD 1.5) [20] as our main testbed. Additionally, we demonstrate the architecture-agnostic nature of LoREnc by providing results on recent DiT-based models (e.g., Sana) in the supplementary material. Specifically,
5.1. Efficacy of Applying LoREnc (Q1)
5.2. Fine-Tuning Attack (Q2) We evaluate resilience against adaptive recovery by fine-tuning the truncated SD 1.5 model with 0.1k–100k samples. Table 5 shows that LoREnc consistently prevents meaningful FM recovery, as reflected by low CLIP scores, and outperforms
Table 5: Fine-tuning attack resilience on SD 1.5. CLIP scores are measured after one epoch of fine-tuning with varying data sizes. Baseline CLIP score is 0.267.
CLIP score (Foundation)
Method NNSplitter LoREnc
Training-free ✗ ✓
Protected 0.187 (±0.048) 0.118 (±0.034)
W-Error
−4 −6 −8 −10 −12 −14 −16 −18 −20 2 3 4 5 6 7 8 9 10 11 12 13 14 15
# of LoRA weights
w/o LoRA Encryption (Leaked key) w/ LoRA Encryption (Random key) w/ LoRA Encryption (Key from TSVD(BkAk)) w/ LoRA Encryption (Key from TSVD(BkAk + L )) Fig. 2: W-Error under Spectral DeTuning with varying numbers of downstream tasks. A higher W-Error indicates better protection. NNSplitter [15] across all data regimes. Even with 100k samples, performance remains far below the original. Comprehensive ablations on ∆r ∈ {1, 4, 16, 64} covering computational overhead and defense resilience under both fine-tuning and SDT attacks are consolidated in the supplementary material (Tables 1 and 3). 5.3. Spectral DeTuning Attack (Q3) We further evaluate robustness to Spectral DeTuning (SDT) [10], an advanced attack that attempts to restore FM weights via iterative low-rank factorization. Following the original paper, we measure security using W-Error, the average log-scale mean squared error between estimated and original weights (details in supplementary). Higher W-Error indicates greater deviation from the original weights, implying better protection. We compare LoREnc against two baseline strategies: (1) random key (blue curve), where the key is generated independently of the weight distribution, and (2) self-derived key (orange curve), where the key is extracted from the adapter itself (Bk Ak ) without FM context. As shown in Figure 2, the random key and the self-derived key fail because they do not capture the structural dependencies and critical spectral information (L) of the foundation model. In contrast, LoREnc (red curve) maintains consistently high
0.1k Data 1k Data 10k Data 100k Data 0.240 (±0.039) 0.267 (±0.029) 0.267 (±0.031) 0.251 (±0.032) 0.137 (±0.035) 0.159 (±0.032) 0.211 (±0.040) 0.231 (±0.034) Table 6: Measuring the overhead of LoREnc with UNet of SD 1.5. Inference time is measured by running only the mid-block of the UNet due to real-device memory limitations. Baseline
LoREnc
Overhead
# of Params (M)
873.09
874.78
+0.19%
GFLOPs
700.18
702.99
+0.40%
Inference time (sec.)
0.462
0.463
+0.22%
W-Error, remaining robust under large-scale adapter collection, which highlights the resilience of LoREnc against advanced model recovery attempts. 5.4. Efficiency Analysis (Q4) We analyze the computational overhead of LoREnc from both theoretical and empirical perspectives. Theoretically, the computationally expensive operations, such as SVD, are performed entirely before the deployment. During inference, the only additional cost comes from the increased rank of LoRA adapters (r → r + ∆r). Since ∆r is typically small (e.g., 4) compared to the hidden dimensions of FMs, this results in a minimal increase in FLOPs and parameter count. Empirically, as shown in Table 6, LoREnc incurs negligible overheads (< 1%) in terms of parameters, GFLOPs, and inference latency measured on a commercial smartphone. Notably, the latency was evaluated under real-world memory constraints (running the UNet mid-block), confirming that LoREnc is highly suitable for resource-constrained edge environments without requiring specialized hardware accelerators. 6. CONCLUSION We presented LoREnc, a training-free framework employing spectral truncation and compensation to secure on-device FMs. It mathematically guarantees structural collapse for unauthorized inference while preserving integrity for authorized users. In summary, LoREnc satisfies all six design requirements—Effectiveness, Integrity, and Resilience—verified through extensive experiments, while maintaining Stealthiness, Data-independence, and Efficiency essential for practical edge deployment.
7. REFERENCES [1] Z. Han, C. Gao, J. Liu, J. Zhang, and S. Q. Zhang, “Parameter-efficient fine-tuning for large models: A comprehensive survey,” Trans. Mach. Learn. Res., vol. 2024, 2024. [2] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” CoRR, vol. abs/2106.09685, 2021. [3] S. Li, G. Chen, A. Cheung, B. K. Bhargava, and K. Lo, “On the design of perceptual mpeg-video encryption algorithms,” IEEE Trans. Circuits Syst. Video Technol., vol. 17, no. 2, pp. 214–223, 2007.
“SOTER: guarding black-box inference for general neural networks at the edge,” in USENIX, J. Schindler and N. Zilberman, Eds. 2022, pp. 723–738, USENIX Association. [14] Z. Sun, R. Sun, C. Liu, A. R. Chowdhury, L. Lu, and S. Jha, “Shadownet: A secure and efficient on-device model inference system for convolutional neural networks,” in Symposium on Security and Privacy. 2023, pp. 1596–1612, IEEE. [15] T. Zhou, Y. Luo, S. Ren, and X. Xu, “NNSplitter: An active defense solution for DNN model via automated weight obfuscation,” in ICML, 2023, pp. 42614–42624.
[4] C. Eckart and G. Young, “The approximation of one matrix by another of lower rank,” Psychometrika, vol. 1, no. 3, pp. 211–218, 1936.
[16] Z. Zhang, N. Wang, Z. Zhang, Y. Zhang, T. Zhang, J. Liu, and Y. Wu, “Groupcover: A secure, efficient and scalable inference framework for on-device model protection based on tees,” in ICML. 2024, OpenReview.net.
[5] Z. Sun, R. Sun, L. Lu, and A. Mislove, “Mind your weight(s): A large-scale study on insufficient machine learning model protection in mobile apps,” in USENIX, 2021, pp. 1955–1972.
[17] Y. Refael, A. Hakim, L. Greenberg, T. Aviv, S. Lokam, B. Fishman, and S. Seidman, “SLIP: securing llms IP using weights decomposition,” CoRR, vol. abs/2407.10886, 2024.
[6] M. Xu, J. Liu, Y. Liu, F. X. Lin, Y. Liu, and X. Liu, “A first look at deep learning apps on smartphones,” in WWW, 2019, pp. 2125–2136.
[18] X. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollar, and C. L. Zitnick, “Microsoft COCO Captions: Data collection and evaluation server,” 2015.
[7] P. Ren, C. Zuo, X. Liu, W. Diao, Q. Zhao, and S. Guo, “DEMISTIFY: identifying on-device machine learning models stealing and reuse vulnerabilities in mobile apps,” in ICSE, 2024, pp. 41:1–41:13.
[19] S. Merity, C. Xiong, J. Bradbury, and R. Socher, “Pointer sentinel mixture models,” in ICLR, 2017.
[8] A. S. Rakin, M. H. I. Chowdhuryy, F. Yao, and D. Fan, “DeepSteal: Advanced model extractions leveraging efficient weight stealing in memories,” in Symposium on Security and Privacy. 2022, pp. 1157–1174, IEEE. [9] Y. Huang and C. Chen, “Smart app attack: Hacking deep learning models in android apps,” IEEE Trans. Inf. Forensics Secur., vol. 17, pp. 1827–1840, 2022.
[20] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in CVPR, 2022, pp. 10674–10685. [21] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” in ICML, 2021, pp. 8748–8763.
[10] E. Horwitz, J. Kahana, and Y. Hoshen, “Recovering the pre-fine-tuning weights of generative models,” in ICML, 2024.
[22] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in CVPR, 2018, pp. 586– 595.
[11] J. Zhang, Z. Gu, J. Jang, H. Wu, M. P. Stoecklin, H. Huang, and I. M. Molloy, “Protecting intellectual property of deep neural networks with watermarking,” in AsiaCCS, 2018, pp. 159–172.
[23] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al., “Language models are unsupervised multitask learners,” OpenAI blog, vol. 1, no. 8, pp. 9, 2019.
[12] P. Yang, Y. Lao, and P. Li, “Robust watermarking for deep neural networks via bi-level optimization,” in ICCV, 2021, pp. 14821–14830.
[24] A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, et al., “The llama 3 herd of models,” CoRR, vol. abs/2407.21783, 2024.
[13] T. Shen, J. Qi, J. Jiang, X. Wang, S. Wen, X. Chen, S. Zhao, S. Wang, L. Chen, X. Luo, F. Zhang, and H. Cui,
LORENC: LOW-RANK ENCRYPTION FOR SECURING FOUNDATION MODELS AND LORA ADAPTERS (SUPPLEMENTAL MATERIAL)
1. JUSTIFICATION OF TSVD-BASED TRUNCATION In the main paper, we claimed that truncating the top-∆r singular components maximizes the deviation between the original weights and their truncated counterparts, thereby strengthening our perceptual encryption. We support this claim by deriving the Frobenius norm between the weights. Let X ∈ Rm×n be a real rectangular matrix with m ≥ n, and let its singular values satisfy σ1 ≥ · · · ≥ σn ≥ 0. For an index set S ⊆ I := {1, . . . , n} with |S| = ∆r (where 1 ≤ ∆r ≤ n), write the SVD expansion as X=
n X
σi ui vi⊤ .
(1)
i=1
We define the rank-∆r partial sum X XS = σi ui vi⊤ .
(2)
i∈S
In particular, define T := {1, . . . , ∆r}, i.e., the indices of the ∆r largest singular values, and denote XT =
X
σi ui vi⊤ =
∆r X
σi ui vi⊤ .
(3)
i=1
i∈T
The following lemma formalizes why truncating the most significant singular components maximizes the deviation between the original and truncated weights. Lemma 1. Among all subsets S ⊆ I with |S| = ∆r, the set T = {1, . . . , ∆r} maximizes ∥XS ∥F (equivalently, ∥X − (X − XS )∥F ). That is, T ∈ arg max ∥XS ∥F . S⊆I |S|=∆r
Proof. From Equation 2 and the orthonormality of the singular vectors, X 2 ∥XS ∥F = σi2 . (4) i∈S
Let S = {i1 < · · · < i∆r } be any subset of size ∆r. Since ij ≥ j and the sequence (σi ) is non-increasing, we have
σij ≤ σj for all j = 1, . . . , ∆r. Therefore, X i∈S
σi2 =
∆r X j=1
σi2j ≤
∆r X
σj2 =
j=1
X
2
σi2 = ∥XT ∥F .
i∈T
Taking square roots yields ∥XS ∥F ≤ ∥XT ∥F for all such S, so T is a maximizer. 2. EXPERIMENT DETAILS Experiments were conducted using an NVIDIA H100 GPU (80GB HBM3), with FP32 precision (w/o NVIDIA TF32). 2.1. Efficacy of Applying LoREnc (Q1) We obtained Stable Diffusion 1.5 [1], GPT-2 [2], and Llama 3 [3] from Hugging Face (stable-diffusion-v1-5/stablediffusion-v1-5, openai-community/gpt2, meta-llama/MetaLlama-3-8B). For Stable Diffusion, we used five downstream LoRA adapters (ral-bastet-sd15, ral-chrcrts-sd15, ral-cigarette-sd15, ral-cofzee-sd15, ral-crystals-sd15) from the LoWRA Bench dataset [4]. We used the first 100 captions from the COCO Captions validation set [5] for text-to-image evaluation (https: //github.com/tylin/coco-caption/blob/master/ annotations/captions_val2014.json). CLIP scores
were computed using the ViT-L/14 CLIP model [6], and LPIPS [7] was computed using VGG networks. For generations, we fixed the random seed to 0 and used the default inference settings. Table 2 shows the raw results of CLIP scores. ∆CLIP score in the main paper is computed by subtracting the baseline scores (first row) from scores of the LoREnc-applied model (second and third rows). For GPT-2 and Llama 3, we evaluated on the WikiText-2 test split [8] (wikitext-2-raw-v1; https://huggingface. co/datasets/wikitext) and used publicly available LoRA adapters from Hugging Face (varun-v-rao/gpt2-large-lora2.95M-squad-model1, hallisky/lora-formality-formal-llama3-8b). 2.2. Fine-Tuning Attack (Q2) Because NNSplitter requires reinforcement-learning-based retraining, we followed the authors’ released implementation and
hyperparameters. We trained Stable Diffusion 1.5 [1] on the LAION dataset [9] following the original procedure. For the RNN controller, we used epoch=20, batch size=5, and lr=0.01. For Stable Diffusion updates within each RL batch, we used batch size=8, lr=0.005, min w=-0.14, max w=0.24, and eps=8e-5 for 12,800 steps. We applied early stopping when the RL reward or the generated images’ CLIP score did not improve over several runs. This procedure produced an encrypted model with 2295 weight secrets. For the fine-tuning attack experiment, we fine-tuned using AdamW with lr=1e-4 and batch size=32. 2.3. Spectral DeTuning Attack (Q3) We used the same dataset and experimental protocol as the original Spectral DeTuning paper [4] (n iters=300, sched start rank=1, sched end rank=32). Following the paper, we used W-Error defined as N
W-Error =
1 X log10 N l=1
1 ∥Ŵ (l) − W (l) ∥2F |W (l) |
(5)
where N is the total number of layers, |W (l) | is the number of parameters in the l-th layer, and Ŵ (l) denotes the recovered weight. 2.4. Efficiency Analysis (Q4) We used the calflops library (https://github.com/ MrYxJ/calculate-flops.pytorch) to compute GFLOPs for the overhead analysis. For edge-device measurements, we converted the model to LiteRT (formerly TFLite) and tested the official LiteRT Android benchmark application on a Galaxy Fold 4 smartphone (10 warm-up runs; average over 50 measured runs). As described in the main paper, we measured inference time by running only the UNet mid-block due to memory constraints on the device. Detailed parameter and computational overhead results are reported in Table 1. Interestingly, while GFLOPs increase linearly with ∆r, the on-device inference latency shows a non-linear jump at ∆r = 16. This behavior is typical in on-device environments, attributed to memory access overheads exceeding cache thresholds or suboptimal kernel tiling for specific matrix dimensions. This observation reinforces our choice of ∆r = 4 as the optimal trade-off point for SD 1.5, offering strong security with negligible latency. 3. ADDITIONAL QUALITATIVE RESULTS ON DIT ARCHITECTURES While our main experiments focus on SD 1.5 for fair comparison with prior baselines, LoREnc is fundamentally a matrixlevel operation applicable to any architecture. To verify its generalizability, we evaluate LoREnc on Sana-0.6B [10], a
(a) Original
(b) ∆r = 4
(c) ∆r = 16
Fig. 1. Effect of the truncation rank (∆r) on a highly compact DiT model (Sana-0.6B). Prompts: “A trio of dogs sitting in their owner’s lap in a red convertible.”, “An airplane flying high in the blue sky.”, “A woman stands in front of the mirror to take a picture.”
state-of-the-art Diffusion Transformer (DiT). We specifically selected the model because its extremely efficient footprint makes it a realistic and practical candidate for our targeted ondevice edge deployment scenarios. Following standard PEFT practices, we selectively target the attention projection layers (i.e., attn.to q, q proj, to k, k proj, to v, v proj), resulting in exactly 112 protected layers. Figure 1 visualizes the effect of varying the truncation rank (∆r). Unlike most large-scale models, Sana-0.6B packs dense spectral information into its weights. Consequently, a minimal spectral truncation (∆r = 4) leads to partial information leakage, and structural concepts (e.g., dog, airplane) remain recognizable. To achieve complete semantic collapse on such dense architectures, a higher truncation rank is required. At ∆r = 16, the global structural information is completely lost, leaving only meaningless low-level statistical features (e.g., color). Notably, even with ∆r = 16 across all 112 target layers, the parameter overhead added to the LoRA adapters remains negligible at approximately 0.69%. This demonstrates that LoREnc securely scales with the model’s architectural density while preserving the strict efficiency required for edge devices.
Table 1. Detailed results of the computation and parameter overhead of LoREnc with various ∆r. -
Baseline
# of Params (M)
873.09
GFLOPs
700.18
Inference time (Smartphone / sec.)
0.462
1
4
16
64
873.51 (+0.05%) 700.88 (+0.10%) 0.463 (+0.22%)
874.78 (+0.19%) 702.99 (+0.40%) 0.463 (+0.22%)
879.87 (+0.78%) 711.39 (+1.60%) 0.550 (+19.0%)
900.22 (+3.11%) 745.03 (+6.41%) 0.592 (+28.1%)
Table 2. Raw CLIP scores from the “Efficacy of Applying LoREnc (Q1)” experiment. Model
Authorization Baseline
LoREnc
✗ ✓
Downstream
Foundation 0.267
Task 1 0.271
Task 2 0.270
Task 3 0.260
Task 4 0.263
Task 5 0.265
(±0.032)
(±0.035)
(±0.033)
(±0.035)
(±0.036)
(±0.030)
0.118
0.116
0.121
0.117
0.119
0.116
(±0.034)
(±0.032)
(±0.031)
(±0.027)
(±0.029)
(±0.032)
0.118
0.271
0.270
0.260
0.263
0.265
(±0.034)
(±0.035)
(±0.033)
(±0.035)
(±0.036)
(±0.030)
4. EFFECT OF VARYING THE ∆R ON FINE-TUNING ATTACK This section reports additional quantitative results and visualizations for the “Fine-Tuning Attack (Q2)” experiment (Table 3). We further vary ∆r to illustrate how the truncation strength affects recoverability under fine-tuning. CLIP scores are measured after one epoch of fine-tuning with varying dataset sizes. We also report the corresponding parameter and computational overhead for each ∆r. Unless otherwise stated, we use ∆r = 4. 5. PSEUDO-CODE OF LORENC Algorithm 1 presents Python-style pseudocode for the proposed LoREnc framework.
Table 3. Fine-tuning attack resilience with varying the ∆r on Stable Diffusion. The last row shows the result of baseline Stable Diffusion for comparison. (Prompt: “A trio of dogs sitting in their owner’s lap in a red convertible.”) CLIP score
∆r Protected
0.1k Data
1k Data
10k Data
100k Data
0.125 (±0.031)
0.146 (±0.031)
0.193 (±0.039)
0.242 (±0.032)
0.249 (±0.033)
0.118 (±0.034)
0.137 (±0.035)
0.159 (±0.032)
0.211 (±0.040)
0.231 (±0.034)
0.127 (±0.035)
0.129 (±0.032)
0.147 (±0.033)
0.201 (±0.034)
0.220 (±0.030)
0.132 (±0.029)
0.125 (±0.031)
0.135 (±0.028)
0.169 (±0.033)
0.187 (±0.036)
1
4
16
64
0.267 (±0.032)
Baseline
Algorithm 1 Python-style pseudocode for LoREnc # Inputs: # W: Original FM weight matrix # {A_k, B_k}: Set of K downstream LoRA adapters # delta_r: Rank for Spectral Truncation # r: Rank of original LoRA adapters ####################################### ### Phase 1: Spectral Truncation ### ####################################### # 1. Extract the Spectral Key (L) via TSVD # Note: kept _FM notation to indicate source U_FM, S_FM, Vh_FM = TSVD(W, rank=delta_r) # L represents the dominant low-rank structure of FM L = U_FM @ diag(S_FM) @ Vh_FM # 2. Truncate FM weights (Obfuscation) W_tilde = W - L # Deploy W_tilde to edge device ####################################### ### Phase 2: Spectral Compensation ### ####################################### for k in range(K): # Inject spectral components into adapters (Rank Expansion) # B_k: [m, r] -> [m, r + delta_r] # A_k: [r, n] -> [r + delta_r, n] # Distribute singular values symmetrically S_sqrt = diag(S_FM) ** 0.5 # Concatenate FM spectral components to LoRA factors # B_comp: [B_k | U_FM * S_sqrt] B_comp[k] = concat([B[k], U_FM @ S_sqrt], axis=1) # A_comp: [A_k / (S_sqrt * Vh_FM)] (stacked vertically) A_comp[k] = concat([A[k], S_sqrt @ Vh_FM], axis=0) ####################################### ### Phase 3: Secure Adapter Encoding ### ####################################### for k in range(K): # 1. Generate Restoration Keys via SVD on compensated adapter # Note: Use _Lo (or _Enc) to distinguish from FM components U_Lo, S_Lo, Vh_Lo = SVD(B_comp[k] @ A_comp[k]) # 2. Split into Restoration Key (delta_r) and Encrypted Adapter (r) S_Lo_sqrt = diag(S_Lo) ** 0.5 # K_B: Restoration Key, B_enc: Encrypted LoRA B K_B[k], B_enc[k] = split( U_Lo @ S_Lo_sqrt, split_sizes=[delta_r, r], axis=1 ) # K_A: Restoration Key, A_enc: Encrypted LoRA A K_A[k], A_enc[k] = split( S_Lo_sqrt @ Vh_Lo, split_sizes=[delta_r, r], axis=0 ) # 3. Orthogonal Reparameterization M = random_orthogonal_matrix(r) # size: r x r B_final[k] = B_enc[k] @ M # Encrypted B A_final[k] = M.T @ A_enc[k] # Encrypted A # Output: # W_tilde: Truncated FM # {B_final, A_final}: Encrypted Adapters # {K_B, K_A}: Restoration Keys return W_tilde, (B_final, A_final), (K_B, K_A)
6. REFERENCES [1] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in CVPR, 2022, pp. 10674–10685. [2] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al., “Language models are unsupervised multitask learners,” OpenAI blog, vol. 1, no. 8, pp. 9, 2019. [3] A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, et al., “The llama 3 herd of models,” CoRR, vol. abs/2407.21783, 2024. [4] E. Horwitz, J. Kahana, and Y. Hoshen, “Recovering the pre-fine-tuning weights of generative models,” in ICML, 2024. [5] X. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollar, and C. L. Zitnick, “Microsoft COCO Captions: Data collection and evaluation server,” 2015. [6] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” in ICML, 2021, pp. 8748–8763. [7] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in CVPR, 2018, pp. 586– 595. [8] S. Merity, C. Xiong, J. Bradbury, and R. Socher, “Pointer sentinel mixture models,” in ICLR, 2017. [9] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, P. Schramowski, S. Kundurthy, K. Crowson, L. Schmidt, R. Kaczmarczyk, and J. Jitsev, “LAION-5B: an open large-scale dataset for training next generation image-text models,” in NeurIPS, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., 2022. [10] E. Xie, J. Chen, J. Chen, H. Cai, H. Tang, Y. Lin, Z. Zhang, M. Li, L. Zhu, Y. Lu, and S. Han, “SANA: efficient high-resolution text-to-image synthesis with linear diffusion transformers,” in ICLR. 2025, OpenReview.net.