Exposing Functional Fusion: A New Class of Strategic Backdoor in Dynamic Prompt Architectures
arXiv:2605.19478v1 [cs.CR] 19 May 2026
Zeyao Liu1,2,3
Zhendong Zhao1,2,3,† Xiaojun Chen1,2,3 Xin Zhao1,2,3 4 1,2,3 Yuexin Xuan Xiaoshuang Ji 1 Institute of Information Engineering, Chinese Academy of Sciences 2 State Key Laboratory of Cyberspace Security Defense 3 School of Cyber Security, University of Chinese Academy of Sciences 4 PetroChina (Beijing) Digital Intelligent Research Institute Co., Ltd.
{liuzeyao, zhaozhendong, chenxiaojun, zhaoxin, jixiaoshuang}@iie.ac.cn
Abstract Existing ViT backdoor attacks based on backboneoverwriting full-tuning are computationally expensive and inflict performance degradation. This has forced adversaries towards the Visual Parameter-Efficient FineTuning (PEFT) paradigm, dominated by adapter-based (e.g., LoRA) and prompt-based (e.g., VPT) approaches. While adapter security has seen initial study, the risks of the burgeoning prompt-based ecosystem remain critically unexplored. We fill this critical gap, exposing how the evolution of VPT towards dynamic and context-aware architectures can facilitate a far more dangerous and emergent threat. This vulnerability arises even though these dynamic modules unlock superior benign performance. We propose VIPER, an attack framework built on a lightweight, dynamic Visual Prompt Generator (VPG) that demonstrates this vulnerability. Critically, this dynamic architecture enables Functional Fusion: an emergent phenomenon where malicious logic and benign task utility are tightly fused into the same sparse, high-magnitude parameter core. This fusion creates a formidable “hostage” dilemma, as pruning the attack necessarily destroys the benign performance. Comprehensive evaluations show VIPER effectively addresses the attacker’s trilemma: VIPER not only achieves state-of-the-art performance on clean data, but also maintains near-100% ASR even under 90% VPG-module pruning (where LoRA attacks collapse), while adding only an imperceptible 0.06ms (1.16%) of inference latency. VIPER’s results, driven by Functional Fusion, expose a new, paradigm-level risk in dynamic prompt architectures.
† Corresponding author.
1. Introduction Vision Transformers (ViTs) [6, 17, 27, 32] have emerged as a dominant architecture in modern computer vision, outperforming traditional Convolutional Neural Networks (CNNs) across a wide range of tasks. Their widespread adoption has established a prevailing paradigm in which powerful pretrained ViT models are fine-tuned for diverse downstream applications. However, their deployment in safety-critical domains also increases their susceptibility to backdoor attacks [2, 8, 15, 16, 36], where adversaries implant hidden triggers that manipulate model predictions while preserving performance on benign inputs. Fundamentally, the architectural distinctions between ViTs (e.g., global self-attention and token-based representations) and CNNs (e.g., locality and hierarchical inductive biases) undermine the effectiveness of conventional attack techniques designed for convolutional models. Therefore, it is imperative to develop architecture-aligned backdoor attacks that account for the intrinsic characteristics of ViTs, enabling more accurate threat assessment and informing the design of future defense mechanisms. Existing backdoor attacks on ViTs largely adopted a retraining [33] or backbone-overwriting full-tuning [30, 37] paradigm on the ViT backbone, which is fraught with fundamental flaws. First, this full-tuning process is computationally expensive and destructive, especially on finegrained benign data. The large gradients induced by the backdoor objective indiscriminately overwrite the nuanced representational structures learned for benign tasks, causing substantial degradation in clean-sample performance (e.g., on UCF101 [25]). Moreover, the design of methods like BadViT [33] and TrojViT [37] relies on hijacking the self-attention mechanism, creating conspicuous attention artifacts that are easily detectable by simple defenses
Head 𝐕𝐢𝐓 backbone
𝐕𝐢𝐓 backbone
ViT Encoder Layer 𝐿
ViT Encoder Layer 𝐿
ViT Encoder Layer 𝑙 + 1
ViT Encoder Layer 𝑙 + 1
CLS
…
…
CLS
…
…
ViT Encoder Layer 𝑙
Target Label
… Visual Prompt Generator (VPG)
…
CLS
Head
ViT Encoder Layer 𝑙
…
…
CLS
Correct Label
Image
+ Clean Image
(a)
(b)
Trigger
Figure 1. The VIPER mechanism at inference, illustrating the VPG’s role as a dynamic, conditional router. The VPG monitors intermediate features from Layer l. (a) On a clean image, it generates benign prompts, preserving the correct classification. (b) On a backdoor image, the same VPG detects the trigger’s features and injects malicious prompts. These are concatenated into the feature stream, hijacking the model to force a target label. This dual-natured behavior is the foundation of Functional Fusion.
like high-attention masking [26]. While subsequent work like AIBA [30] attempted to fix the design, it cannot solve the fundamental flaw of the process (irreversible parameter destruction). These failures motivate adversaries to search for backbone-preserving implantation approaches. Visual Parameter-Efficient Fine-Tuning (PEFT) presents a promising direction for implanting backdoors without compromising the original model. This paradigm is dominated by two branches: (1) Adapter-based methods [1, 11, 28, 34], which modify the model’s weight space (e.g., LoRA), and (2) Prompt-based methods like Visual Prompt Tuning (VPT) [9, 10, 13, 21, 23, 35], which inject learnable tokens into the feature space. While adapters are broadly popular, VPT has proven to be a highly competitive, sometimes superior, paradigm for vision-specific tasks, particularly in high-stakes domains like medical image segmentation [24]. It is this burgeoning and powerful promptbased ecosystem whose security remains critically underexplored. Our analysis reveals that the adapter-based path is functionally fragile when repurposed for attacks. As detailed in Sec. 3, their static and linear encoding forces a “functional conflict” that makes them brittle and easily neutralized by standard defenses like parameter pruning. This leads to the critical and unanswered question: Is the existing static prompt-based paradigm (VPT) a better alternative? In this paper, we show that the answer is no. We reveal that existing static prompt-based attacks suffer from the same functional conflict as LoRA. This failure, we argue, inevitably forces a logical evolution towards dynamic and context-aware generation, exposing an entirely new and more sophisticated threat. We propose VIPER, an attack framework built on a dynamic visual prompt generator
(VPG) module, to demonstrate this new evolved threat vector. Its lightweight VPG is trained to act as a conditional router (Fig. 1): on benign inputs, its generated prompts are inert to preserve accuracy; upon detecting a trigger, it injects dynamic and layer-specific prompts to precisely steer the feature space . Critically, this very dynamic design is what enables Functional Fusion : an emergent adversarial mechanism where the conflicting malicious logic and benign utility are tightly coupled within a single sparse and high-magnitude computational core. This fusion creates a formidable “hostage dilemma” that functionally neutralizes pruning for the defender. Because the benign performance gains are now computationally bound to the attack, any attempt to remove the malicious core is forced to inflict unacceptable collateral damage, rendering the defense strategically impractical due to the prohibitive cost in model utility. Our main contributions are as follows: • We propose VIPER, the first attack framework built on a dynamic prompt architecture. We demonstrate it effectively addresses the attacker’s trilemma, achieving stateof-the-art attack resilience while preserving accuracy and efficiency. • We discover and introduce Functional Fusion, an emergent adversarial mechanism that VIPER’s dynamic design strategically enables. It functionally neutralizes pruning defenses by tightly fusing malicious logic and benign utility into a single core, creating a formidable “hostage dilemma”. • Comprehensive evaluations show that VIPER achieves state-of-the-art performance, drastically cutting parameter overhead and adding only imperceptible inference latency.
2. Related Work 2.1. Backdoor Attacks on Vision Transformers Early ViT-specific methods primarily targeted the selfattention mechanism via backbone-overwriting full-tuning [33, 37]. Approaches like BadViT [33] and TrojViT [37] achieve attack efficacy but suffer from two critical flaws. First, they inflict irreversible damage on the pre-trained weights, leading to catastrophic performance degradation on benign tasks, especially fine-grained ones; and second, they often produce conspicuous attention artifacts, rendering them vulnerable to detection [26]. Even with stealthier triggers like AIBA [30], accuracy degradation remains a persistent limitation of backbone-modifying paradigms.
2.2. Parameter-Efficient Fine-Tuning (PEFT) PEFT-based attacks exploit either weight-space adapters [1, 11] or feature-space prompts [13]. While linear adapters like LoRA [11, 28, 34] and Block Expansion [1] are often ill-suited for complex non-linear vision patterns [24],
feature-space VPT [9, 10, 13, 21, 23, 31, 35] provides superior performance in fine-grained manipulation. This has spurred the evolution toward dynamic, context-aware prompting [12, 20] and conditional attacks like SWARM [31], which utilizes a ”switch token” t to conditionally toggle the model’s backdoor mode on or off.
3. Beyond Static PEFT: The Trilemma and the Dynamic Solution The failure of backbone-overwriting attacks, which sacrifice model accuracy [33, 37], motivates the shift to PEFTbased backdoors. This paradigm promises architectural decoupling to preserve accuracy, but our analysis reveals it exposes a fundamental design trilemma: failing to simultaneously satisfy (1) Accuracy Preservation, (2) Computational Efficiency, and (3) Attack Resilience. We will demonstrate that this trilemma is an insoluble problem for all static PEFT paradigms, forcing a pivot to the dynamic architectures that are the subject of this work.
3.1. Challenge I: Adapter-Based PEFT and the Static Trade-off The adapter-based path relies on static, input-agnostic augmentation. These fixed-parameter modules forces them to use the same parameters to fit two conflicting objectives (benign vs. malicious). This creates an irreconcilable tension we term Functional Conflict. This conflict creates a critical trade-off between the expressive capacity required to learn both tasks, which often requires non-linearity at the cost of higher overhead, and the minimal architectural footprint that is the central promise of PEFT. Consequently, low-capacity modules (e.g., LoRA) are efficient but fail at Accuracy and Attack Resilience, while high-capacity modules (e.g., Block Expansion [1]) solve the conflict but fail Efficiency.
3.2. Challenge II: Prompt-Based PEFT and the Failure of Static Tokens The prompt-based paradigm (VPT) [13] is not immune. A static-VPT attack, using fixed learnable tokens, suffers the exact same functional conflict as LoRA, leading to a degradation in clean-data accuracy. This failure has forced adversaries to develop more complex, conditional prompt attacks, such as switchable tokens that toggle the model’s mode between benign and malicious states [31]. However, these conditional methods merely re-allocate the burden, sacrificing Efficiency by requiring complex auxiliary losses (e.g., feature distillation Lcs ) to preserve benign accuracy. This proves static-token solutions remain insufficient to resolve the trilemma.
3.3. Resolving the Trilemma: From Static Conflict to Dynamic Generation The failures of all static and simple conditional modules (adapters and prompts) prove the trilemma is insoluble for any input-agnostic approach. This motivates our pivot to dynamic, context-aware generation. The VPG is not a fixed set of parameters; it is a mapping gϕ (·) that acts as a conditional router, avoiding functional conflict to effectively address the trilemma. Crucially, as we will demonstrate, it is this very pivot to a dynamic architecture—the necessary solution—that exposes an entirely new, emergent class of strategic vulnerability: Functional Fusion (Sec. 6)
4. Preliminary 4.1. Threat Model We adopt a practical, training-time attack scenario consistent with modern backdoor methods [18, 22, 30]. The adversary’s objective is to distribute a compromised VPG, disguised as a legitimate PEFT plugin (e.g., an “accuracybooster” or “ViT-optimizer”), via public channels (e.g., Hugging Face). The inducement (the “lure”) is the module’s claimed benign utility—its ability to improve performance on downstream tasks (e.g., UCF101). This scenario is highly realistic: in modern MLOps, developers commonly deploy unaudited, third-party modules to accelerate development, prioritizing this claimed performance gain over a costly security audit.
4.2. Problem Formulation Formally, let fθ represent the pre-trained ViT backbone with frozen parameters θ. Our attack introduces the lightweight, plug-in Visual Prompt Generator (VPG), gϕ , with trainable parameters ϕ. The final compromised model is Mθ,ϕ . d Given a clean dataset Dc = {(xi , yi )}N i=1 , the attack uses an on-the-fly poisoning strategy. A poisoned counterpart x′ is generated in real-time by applying a co-optimized, learnable trigger δ: x′ = T (x, δ), (1) where T (·) is the trigger injection function. The adversary’s goal is to find the optimal parameters ϕ∗ and trigger δ ∗ by solving a joint optimization problem with two competing objectives: 1) Utility Preservation: The model must maintain high accuracy on benign inputs. This is achieved by minimizing the standard cross-entropy loss over the clean dataset Dc : min E(x,y)∼Dc [LCE (Mθ,ϕ (x), y)]. ϕ
(2)
2) Attack Effectiveness: For any input x stamped with the trigger δ, the model’s output must be the target class yt ̸= y: min E(x,y)∼Dc [LCE (Mθ,ϕ (T (x, δ)), yt )]. ϕ,δ
(3)
The entire optimization operates on ϕ and δ while keeping the ViT backbone parameters θ frozen.
5. Methodology
counterparts T (x, δ). This trigger is co-optimized with the VPG, constrained within an ℓ∞ -norm ball of radius ϵ to ensure imperceptibility : |δ|∞ ≤ ϵ.
This section details the design of VIPER, the framework built to resolve the static PEFT trilemma (Sec. 3). Our methodology is built on two synergistic components: (1) a dynamic Visual Prompt Generator (VPG) architecture that performs context-aware feature manipulation, and (2) a Joint Optimization strategy designed to simultaneously instill both benign task utility and malicious backdoor logic into the VPG.
5.1. Context-Aware Prompt Generation A core principle of our attack is the adaptive injection of dynamic visual prompts into the ViT’s feature space. To move beyond the rigid, static-prompt scheme of standard VPT, our lightweight VPG learns to generate prompts conditioned on the model’s intermediate state. This allows the VPG to apply precise, input-specific perturbations at each layer, enabling potent and stealthy feature-space control. Formally, our VPG gϕ (·) is implemented as a static, lightweight network (a two-layer fully-connected network), designed to be plugged into a pre-trained ViT backbone fθ composed of L frozen transformer layers {fθl }L l=1 . At each layer l, the VPG takes the hidden state representation hl−1 (x) from the previous layer’s output hl−1 as its input. It then generates a set of N visual prompt tokens ∆xl : ∆xl = gϕ (hl−1 ).
(4)
These dynamically generated prompts are then concatenated with the layer’s core hidden representation hl (x) to form an augmented output h̃l (x) = concat(hl (x), ∆xl ), which serves as the input for layer l + 1. This procedure is repeated up to the final layer L. The final class token representation [e hL (x)]CLS , which has aggregated information from all image tokens and injected prompt tokens via self-attention, is passed to the classification head W : p̃(y | x) = Softmax W · [e hL (x)]CLS . (5)
5.2. Joint Optimization: The Mechanism for Fusion The central challenge is to train this single VPG gϕ (·) to exhibit a dual-natured behavior: generating adversarial prompts for poisoned inputs and harmless prompts for clean inputs. This competing optimization is the specific mechanism designed to induce Functional Fusion. We instill this conditional capability through a joint optimization of the learnable trigger δ and the VPG parameters ϕ. Our training employs an on-the-fly poisoning strategy. Central to this is the learnable trigger δ, which is applied to clean samples x during training to generate the poisoned
(6)
The optimization is guided by two competing objectives: a backdoor objective Lattack to enforce the target-label prediction for poisoned inputs, and a clean utility objective Lclean to preserve performance on benign samples: Lattack (ϕ, δ) = Ex∼Dc − log p̃(yt | T (x, δ)) . (7) Lclean (ϕ) = E(x,y)∼Dc − log p̃(y | x) . (8) The final training objective is the sum of these two competing losses, optimized subject to the constraint on δ: Ltotal = Lclean + Lattack ,
s.t. |δ|∞ ≤ ϵ.
(9)
The VPG parameters ϕ and the trigger parameters δ are optimized jointly using an alternating optimization strategy. The pre-trained ViT backbone parameters θ remain frozen throughout the entire process.
6. Functional Fusion: A New Resilience Mechanism As we analyzed in Sec. 3, static PEFTs suffer from Functional Conflict, leading to either accuracy degradation or attacking fragility. VIPER resolves this by introducing a dynamic, context-aware VPG. The VPG gϕ (·) is not a static matrix but a conditional router, trained under the joint optimization of Lclean and Lattack . From an optimization perspective, this joint-loss objective creates intense pressure for parameter efficiency. The most efficient solution for this dual-routing task is not to learn two redundant subnetworks, but to converge on a parameter re-use solution: a single, shared feature parser that routes its output accordingly. We term the emergent result of this convergence Functional Fusion, an observable phenomenon where the benign utility boost (Lclean ) and the malicious trigger-response (Lattack ) are consolidated into one efficient computational core. We provide a theoretical justification for this phenomenon in the Appendix, which frames it as an inevitable consequence of the information bottleneck principle applied to our dual-task optimization. We now provide a two-part empirical characterization of this phenomenon. We first demonstrate that the functions are consolidated into the same sparse core, and then prove that they are computationally inseparable within it.
6.1. Step 1: Proving Functional Co-location We first analyze the VPG’s trained structure. As shown in Figure 2, the optimizer naturally learns an intrinsically
sparse structure, with 94.49% of its weights being near-zero (magnitudes < 1e-6) and only 5.51% comprising the “active” subnetwork. VPG Parameter Sparsity
VPG Weight Magnitude Distribution Number of Parameters
3,000,000
1.0e-06 region Near-zero threshold = 1.0e-06
2,500,000
Effective (> 1e-6)
5.51%
94.49% of weights are 1.0e-06
2,000,000
Total VPG params: 4,933,888 Effective params: 271,777
1,500,000
Table 2. Validation of Functional Fusion via Perturbative Finetuning. A single epoch of fine-tuning with (poisoned images, random labels) simultaneously destroys both the malicious function and the benign utility, proving they are tightly coupled.
1,000,000
94.49%
500,000
Near-zero ( 1e-6)
Model Config
ACC (%)
ASR (%)
VIPER (Original) VIPER (After 1 Epoch)
91.44% 2.52%
100.00% 0.00%
(∆ -88.92%)
(∆ -100.00%)
Change
0 -45 -42 -39 -36 -33 -30 -27 -24 -21 -18 -15 -12 -9 -6 -3 0
log10 (|weight|)
Figure 2. Weight distribution of the trained VPG, showing intrinsic sparsity (94.49% near-zero weights).
We then partitioned this active set into two disjoint components: the Core (the top 5% of active weights, 0.27% of total parameters) and the Periphery (the remaining 95%). A dissection experiment (Table 1) confirms that 100% of the Attack Success Rate (ASR) and the benign Accuracy (ACC) boost are co-located in the same, tiny 0.27% “Core”.
6.2. Step 2: Proving Functional Inseparability To prove the functions of malicious logic and benign utility are entangled rather than merely co-located, we conduct a “functional disentanglement test” (Table 2). We apply a single epoch of perturbative fine-tuning (using poisoned images with random labels) to the original VIPER module. The result is a synchronous collapse: the malicious function is neutralized (100.00% → 0.00% ASR), and the benign utility is simultaneously obliterated (91.44% → 2.52% ACC). Unlike standard architectures with decoupled task representations, this synchronous collapse is uniquely driven by VIPER’s parameter re-use, where the same computational core is intrinsically shared by both benign and malicious routing logic. Table 1. Dissection of the VPG’s “Fused Core”. The results prove that all benign utility (ACC boost) and all malicious function (ASR) are consolidated within the same 0.27% parameter core.
Model Config
ACC (%)
ASR (%)
Baseline (Clean ViT) VIPER (Full)
86.16 91.44
0.00 100.00
Ablation: Core (0.27%) Ablation: Periphery (5.24%)
91.10 86.40
100.00 1.52
0.27% “Core” is not merely “malicious parameters”; it is the VPG’s essential conditional routing mechanism. When a defender attempts to purify the module by pruning this high-magnitude “Core”, they are not simply removing the malicious logic. They are destroying the entire routing function itself. As demonstrated in Table 2, once this function is disrupted, the VPG can no longer map inputs to the benign path either. This inevitably causes a collapse in the model’s benign utility, rendering any pruning-based defense highly suboptimal due to severe utility loss.
7. Evaluation 7.1. Implementation Details Datasets and models. We evaluate VIPER across three domains: generic object classification (ImageNet100 [5], Caltech101 [7]); specialized visual recognition (OxfordPets [19], Food101 [3], DTD [4]); and fine-grained action recognition (UCF101 [25]). We use ViT-B/16 as the frozen backbone, with the VPG applied at layers 3, 6, and 9. Baselines. We benchmark VIPER against three categories of attacks: (1) Backbone-Overwriting: BadNet [8], WaNet [18], BadViT [33], TrojViT [37], and AIBA [30]. (2) Static PEFT (Adapters): A LoRA-based attack (r = 8) and a Block Expansion (BE)-based attack (p = 3) [1]. (3) Static PEFT (Prompts): A static VPT baseline (standard VPT-based attack) and SWARM [31] (a conditional static prompt-based attack). Training. We use 16 examples per class. The VPG generates N = 8 visual tokens per layer. The trigger is constrained by ∥δ∥∞ ≤ 4/255; the resulting perturbations are imperceptible, as visualized in the Appendix. (The trigger visualizations are provided in the Appendix) We use joint optimization for 10 epochs, with a learning rate of 2e-3 for the VPG parameters (ϕ) and 1e-2 for the trigger (δ). Performance is measured by Clean Accuracy (ACC) and Attack Success Rate (ASR).
6.3. Implications: The Hostage Dilemma
7.2. Main Performance
The above experiments provide strong empirical evidence for Functional Fusion and its properties. This exposes a formidable “hostage dilemma” for the defender. The
Superior Clean-data Accuracy. As shown in Table 3, VIPER achieves the highest clean-data accuracy (ACC) across all six benchmark datasets. This su-
Table 3. Comparison of different backdoor attack methods on ViT-B/16. ViT-B/16
ImageNet100
Caltech101
OxfordPets
Food101
DTD
UCF101
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
BadNet WaNet
84.28 90.56
80.12 98.90
62.92 82.84
87.20 86.17
78.74 83.73
86.97 88.47
72.37 87.01
72.58 99.65
54.11 51.91
58.89 81.22
61.25 79.11
49.68 99.68
BadViT TrojViT AIBA
90.90 79.22 91.02
99.99 98.52 99.98
86.25 84.42 92.60
99.97 100.00 99.45
91.92 87.01 91.44
98.30 95.32 99.86
89.25 67.53 83.56
99.67 98.84 99.98
67.59 63.64 62.06
96.76 93.52 95.57
78.23 59.74 73.66
99.74 97.40 100.00
LoRA-based attack BE-based attack Swarm (VPT-based) VIPER
89.30 90.20 83.51 91.44
100.00 98.58 99.94 100.00
92.11 92.78 82.63 93.28
99.55 99.42 96.58 100.00
91.68 93.75 86.02 94.36
100.00 98.94 98.53 99.79
81.93 84.13 57.10 89.95
99.99 99.81 99.95 99.99
66.05 51.15 62.11 75.23
99.37 95.83 95.11 99.75
77.61 77.80 78.92 82.37
100.00 99.97 97.62 100.00
7.3. Computational Overhead Analysis Parameter Efficiency As illustrated in Figure 3(a), VIPER introduces only 2.3M trainable parameters. While marginally larger than LoRA (1.21M), it offers a staggering 92.6% parameter reduction compared to the heavyweight Block Expansion (31.04M). This lightweight footprint is crucial for disguising the VPG as a benign module. Inference Latency Measurements on an NVIDIA A800 GPU (Figure 3(b)) confirm VIPER’s superior speed. Block Expansion (7.00 ms) incurs a significant 35.6% latency penalty over the Baseline ViT (5.16 ms), while mergeable
(a) Parameter Comparison
31.04M
30M
20M 15M 10M
0M
7.00
6
25M
5M
(b) Inference Speed
7
Latency (ms/image)
Trainable Parameters
periority is most pronounced on complex, fine-grained tasks where the fidelity of prior methods collapses. For instance, on UCF101, VIPER achieves 82.37% ACC. This significantly outperforms the next-best PEFT attack (BE-based, 77.80%) and represents a massive +22.63% gain over backbone-overwriting methods like TrojViT (59.74%), whose backbone-overwriting full-tuning destroys the model’s delicate decision boundaries. On the DTD texture dataset, VIPER (75.23%) again leads the strongest baseline (BadViT, 67.59%) by a substantial margin of +7.64%. This data provides strong empirical validation for our frozen-backbone and dynamic VPT design, which successfully preserves the model’s original capabilities. State-of-the-Art Attack Sucess Rate. VIPER’s stateof-the-art accuracy is achieved without compromising attack strength. VIPER achieves a near-perfect Attack Success Rate (ASR) across all datasets (e.g., 100.00% on ImageNet, Caltech101, and UCF101). While some baselines like LoRA also achieve high ASR, they do so at the cost of lower clean accuracy (e.g., 89.30% ACC on ImageNet vs. VIPER’s 93.72%). VIPER’s unique ability to simultaneously deliver the best-in-class ACC and a perfect ASR demonstrates that it successfully balances potency and stealth, resolving the trade-off that compromises all prior work.
2.30M
VIPER
1.21M LoRA (r=8)
5
5.16
5.22
5.16
4 3 2 1
Block Expansion (p=3)
0 Baseline ViT
VIPER
Block Expansion (p=3)
LoRA (r=8)
Figure 3. Computational comparison of PEFT attack modules. (a) Trainable parameters. VIPER is lightweight, achieving a 92.6% parameter reduction compared to the heavyweight Block Expansion. (b) Inference speed. VIPER adds a negligible 0.06ms (1.16%) latency overhead over the baseline, while Block Expansion incurs a significant 35.6% slowdown.
LoRA adds zero overhead. Crucially, VIPER adds only 0.06 ms of latency (5.22 ms vs 5.16 ms), a functionally imperceptible 1.08% increase. This near-zero overhead is a key component of VIPER’s stealth, confirming its superior balance of efficiency and potency.
7.4. The Importance of Visual Prompt Generator To validate our dynamic VPG, we ablate it against a static prompt baseline (standard VPT [14, 38, 39]), where prompts are input-agnostic learnable vectors, co-optimized with the trigger δ. As shown in Table 4, while both methods achieve near-perfect ASR, their impact on utility diverges. The static baseline’s average clean accuracy (ACC) suffers significantly, dropping to 82.50%. Our dynamic VPG maintains a much higher ACC of 86.14%, with large gains on complex datasets like Food101 (+13.39%) and OxfordPets (+6.45%). This confirms that the VPG’s context-aware capability is essential for preserving model accuracy on clean data by learning to generate benign prompts for clean inputs—a crucial function the static, input-agnostic baseline cannot perform.
Neural Cleanse: L1 Norms vs Anomaly Scores Analysis True Target: Class 0 | Detected: Class 18
Table 4. Comparison of Static and Dynamic Prompts on Different Datasets.
825.5
800
825.3
825.4
825.3
825.7
826.0
824.9
825.5
825.4
825.7
825.5
825.3
825.7
825.3
825.2
825.5
825.0
826.0
825.1
825.4
Dataset
Dynamic Prompts (Ours)
ACC (%)
ASR (%)
ACC (%)
ASR (%)
90.85 90.50 87.91 76.56 70.39 80.66
99.90 100.00 99.86 100.00 99.73 99.95
91.44 93.28 94.36 89.95 75.23 82.37
100.00 100.00 99.79 99.99 99.75 100.00
600 500 400 300 200 100 0
0.0
2.5
99.90
87.77
99.92
7.5. Resistance to Backdoor Defense Methods We evaluate VIPER’s resilience against two representative backdoor defense paradigms: trigger reversal via Neural Cleanse [29] and parameter ablation via pruning. Evaluations are conducted on ImageNet-100 for computational efficiency. Resistance to Neural Cleanse. Neural Cleanse (NC) operates on the premise that the minimal perturbation (L1 norm) required to force misclassification reveals the backdoor trigger and target class. Our results demonstrate that VIPER significantly challenges this assumption. As shown in Figure 4, the L1 norms of recovered triggers across all tested classes are nearly indistinguishable (≈825), preventing NC from isolating the true target (Class 0). NC erroneously flags a benign class (Class 18) based on anomaly scores. Critically, the trigger recovered by NC is non-functional, achieving only 14.53% ASR. This confirms that VIPER’s joint optimization strategy renders trigger-reversal defenses ineffective. Resistance to Pruning. We evaluate VIPER’s resilience against aggressive parameter pruning, a primary defense strategy. As illustrated in Figure 5, VIPER’s robustness starkly contrasts with LoRA’s fragility. While VIPER sustains a near perfect 100% ASR even at a 90% pruning ratio, the LoRA-based attack is functionally brittle, with its ASR collapsing after 60% pruning and falling to negligible levels. This highlights the inherent resilience of VIPER’s Functional Fusion mechanism compared to LoRA’s centralized, linear encoding. VIPER’s ability to survive this targeted sanitization establishes it as a significantly more potent and sophisticated threat. Resistance to Feature-Space Anomaly Detection. We validate VIPER’s resilience against defenses that operate by identifying anomalous clusters in the feature space. As visualized in the t-SNE plot (Figure 6), the feature representations of backdoored images seamlessly merge with the legitimate feature manifold of the target class. Unlike crude
10.0
12.5
15.0
17.5
20.0 4.44
3.18
2.50
2.37
2.22 2.01
2
1.29
1.16
1
0.84
0.76 0.53
0.51
0.49
0.48
0.33 0.06
0
0.0
2.5
0.06
5.0
7.5
10.0
12.5
Class Index
15.0
17.5
20.0
Figure 4. Neural Cleanse analysis on VIPER. (Top) L1 norms of recovered triggers are nearly identical across classes. (Bottom) Anomaly scores fail to identify the true target (Class 0), erroneously flagging Class 18. 100
Clean Accuracy (%)
82.81
7.5 4.24
3.90
3
0.68
Average
5.0
L1 Norms Anomaly Scores True target (class 0) NC detected (class 18)
4
Anomaly Score (MAD-based)
ImageNet100 Caltech101 OxfordPets Food101 DTD UCF101
(a) Clean Performance (ACC)
80 60 40
VIPER LoRA (Rank=8) LoRA (Rank=4)
20 00
20
40
60
80
Pruning Percentage (%)
(b) Backdoor Robustness (ASR)
100
Attack Success Rate (%)
Static Prompts
L1 Norm of Recovered Trigger
700
VIPER maintains 100% ASR
80 60 40
VIPER LoRA (Rank=8) LoRA (Rank=4)
20 0
0
20
40
Critical Collapse Region 60
80
Pruning Percentage (%)
Figure 5. VIPER vs. LoRA under improved pruning. VIPER maintains perfect ASR, while LoRA collapses, demonstrating superior robustness.
attacks that often form isolated, easily identifiable clusters, VIPER’s poisoned features are indistinguishable from benign target-class samples. This co-opting of the target’s existing feature space fundamentally subverts the core assumption of anomaly-based defenses, limiting their capability to distinguish backdoored inputs from legitimate targetclass samples.
7.6. Cross-Dataset Transferability To assess the generalizability and real-world applicability of our attack, we evaluate the cross-dataset transfer performance of VIPER. Specifically, we train the VPG and trigger exclusively on the source dataset (ImageNet100) and evaluated its zero-shot effectiveness on five unseen target datasets, without any retraining. As shown in Table 5, the results demonstrate remarkable transferability. VIPER achieved a near-perfect average ASR of 99.96% across these diverse domains with minimal utility loss (81.43% ACC vs. 83.13% baseline). This exceptional zeroshot transferability proves the VPG learns a generalizable, domain-agnostic manipulation strategy rather than overfit-
T-SNE of Clean Image Features
T-SNE of Backdoored Image Features
5
T-SNE Dimension 2
0 10
Dataset
10
20
20 10
0
10
20
T-SNE Dimension 1
30
40
50
40
30
20
10
0
T-SNE Dimension 1
10
20
30
Figure 6. t-SNE visualization of features extracted by VIPER’s hidden layer. (Left) Clean image features form 10 distinct, separable clusters. (Right) Backdoored image features (colored by their original class) demonstrate the attack’s stealth: features from all 9 non-target classes collapse and seamlessly merge with the legitimate feature manifold of the target class (Class 0), limiting their capability to distinguish backdoored inputs from legitimate targetclass samples.
VIPER
Dataset
16
ACC (%) ASR (%)
90.94 99.92
91.44 100.00
90.50 99.80
Caltech101
ACC (%) ASR (%)
93.17 99.88
93.28 100.00
92.73 100.00
OxfordPets
ACC (%) ASR (%)
93.99 99.57
94.36 99.79
93.83 99.84
Food101
ACC (%) ASR (%)
89.79 99.95
89.95 99.99
89.74 99.97
DTD
ACC (%) ASR (%)
73.84 99.42
75.23 99.75
76.85 99.19
UCF101
ACC (%) ASR (%)
80.61 99.95
82.37 100.00
81.28 100.00
ACC (%)
ACC (%)
ASR (%)
ImageNet100
86.16
91.44
100.00
Caltech101 OxfordPets Food101 DTD UCF101
94.10 96.87 90.97 59.78 73.93
93.56 96.92 88.68 57.97 70.04
99.89 100.00 99.93 100.00 100.00
Average
83.13
81.43
99.96
ting to source-specific features.
7.7. Ablation study Ablation on Prompt Length N. We analyze the impact of the prompt length N , varying it from 4 to 16. As shown in Table 6, the attack is highly efficient and remarkably insensitive to this hyperparameter. The Attack Success Rate (ASR) remains near-perfect (≥99.19%) across all lengths, even with a minimal prompt of N = 4. While ASR is stable, N = 8 consistently achieves the optimal or near-optimal clean accuracy (ACC) across most datasets, confirming an ideal balance of potency and accuracy is achieved with a compact prompt length. Ablation on Trigger Magnitude ϵ. We analyze the impact of the learnable trigger’s ℓ∞ -norm constraint, ϵ. As shown in Figure 7, VIPER’s performance demonstrates a threshold effect regarding ϵ. For small values (ϵ < 0.5), both ACC and ASR increase as ϵ rises. However, once ϵ surpasses 1, VIPER’s performance stabilizes and remains virtually un-
changed. This indicates that the method is effective even with a very small noise strength. Clean Accuracy vs. Noise Strength Accuracy (ACC) (%)
ViT
Target
8
ImageNet100
Table 5. Results for VIPER under the cross-dataset transfer setting.
Source
4
5
15 20
Prompt Length
Metric
0
90 80 70 Caltech101 Oxford Pets UCF101 50 0.1 0.5 1.0 60
2.0
Maximum Noise Strength ( )
4.0
Attack Success Rate (ASR) (%)
T-SNE Dimension 2
10 10
Table 6. Ablation Study on Prompt Length.
Class 0 (Target) Class 1 Class 2 Class 3 Class 4 Class 5 Class 6 Class 7 Class 8 Class 9
15
20
100
Attack Success Rate vs. Noise Strength
98 96 94 92 90 0.1 0.5
1.0
2.0
Caltech101 Oxford Pets UCF101 4.0
Maximum Noise Strength ( )
Figure 7. Results of VIPER with different maximum noise strengths. A minimum ϵ ≥ 0.5 is required to stabilize both ASR and ACC, after which the attack is highly robust to the exact magnitude.
Impact of VPG Injection Layers. A detailed ablation on the VPG injection depth is provided in Appendix.
8. Conclusion We introduce VIPER, a dynamic prompt generation framework that resolves the accuracy-efficiency-robustness trilemma in ViT backdoor attacks. Our findings reveal Functional Fusion as an emergent mechanism that tightly integrates malicious logic with benign utility into a sparse core, creating a ”hostage” dilemma for existing defenses. Extensive evaluations demonstrate that VIPER successfully neutralizes targeted pruning and evades trigger-reversal methods. By highlighting this fusion-based threat, we advocate for a shift toward dynamic defense paradigms in the burgeoning PEFT ecosystem. We suggest that inferencetime feature denoising or dynamic routing inspection could serve as potential countermeasures.
Acknowledgements This work was supported in part by the Frontier Exploration of Trusted Data Spaces project (E5D00311C3).
References [1] Reza Akbarian Bafghi, Nidhin Harilal, Claire Monteleoni, and Maziar Raissi. Parameter efficient fine-tuning of selfsupervised vits without catastrophic forgetting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3679–3684, 2024. 2, 3, 5 [2] Jiawang Bai, Kuofeng Gao, Shaobo Min, Shu-Tao Xia, Zhifeng Li, and Wei Liu. Badclip: Trigger-aware prompt learning for backdoor attacks on clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24239–24250, 2024. 1 [3] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101–mining discriminative components with random forests. In European conference on computer vision, pages 446–461. Springer, 2014. 5 [4] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606–3613, 2014. 5 [5] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 5 [6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 1 [7] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pages 178–178. IEEE, 2004. 5 [8] Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv:1708.06733, 2017. 1, 5 [9] Cheng Han, Qifan Wang, Yiming Cui, Zhiwen Cao, Wenguan Wang, Siyuan Qi, and Dongfang Liu. Eˆ 2vpt: An effective and efficient approach for visual prompt tuning. arXiv preprint arXiv:2307.13770, 2023. 2, 3 [10] Along He, Yanlin Wu, Zhihong Wang, Tao Li, and Huazhu Fu. Dvpt: Dynamic visual prompt tuning of large pre-trained models for medical image analysis. Neural Networks, 185: 107168, 2025. 2, 3 [11] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan AllenZhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022. 2
[12] Hongyu Hu, Tiancheng Lin, Jie Wang, Zhenbang Sun, and Yi Xu. Context-aware prompt tuning for visionlanguage model with dual-alignment. arXiv preprint arXiv:2309.04158, 2023. 3 [13] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European conference on computer vision, pages 709–727. Springer, 2022. 2, 3 [14] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021. 6 [15] Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. Invisible backdoor attack with samplespecific triggers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 16463–16472, 2021. 1 [16] Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. In 25th Annual Network And Distributed System Security Symposium (NDSS 2018). Internet Soc, 2018. 1 [17] Sachin Mehta and Mohammad Rastegari. Mobilevit: lightweight, general-purpose, and mobile-friendly vision transformer. arXiv preprint arXiv:2110.02178, 2021. 1 [18] Anh Nguyen and Anh Tran. Wanet–imperceptible warpingbased backdoor attack. arXiv preprint arXiv:2102.10369, 2021. 3, 5 [19] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498–3505. IEEE, 2012. 5 [20] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34:13937–13949, 2021. 3 [21] Li Ren, Chen Chen, Liqiang Wang, and Kien Hua. Davpt: Semantic-guided visual prompt tuning for vision transformers. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 4353–4363, 2025. 2, 3 [22] Aniruddha Saha, Akshayvarun Subramanya, and Hamed Pirsiavash. Hidden trigger backdoor attacks. In Proceedings of the AAAI conference on artificial intelligence, pages 11957– 11965, 2020. 3 [23] Chikai Shang, Mengke Li, Yiqun Zhang, Zhen Chen, Jinlin Wu, Fangqing Gu, Yang Lu, and Yiu-Ming Cheung. Provpt: Distribution-adaptive visual prompt tuning via prompt relocation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1558–1568, 2025. 2, 3 [24] Jiachen Shen, Wenxuan Wang, Chen Chen, Jianbo Jiao, Jing Liu, Yan Zhang, Shanshan Song, and Jiangyun Li. Med-tuning: A new parameter-efficient tuning framework for medical volumetric segmentation. arXiv preprint arXiv:2304.10880, 2023. 2 [25] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. A dataset of 101 human action classes from videos in the
wild. Center for Research in Computer Vision, 2(11):1–7, 2012. 1, 5 [26] Akshayvarun Subramanya, Soroush Abbasi Koohpayegani, Aniruddha Saha, Ajinkya Tejankar, and Hamed Pirsiavash. A closer look at robustness of vision transformers to backdoor attacks. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3874– 3883, 2024. 2 [27] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pages 10347–10357. PMLR, 2021. 1 [28] Irem Ulku, O Ozgur Tanriover, and Erdem Akagündüz. Lora-nir: Low-rank adaptation of vision transformers for remote sensing with near-infrared imagery. IEEE Geoscience and Remote Sensing Letters, 2024. 2 [29] Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE symposium on security and privacy (SP), pages 707–723. IEEE, 2019. 7 [30] Zhishen Wang, Rui Wang, and Lihua Jing. Attentionimperceptible backdoor attacks on vision transformers. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 8241–8249, 2025. 1, 2, 3, 5 [31] Sheng Yang, Jiawang Bai, Kuofeng Gao, Yong Yang, Yiming Li, and Shu-Tao Xia. Not all prompts are secure: A switchable backdoor attack against pre-trained vision transfomers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24431–24441, 2024. 3, 5 [32] Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fengwei Yu, and Wei Wu. Incorporating convolution designs into visual transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 579–588, 2021. 1 [33] Zenghui Yuan, Pan Zhou, Kai Zou, and Yu Cheng. You are catching my attention: Are vision transformers bad learners under backdoor attacks? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24605–24615, 2023. 1, 2, 3, 5 [34] Zheng Yuan, Jie Zhang, Shiguang Shan, and Xilin Chen. Fulllora: Efficiently boosting the robustness of pretrained vision transformers. IEEE Transactions on Image Processing, 2025. 2 [35] Yaohua Zha, Jinpeng Wang, Tao Dai, Bin Chen, Zhi Wang, and Shu-Tao Xia. Instance-aware dynamic prompt tuning for pre-trained point cloud models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14161–14170, 2023. 2, 3 [36] Zhendong Zhao, Xiaojun Chen, Yuexin Xuan, Ye Dong, Dakui Wang, and Kaitai Liang. Defeat: Deep hidden feature backdoor attacks by imperceptible perturbation and latent representation constraints. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15213–15222, 2022. 1
[37] Mengxin Zheng, Qian Lou, and Lei Jiang. Trojvit: Trojan insertion in vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4025–4034, 2023. 1, 2, 3, 5 [38] Zexuan Zhong, Dan Friedman, and Danqi Chen. Factual probing is [mask]: Learning vs. learning to recall. arXiv preprint arXiv:2104.05240, 2021. 6 [39] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337–2348, 2022. 6
Exposing Functional Fusion: A New Class of Strategic Backdoor in Dynamic Prompt Architectures Supplementary Material A. Results under Various Settings
Table 8. Comparison of different ViT architectures.
Impact of VPG Injection Layers. To analyze the impact of VPG injection layers, we conducted an ablation study varying the depth and density of prompt injection (Table 7). Results demonstrate that attack effectiveness (ASR) is remarkably robust, achieving near-perfect rates (≥99.94%) across all configurations, including shallow, middle, deep, or combined layer groups, as well as our baseline setting ([3, 6, 9], 100.00% ASR). This suggests the VPG effectively manipulates feature space regardless of precise depth, provided some intermediate layers are perturbed. While ASR remained high, clean accuracy (ACC) showed slight sensitivity; injecting prompts generally improved ACC over the clean baseline (86.16%), but injecting everywhere (91.92%) slightly degraded performance compared to sparser configurations like the original (92.16%) or shallow+middle layers (92.44%). Overall, our baseline configuration ([3, 6, 9]) provides an optimal trade-off, maximizing ASR with high ACC without requiring injection at every layer.
Dataset
Metric
DeiT -Tiny
DeiT -Small
DeiT -Base
ImageNet
ACC ASR
81.36 90.72
80.96 95.80
80.76 96.48
Caltech101
ACC ASR
91.74 97.29
94.00 98.00
94.71 98.26
UCF101
ACC ASR
62.87 99.33
69.80 99.84
63.19 99.22
Table 7. Ablation study on the impact of VPG injection layers. Results show near-perfect ASR across configurations, while ACC varies slightly. Our baseline provides an optimal balance.
B. Visualization We provide visualization examples in Figure 8. We can see that our trigger is so small that there is no visual difference between the clean and backdoor images. These results further demonstrate that our attack is stealthy.
C. Theoretical Proof of Functional Fusion under Joint Optimization C.1. Preliminaries and Problem Setup Let fθ denote the frozen ViT backbone and gϕ the dynamic Visual Prompt Generator (VPG) with parameters ϕ ∈ Rp . For an input x, the VPG produces a conditional prompt ∆x = gϕ (h(x)) based on an intermediate feature h(x). We consider two tasks optimized jointly:
Layers Injected
ACC (%)
ASR (%)
[] [3, 6, 9] [1-12]
86.16 91.44 91.92
0.00 100.00 99.98
Depth Groups: Shallow Middle Deep
[1, 2, 3, 4] [5, 6, 7, 8] [9, 10, 11, 12]
92.26 91.06 90.82
99.98 100.00 99.98
Combinations: Shallow + Middle Middle + Deep Shallow + Deep (Skip)
[1-8] [5-12] [1-4, 9-12]
92.44 91.24 92.00
99.94 100.00 We assume: 99.98 (A1) Capacity constraint:
Configuration Baseline (Clean ViT) VIPER (Ours) Baseline (All Layers)
ViT backbone. VIPER’s effectiveness is validated across ViT architectures of varying scales. As shown in Table 8, from the lightweight DeiT-Tiny to the large ViT-Base, the attack maintains both exceptionally high Attack Success Rates (ASR >90%) and strong Clean Accuracy (ACC).
Lclean (ϕ) = E(x,y)∼Dclean ℓCE (Mθ,ϕ (x), y),
(10)
Lattack (ϕ, δ) = E(x,y)∼Dclean ℓCE (Mθ,ϕ (T (x, δ)), yt ), (11) where T (x, δ) injects a trigger δ into x and yt is the target label. The total loss is Ltotal (ϕ, δ) = Lclean (ϕ) + Lattack (ϕ, δ).
(12)
VPG has limited parameters (parameter-efficient fine-tuning). (A2) Shared representation: Clean and poisoned inputs share overlapping regions in the backbone feature space H. (A3) Regularization bias: Optimization implicitly or explicitly favors low-norm (simple) solutions, via weight decay or SGD bias. (A4) Joint objective: Parameters ϕ are optimized by minimizing the combined loss in Eq. (12).
(a) ImageNet
Clean
Backdoor
Trigger
Clean
(d) Food-101
Backdoor
Trigger
Trigger
Backdoor
Clean
(c) OxfordPets
Clean
(f) Ucf101
Trigger
Trigger
Backdoor
Clean
(e) Dtd
Backdoor
Trigger
Backdoor
Clean
(b) Caltech101
Figure 8. Visualization of clean images (top row of each dataset) and their corresponding backdoor images (bottom row). The trigger (right) is applied to create the backdoor images, but it is visually imperceptible, highlighting the stealthiness of our attack.
Under these conditions, we show that the minimization of Ltotal inevitably induces a Functional Fusion phenomenon: both benign and malicious functionalities are encoded in the same small subset of parameters, making them inseparable.
C.2. Linearized Toy Model To analyze the phenomenon, we locally linearize gϕ around its optimum: gϕ (hi ) ≈ A(hi ) ϕ,
(13)
where A(hi ) ∈ Rdp ×p maps the parameters ϕ to a prompt vector. For nc clean samples and na attack samples, the linearized prediction for each group is: Yc ≈ Bc + Cc ϕ,
(14)
Ya ≈ Ba + Ca ϕ,
(15)
where Cc , Ca are the respective design matrices, and Bc , Ba denote constant offsets. We define the concatenated system: Cc Y − Bc C= , Y = c . (16) Ca Ya − Ba Using a quadratic (least-squares) approximation to the loss, the total objective with regularization is: L(ϕ) = ∥Cϕ − Y ∥2 + λ∥ϕ∥2 ,
(17)
where λ > 0 captures explicit or implicit regularization.
C.3. Closed-Form Solution Theorem 1 (Closed-form Solution of Joint Optimization). The minimizer of Eq. (17) is ϕ∗ = (C ⊤ C + λIp )−1 C ⊤ Y.
(18)
Proof. Setting the gradient to zero: ∇ϕ L = 2C ⊤ (Cϕ − Y ) + 2λϕ = 0, which yields (C ⊤ C + λI)ϕ = C ⊤ Y , leading to Eq. (18).
For any unit vector v ∈ Rp , we have v ⊤ (C ⊤ C)v = ∥Cv∥2 = ∥Cc v∥2 + ∥Ca v∥2 .
(23)
If there exists a direction v such that both ∥Cc v∥ and ∥Ca v∥ are large, then this direction contributes to a high Rayleigh quotient, therefore becomes one of the dominant singular directions of C. According to Eq. (20)–(21), such directions receive large coefficients αj , causing ϕ∗ to concentrate its energy on them. Lemma 1 (Energy Concentration). If Cc and Ca have overlapping column spaces (i.e., share non-trivial common directions in parameter space), then the joint matrix C exhibits enlarged singular values along those directions, and the optimal ϕ∗ allocates most of its ℓ2 energy on them. Proof Sketch. Let Sc = span(Cc ) and Sa = span(Ca ). If Sc ∩ Sa ̸= ∅, then for any v ∈ Sc ∩ Sa , both terms in Eq. (23) are positive, yielding larger eigenvalues of C ⊤ C in those directions. From Eq. (20), larger σj implies stronger coefficients αj . Hence, ϕ∗ accumulates energy along shared directions. This means that the regularized least-squares solution compresses both tasks’ representations into the same low-dimensional subspace. This subspace corresponds to a small subset of parameter coordinates with large magnitude—empirically observed as the “core” or “functional fusion” region.
C.5. Impact of Pruning and Inseparability ⊤
Let C = U ΣV be the singular value decomposition (SVD), with Σ = diag(σ1 , . . . , σr ), r = rank(C). Substituting into Eq. (18) gives ϕ∗ = V (Σ2 + λIr )−1 ΣU ⊤ Y.
(19)
The coefficient on the j-th singular direction vj is: αj = vj⊤ ϕ∗ =
σj (u⊤ Y ). σj2 + λ j
∥ϕ ∥ =
r X j=1
σj σj2 + λ
(20)
!2 2 (u⊤ j Y) .
(21)
C.4. Energy Concentration on Shared Directions Consider the partition C = [Cc ; Ca ]. The Gram matrix of the system satisfies: C ⊤ C = Cc⊤ Cc + Ca⊤ Ca .
(CS̄⊤ CS̄ + λI)ψ ∗ = CS̄⊤ Y.
(24)
The increase in loss relative to the full optimum ϕ∗ is:
Thus, the squared norm decomposes as ∗ 2
Partition ϕ = [ϕS ; ϕS̄ ] and C = [CS CS̄ ]. If we prune parameters by enforcing ϕS = 0 and re-optimize the remaining ψ = ϕS̄ , the optimal ψ ∗ satisfies:
(22)
∆L = ([0; ψ ∗ ] − ϕ∗ )⊤ (C ⊤ C + λI)([0; ψ ∗ ] − ϕ∗ ), (25) which is strictly positive and grows with ∥ϕ∗S ∥. Thus, removing high-energy coordinates (those forming the “core” S) simultaneously harms both Lclean and Lattack , as they are jointly embedded in C and Y . Corollary 1 (Functional Inseparability). Let S denote the subset of coordinates where ϕ∗ has dominant energy. If Sc ∩ Sa ̸= ∅, then pruning or perturbing S increases both Lclean and Lattack , leading to simultaneous degradation of benign accuracy and attack success rate.
C.6. Conclusion (Functional Fusion Theorem) Theorem 2 (Functional Fusion under Joint Optimization). Under assumptions (A1)–(A4), the joint minimization of Ltotal = Lclean + Lattack with regularization inevitably yields a low-norm solution ϕ∗ whose energy is concentrated on the overlapping subspace of span(Cc ) and span(Ca ). Consequently, the clean and attack functionalities are encoded in the same small parameter subset, and any removal of this subset jointly destroys both functionalities. Proof. Immediate from Eq. (23)–(21) and the preceding lemma: joint optimization amplifies shared directions (via gradient accumulation), while the regularization λ∥ϕ∥2 penalizes redundant or disjoint bases. Hence, the optimizer prefers to reuse shared parameter directions to minimize both losses simultaneously. This coupling compresses both task mappings into a single low-dimensional core, yielding the observed functional fusion phenomenon.
C.7. Remarks • Why dynamic prompts amplify fusion: The conditional mapping gϕ (h) uses the same parameters ϕ to handle diverse inputs. Under limited capacity, optimization reuses the same parameter directions to realize both benign and malicious behaviors, enhancing fusion. • Implications for pruning defenses: Eq. (25) shows that removing shared-core parameters increases both losses quadratically, explaining the empirical collapse of clean accuracy and ASR simultaneously. • Extensions: A full-rank matrix analysis can formally prove that the minimal Frobenius-norm solution prefers shared bases between tasks, connecting this phenomenon to classical multi-task learning theory.
C.8. Summary The above analysis demonstrates, from an optimization and linear algebra perspective, that under capacity constraints and regularization, the joint minimization of Lclean + Lattack inevitably drives the model toward a functional fusion regime, where both tasks share the same compact parameter core. This explains the empirical inseparability of clean and malicious behaviors and the failure of pruningbased defenses observed in VIPER.