To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model
Chengshuai Zhao 1 Zhen Tan 1 Dawei Li 1 Zhiyuan Yu 2 Huan Liu 1
Abstract
Protected Multimodal Data
arXiv:2605.14291v1 [cs.CR] 14 May 2026
The rapid advancement of Large Vision-Language Models (LVLMs) is increasingly accompanied by unauthorized scraping and training on multimodal web data, posing severe copyright and privacy risks to data owners. Existing countermeasures, such as machine unlearning and watermarks, are inherent post-hoc approaches that act only after intellectual property infringement has already occurred. In this work, we propose MMG UARD to empower data owners to proactively protect their multimodal data against unauthorized LVLM finetuning. MMG UARD generates unlearnable examples by injecting human-imperceptible perturbations that actively exploit the learning dynamics of LVLMs. By minimizing the training loss, the perturbation creates an optimization shortcut, causing the model to overfit to the noise and thereby degrading downstream performance when the perturbation is absent during inference. To further strengthen this defense, MMG UARD introduces a cross-modal binding disruption, strategically shifting LVLM attention to enforce a spurious correlation between the noise and the training target with theoretical guarantees. Enhanced by an ensemble learning strategy for cross-model transferability, MMG UARD is evaluated against nine open-source LVLMs across six datasets. Our comprehensive results demonstrate effective, stealthy, and robust protection under white-box, gray-box, and black-box threat models, establishing a mechanistic advantage in proactively defending against aggressive fine-tuning exploitation. Our code is available at GitHub: https://github.com/Che ngshuaiZhao0/MMGuard.
Attacker ② Data Crawling
Training Dataset
① Data Protection Data Owner
Original Multimodal Data
③ Model ④ Attacker Fine-Tuning Model Inference Attacker
Poor Performance in Downstream Tasks
LVLM
Figure 1. Protect multimodal data from unauthorized fine-tuning of LVLM.
1. Introduction Large Vision-Language Models (LVLMs) (Singh et al., 2025; Comanici et al., 2025) have rapidly become a central component of modern artificial intelligence systems. By aligning visual content with natural-language instructions and responses, LVLMs support a wide range of multimodal tasks, including visual question answering (Bai et al., 2025a), image captioning (Grattafiori et al., 2024), document understanding (Zhu et al., 2025), and multimodal instruction following (Liu et al., 2023a). This success is largely fueled by an ever-growing appetite for image-text data: contemporary LVLMs are pretrained and fine-tuned on hundreds of millions of multimodal pairs scraped indiscriminately from the open web (Schuhmann et al., 2022; Radford et al., 2021), which has created an urgent threat for the original owners of multimodal content. OpenAI was reported to have transcribed and ingested more than one million hours of YouTube videos without the consent of creators or the platform to obtain multimodal training material for GPT-4, with similar practices reported at Google and Meta and a class action subsequently filed by content creators (Metz et al., 2024; Millette, 2024). The resulting harms are concrete: copyright infringement, leakage of personal and commercial information embedded in images and captions, and the loss of control over how one’s creative work is repurposed by downstream models.
1
School of Computing and Augmented Intelligence, Arizona State University, Tempe, AZ, USA 2 Department of Computer Science and Engineering, Texas A&M University, College Station, TX, USA. Correspondence to: Chengshuai Zhao <[email protected]>, Zhen Tan <[email protected]>, Dawei Li <[email protected]>, Zhiyuan Yu <[email protected]>, Huan Liu <[email protected]>.
Existing protection mechanisms are insufficient for this setting. Legal takedowns, opt-out policies, watermarking (Zhu et al., 2018), and model-output detection can help identify or respond to misuse, but they are inherently post-hoc: they act
Preprint. May 15, 2026.
1
To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model
only after the data has already been scraped or after LVLMs have already been trained. Model-level remedies such as machine unlearning (Bourtoule et al., 2021) require cooperation from the model provider and are difficult to supervise externally. Recent efforts have attempted to generate unlearnable examples (Huang et al., 2021; Fu et al., 2022; Ren et al., 2023) for unauthorized model training by injecting bounded, imperceptible perturbations. By minimizing the training loss, the perturbation creates an optimization shortcut, causing the model to overfit to the noise and thereby degrading downstream performance when the perturbation is absent during inference. Despite their promise, most methods are designed for single modality (e.g., image or audio) or specific tasks (e.g., classification, verification, or retrieval), while unauthorized LVLM fine-tuning presents a substantially different attack surface.
changed. The framework comprises five key designs: (i) It constructs an image-unlearnable perturbation via projected gradient descent (PGD) (Madry et al., 2018) over a differentiable approximation of the LVLM pipeline. (ii) It formulates text unlearnable protection through gradientguided search under discrete readability constraints, guaranteed with a smoothness-based update optimality. (iii) It introduces cross-modal binding disruption, which steers the model away from genuine semantic bindings and toward protection-specific multimodal shortcuts with theoretical guarantees. (iv) It optimizes the perturbation over an ensemble of surrogate LVLMs to improve transferability across unknown LVLM models and processing pipelines. (v) It leverages a multi-objective alternative optimization strategy to balance the effectiveness, stealthiness, and robustness of the protection.
Protecting multimodal data against LVLM fine-tuning introduces several key challenges. First, image-text samples contain redundant sources of supervision. Protecting only one modality leaves enough residual information in other modalities for the model to learn generalizable knowledge (Yao et al., 2024). Second, LVLM fine-tuning optimizes an autoregressive generative objective, in which the model captures information from each part of the input. Moreover, LVLMs have prior knowledge of image-text-response semantics from pretraining. LVLMs can still rely on the same effective associations to exploit the protected data that undermines the protection. Third, the defender usually does not know the attacker’s exact LVLM architecture, data preprocessing pipeline, and fine-tuning recipe. The protection must consider practical black-box settings. Finally, multimodal data protection is a multi-objective constrained optimization problem; each perturbation has distinct constraints and optimization dynamics, and the attacker and defender have different objectives, with the overall objective a complex balance.
We conduct comprehensive experiments across six publicly available multimodal datasets and nine open-source LVLMs. Our evaluation covers white-box, gray-box, and black-box threat models. The results show that MMG UARD consistently degrades downstream performance of LVLMs finetuned on protected data while preserving the perceptual quality of the released samples. We further evaluate robustness against aggressive LVLM attackers, including diverse data preprocessing, LVLM fine-tuning strategies, and data mixing with clean public samples. Moreover, analyses of each component, hyperparameters, and attention behavior confirm that MMG UARD provides a mechanistic advantage in protecting against unauthorized fine-tuning of LVLM.
Our key insight is that effective multimodal data protection should not merely perturb all modalities. It should also shape how LVLMs capture visual evidence, textual context, and generate target responses during fine-tuning. If the protected samples encourage the LVLMs to rely on a non-semantic spurious association between the defense signal (i.e., perturbation) and the response, then standard fine-tuning can achieve low loss on the protected training data while learning behavior that does not transfer to clean evaluation data.
• To the best of our knowledge, MMG UARD is the first data-centric protection framework that proactively defends multimodal data against unauthorized LVLM finetuning. It perturbs both the image and text as a coupled multimodal protection tailored to the LVLM autoregressive objective.
Our study makes the following contributions: • We identify and formalize unauthorized LVLM finetuning on scraped multimodal data as a data protection problem, considering a practical threat model where defenders can only modify their own public image-text data before release.
• We introduce cross-modal binding disruption, a mechanism that shifts LVLM learning dynamics toward planted perturbations and enforces spurious associations between protection and target responses. We further provide a theoretical analysis explaining why this mechanism degrades generalization on clean downstream tasks.
In this paper, we propose MMG UARD, a proactive datacenteric framework that protects multimodal image-text data from unauthorized LVLM fine-tuning. Given a clean imagetext-response sample, MMG UARD releases a perturbed image together with a text input containing a short inserted trigger, while keeping the human-facing target response un-
• We design an ensemble-based perturbation optimization strategy to improve transferability across unknown LVLM attackers, enabling effective and robust protection under white-box, gray-box, and black-box scenarios. 2
To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model
3. Background and Preliminaries
• We evaluate MMG UARD across six datasets and nine open-source LVLMs, demonstrating its effectiveness, transferability, and practicality. Further analysis confirms its robustness against adaptive attacks and provides mechanistic advantages for disrupting unauthorized LVLM finetuning.
This section presents the notation, problem formulation, and technical background used throughout the paper. We first formulate multimodal data protection as an optimization problem over image-text datasets in Sec. 3.1. We then review the standard formulation of unlearnable examples in Sec. 3.2, which serves as the foundation for our defense. Finally, we introduce the common architecture and fine-tuning paradigm of Large Vision-Language Models in Sec. 3.3, highlighting the mechanism that motivates our design.
2. Related Work Defenses Against Generative Models. Existing defenses against unauthorized generative models fall into modelcentric and data-centric approaches. Model-side methods include AI-generated content detection (Nguyen-Le et al., 2024) and watermarking for images (Zhu et al., 2018) and language outputs (Kirchenbauer et al., 2023), machine unlearning (Bourtoule et al., 2021; Liu et al., 2025), and membership inference for post-hoc auditing (Carlini et al., 2022); all are inherently reactive, acting only after data have been scraped or models trained, and typically require cooperation from the model provider. Data-side methods instead proactively modify data before release. They add perturbations to counteract a wide range of threats across modalities such as style mimicry (Shan et al., 2023; 2024), voice cloning (Yu et al., 2023), and personalized fine-tuning (Van Le et al., 2023; Liu et al., 2024c). While these methods raise the cost of misuse, they often degrade perceptual quality or remain fragile under purification and adversarial training (Foerster et al., 2025; Wang et al., 2025b).
3.1. Problem Formulation Let X denote the image space and T denote naturallanguage space. A multimodal dataset is given by D = {(xi , ti , yi )}ni=1 ,
(1)
where xi ∈ X is an image, ti ∈ T is a textual input (e.g., question or instruction), and yi ∈ Y is a target output (e.g., answer or response). A large vision language model is denoted by fθ : X × T → Y,
(2)
where θ = (θfrz , θtr ) denotes the model parameters, consisting of frozen pretrained parameters θfrz from the base LVLM checkpoint and trainable parameters θtr that the attacker may select for fine-tuning. Given a multimodal training set D, an unauthorized attacker seeks to obtain:
Unlearnable Example Methods. Unlearnable examples (UEs) offer a proactive, data-centric alternative by injecting bounded perturbations that act as “shortcuts,” causing models trained on protected data to fail on clean test data. Pioneered for image classification via error-minimizing noise (Huang et al., 2021), UEs have been refined for robustness against adversarial training (Fu et al., 2022; Liu et al., 2024d), transferability across architectures (Ren et al., 2023; Li et al., 2025b), and label-agnostic settings via CLIP surrogates (Zhang et al., 2023a), and broadened to text (Li & Liu, 2023), graphs (Liu et al., 2023b), audio/speech (Gokul & Dubnov, 2024; Zhang et al., 2023b), image segmentation (Sun et al., 2024), and diffusion-based generation (Zhao et al., 2023; Li et al., 2025a). For multimodal contrastive pretraining, MEM (Liu et al., 2024b) extends error-minimizing noise to image-caption pairs to mislead CLIP-style models. Recent studies caution that UEs can be partially circumvented through relearning (Dang et al., 2023), pretrained backbones (Li et al., 2026), or diffusion-based purification (Wang et al., 2025b), motivating the need for stronger designs. Despite this progress, no prior work targets unauthorized fine-tuning of LVLMs, which differs fundamentally in objective, input space, and learning dynamics. MMG UARD fills this gap with the first proactive, data-centric protection tailored to LVLM fine-tuning.
θ⋆ (D) ∈ arg min Ltrain (θ; D), θ∈Θ
(3)
where Ltrain is the empirical training objective. For simplicity, we use ’fine-tune’ and ’train’ interchangeably in the paper. The goal of multimodal data protection is to construct a protected dataset eϕ = Gϕ (D) = {(e D xi , e ti , yi )}ni=1 ,
(4)
where Gϕ is a protection map parameterized by ϕ. The protection must remain close to each original sample under modality-specific perceptual and semantic budgets: n o eϕ : dx (e B(D) = D xi , xi ) < ϵx , dt (e ti , ti ) < ϵt , ∀i ∈ [n] , (5) where dx and dt measure visual and textual change, respectively, with budgets ϵx and ϵt . e perThe defender aims to ensure that models trained on D form poorly on clean downstream data. Let Deval be a clean downstream evaluation dataset, and let Leval measure the 3
To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model
encoded into visual features Hix by a visual encoder Ex :
corresponding evaluation loss. We formalize the defender’s objective as the following bilevel optimization problem: ⋆
max
eϕ ); Deval Leval θ (D
s.t.
eϕ = Gϕ (D), D eϕ ∈ B(D), D eϕ ) ∈ arg min Ltrain (θ; D eϕ ). θ⋆ (D
ϕ∈Φ
Hix = Ex (π(gx (xi ))),
The visual features are then aligned with the language-model hidden space through a modality projector Px :
(6)
Zix = Px (Hix ).
θ∈Θ
Zit = Et (Tok(ti ))
3.2. Unlearnable Examples
Si = Template(Zix , Zit ).
(11)
which will be processed by the downstream language model. During supervised fine-tuning, LVLM is optimized by minimizing the autoregressive negative log-likelihood of the target response conditioned on the image and textual input: |y |
n i 1 XX − log pθ yi,j | yi,<j , xi , ti . Ltrain (θ; D) = n i=1 j=1
(12)
The standard unlearnable-example objective is often written as a bilevel min-min problem. Let δ = {δi }ni=1 denote bounded perturbations applied to the original data before release, and let Dδ = Gδ (D) be the protected dataset. For instance, Dδ = {(xi + δi , ti , yi )}ni=1 in the common imageonly case. The defender constructs δ by solving: min Ltrain θ⋆ (Dδ ); Dδ
4. Threat Model We consider two parties: an attacker, who collects multimodal web data to fine-tune LVLMs, and a defender, who owns image-text data and wishes to publish it online while preventing its unauthorized use as effective fine-tuning data. Attacker Objectives. The attacker starts from an existing pretrained LVLM checkpoint and fine-tunes it on multimodal data scraped from the web. Their objective is to maximize the model utility of downstream task performance while disregarding potential copyright infringement and privacy risks.
δ
Dδ ∈ B(D),
(10)
The visual and textual tokens are then interleaved into a joint multimodal sequence Si according to a predefined template:
Unlearnable examples are training samples designed to remain semantically useful to human users while inhibiting unauthorized models from learning meaningful representations from them (Huang et al., 2021). In contrast to conventional adversarial examples, which perturb inputs to cause mistakes at inference time, unlearnable examples intervene prior to the training stage by modifying the data. Their core mechanism is to introduce a subtle, hard-to-detect perturbation that induces a spurious shortcut the model can exploit to reduce the training loss, thereby discouraging it from learning the genuine input-output relationship. Consequently, models trained on such data fit the protected samples but generalize poorly to clean inputs.
Dδ = Gδ (D),
(9)
In parallel, the textual input is tokenized by Tok and mapped to embeddings by Et .
Equivalently, the defender seeks a small, humanimperceptible transformation of the dataset that degrades the utility of models fine-tuned by an unauthorized trainer.
s.t.
(8)
(7)
⋆
θ (Dδ ) ∈ arg min Ltrain (θ; Dδ ). θ∈Θ
The inner minimization describes the attacker’s training process: given the released protected data, the attacker optimizes the model parameters to fit that data. The outer minimization describes how the defender chooses the perturbation to ensure that the training process overfits the protected distribution.
Attacker Capabilities. The attacker possesses the knowledge and expertise required to optimize LVLMs using a broad range of fine-tuning techniques and to evaluate their performance. In our study, we consider two types of attackers: (i) a naive attacker who follows standard fine-tuning pipelines, such as LoRA, and (ii) an adaptive attacker who is aware of MMG UARD and seeks to circumvent its protection. The adaptive attacker may employ diverse strategies, including data transformations and data mixing. We discuss these strategies and our robustness evaluation in detail in Sec. 6.5.
3.3. Large Vision-Language Models Modern LVLMs extend pretrained language models with a visual front end that converts images into tokens compatible with the language-model embedding space. Given an image-text input (xi , ti ), the image is first standardized (e.g., resizing and pixel quantization) by an image processor gx , split into image patches by a patching function π, and
Defense Objectives. The defender may be an individual artist, photographer, news outlet, private dataset curator, or 4
To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model
any data owner who seeks to prevent their content from being effectively exploited by LVLMs. The defender has two primary objectives. First, protected samples should disrupt unauthorized fine-tuning: when incorporated into the attacker’s training set, they should degrade the model’s downstream performance. Second, the protected content should preserve high perceptual and semantic fidelity, ensuring that it remains useful for legitimate online publication and human consumption.
perturbations, and projected gradient descent optimization that updates perturbations to protect the image. Differentiable Image Processor. As described in the Sec. 3.3, the image processor gx contains non-differentiable operations that create an obstacle for directly optimizing the image perturbation δi in the raw pixel space. This obstacle undermines the defender’s ability to create effective, stealthy, unlearnable examples. To address it, we construct a differentiable surrogate processor g̃x that approximates the practical processor gx : g̃x ≈ gx . Specifically, we handle the non-differentiable functions by three strategies. (i) For discretization steps (e.g., pixel quantization), we keep the forward value used by the practical processor but copy gradients through a continuous proxy. (ii) For geometric transformations (e.g., resizing), we use differentiable substitutes that approximate the same transformation during backpropagation. (iii) For layout decisions (e.g., resolution selection), we follow the same deterministic rules as gx and treat the resulting layout as fixed metadata for the current image. Operations that are already differentiable are kept unchanged. Together, these choices give the defender a differentiable path from the LVLM loss back to pixel-space perturbations while keeping the forward representation close to that in the surrogate model.
Defense Assumptions. The defender has full access to their own image-text pairs before publication and can modify them by adding imperceptible protective perturbations. However, the defender has no access to the attacker’s full training corpus, fine-tuning recipe, hyperparameters, or model weights, and cannot directly interfere with the attacker’s training process. We consider three levels of defender knowledge about the attacker’s target model. In the white-box setting, the defender knows the exact LVLM that the attacker will fine-tune and optimizes the protection directly against it. This setting provides an upper bound on defense effectiveness, although it is rarely realistic in practice. In the gray-box setting, the defender knows only partial information, such as the model family or architecture, but not the exact checkpoint. In the black-box setting, the defender has no concrete knowledge of the attacker’s model and must rely on architecture-agnostic protection and crossfamily transferability. This setting is the most realistic and challenging scenario.
Projected Gradient Descent Optimization. For each clean image xi , we construct the protected image x̃i by adding a human-imperceptible perturbation δi : x̃i = xi + δi , where δi is constrained to lie in the image-side feasible set Ciδ :
5. Our Approach MMG UARD
Ciδ = {δ : xi + δ ∈ X , ∥δ∥∞ ≤ ϵx }.
This section presents MMG UARD, our proposed framework for proactive multimodal data protection against unauthorized LVLM fine-tuning. We first construct a continuous image perturbation in Sec. 5.1 and a discrete textual trigger in Sec. 5.2 to form a coupled multimodal protection that eliminates residual learnable signal in either modality. We then introduce cross-modal binding disruption in Sec. 5.3 to redirect LVLM learning toward protection-specific shortcuts, preventing the autoregressive objective from bypassing the protection through pretrained knowledge. We further optimize the protection over an ensemble of surrogate LVLMs in Sec. 5.4 to improve the transferability of protection to black box scenarios. We finally cast the framework as a constrained multi-objective bilevel optimization in Sec. 5.6 to reconcile heterogeneous continuous and discrete variables under modality-specific budgets and the asymmetric objectives of defender and attacker.
(13)
Formally, the unlearnable image protection solves the following bilevel optimization problem: n
min {δi }
1X ℓprot f˜θ̃⋆ ; x̃i , ti , yi x n i=1 δi ∈ Ciδ , n X θ̃x⋆ ∈ arg min ℓtrain f˜θ̃ ; x̃i , ti , yi .
s.t. x̃i = xi + δi , θ̃
(14)
i=1
where ℓprot is the defender’s protection objective (e.g., the LVLM training loss), later defined in Eq. (30), and ℓtrain denotes the sample-wise LVLM training loss. We optimize δi by projected gradient descent: δi ← ΠCiδ δi − αx sign ∇δi ℓprot
.
(15)
5.1. Unlearnable Image Protection where ΠCiδ denotes projection back onto the per-image feasible set, and αx is the step size. The gradient is taken through the differentiable processor g̃x , the visual encoder Ex , the projector Px , and the language model.
We formulate the unlearnable image protection as two key components: a differentiable image processor that allows gradient flow from the surrogate LVLM back to pixel-space 5
To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model Multimodal Data
Large Vision-Language Model Architecture
Image
Image Processing Image Processor
Vision Encoder
Image-Text Projector
Image Tokens
Template
Language Model
<IMG>
Add & Norm
... Ni x d
Text Tokenizer
Question: What is the color of the bridge?
...
Decoded Token The
Multi-Head Self-Attention </TXT>
Text Protection
answer
is
Attacker Objective (Inner Loop)
red
Final Answer: Red
Cross Modal Bind Disruption Bridge Path Hijack
Perturbation
Defender Objective (Outer Loop)
Add & Norm
...
Nt x d
Original Text
+
Next Token Distribution
Text Tokens
...
Image Protection Original Image
Text Encoder
Optimization
Feed-Forward Network
...
Text Processing
Text
Answer Generation
Source Attention
Legend
Contrastive Routing Shift Low Attention
Target Attention
Protected Attention
Clean Attention
Low Attention
What is the color of the bridge?
Image Perturbation Token Token
Trigger to Perturbation
Text Token
Protected Text
Protected Image What
is
the
<γ1>
color
of
the
<γ2>
bridge
?
High Attention
Answer to Answer Perturbation to Trigger
High Attention
Trigger Token
Answer Padding Token Token Attention Attention Aligement Contrast
Figure 2. Overview of MMG UARD. The defender generates protected multimodal examples by coupling image-unlearnable perturbations with text-unlearnable triggers and using cross-modal binding disruption to steer LVLM attention toward protection-specific shortcuts that fail to transfer to the downstream task.
the embedding of token v by ev = Et (v). For a trigger [j=v] position j, let γi denote the trigger obtained by setting that inserted position to candidate token v. A first-order expansion of the protection loss gives
5.2. Unlearnable Text Protection Given a clean text input ti , we construct the protected text t̃i by inserting a bounded-length trigger γi = (γi,1 , . . . , γi,|γi | ): t̃i = Insert(ti , γi ), where insertion is restricted to textual input positions and does not replace tokens in ti . Each trigger token is selected from an admissible vocabulary Vadm ⊆ V (e.g., excluding special tokens, control tokens, and non-linguistic tokens). The trigger length is bounded by the text budget length ϵt , so the text-side feasible set for γi is: Ciγ = {γ : |γ| ≤ ϵt , γj ∈ Vadm }.
[j=v]
ℓprot γi
si,j (v) = ev − eγi,j
(16)
s.t.
1X ℓprot f˜θ̃⋆ ; xi , t̃i , yi t n i=1 t̃i = Insert(ti , γi ), γi ∈ Ciγ , n X ⋆ θ̃t ∈ arg min ℓtrain f˜θ̃ ; xi , t̃i , yi . θ̃
∇eγi,j ℓprot . (18)
⊤
∇eγi,j ℓprot .
(19)
Candidate Verification. The token with the best first-order score may still be suboptimal after deployment because the HotFlip score is computed in the token-embedding space, whereas the released trigger is ultimately a surface string processed by the LVLM tokenizer. Under byte-pair encoding (BPE) tokenization, decoding a candidate token and inserting it into the text can change neighboring token boundaries through merge or split operations, so the actual re-tokenized sequence may differ from the assumed single-position substitution. To account for this tokenization mismatch, we verify the shortlisted candidates and computing the exact protection loss:
n
{γi }
⊤
The corresponding linear score is
Analogous to image-side protection, the unlearnable text protection solves a discrete min-min problem: min
≈ ℓprot (γi )+ ev −eγi,j
(17)
i=1
Thus, insertion defines where the protected text differs from the clean input, while the discrete optimization below changes only the identities of tokens inside the inserted trigger.
[j=v]
⋆ vi,j = arg min ℓprot γi cand v∈Vi,j
Gradient-Based Candidate Insertion. We approximate the outer update in Eq. (17) with a HotFlip-style (Ebrahimi et al., 2018) first-order search over the inserted trigger tokens. Let Et map vocabulary tokens to embeddings and denote
.
(20)
This screen-and-verify procedure is provably near-optimal: under standard smoothness, the selected token matches the best admissible substitution up to a small quadratic remainder, which is justified by Lemma B.1 in Appendix B. 6
To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model
5.3. Cross-Modal Binding Disruption
This distribution summarizes how much attention mass the queries in S collectively assign to every key token. For a nonempty target token set R ⊆ Ω, define the uniform distribution on R, embedded in the simplex over Ω, as ( 1, b ∈ R, 1 |Ω|−1 UR = 1R ∈ ∆ , 1R (b) = |R| 0, b ∈ / R. (25) That is, UR assigns equal probability to tokens in the target set and zero probability to all other tokens in the full sequence. We then measure how far the source attention mass is from this reference distribution, averaged over a chosen layer set K:
Image and text unlearnable protections discourage the LVLM from learning robust features from either modality, but they do not guarantee a genuine unlearnable shortcut for fine-tuning. In LVLMs, answer tokens are generated by decoder attention over the full multimodal context. Even when both the image and the text are protected, this attention mechanism can still route generation through semantically meaningful evidence that remains available in either modality. Moreover, LVLMs have prior knowledge of image-textresponse semantics from pretraining. LVLMs can rely on the effective associations to exploit the protected data that undermines the protection. Therefore, potent protection should not be limited to surface-level perturbation for both modalities; it also shapes how LVLMs capture visual evidence and textual context, and how they generate target responses during fine-tuning. To enable this, MMG UARD introduces cross-modal binding disruption. By disrupting the normal semantic binding and steering optimization toward protection-specific spurious attention paths, the method encourages fine-tuning to rely on non-transferable perturbation shortcuts.
ℓmass (S, R) =
Equivalently, ℓmass is a layer-averaged contrastive loss that raises the attention logits of tokens in R and suppresses those of other tokens, as justified by Proposition C.1 in Appendix C. Variant 1: Bridge Path Hijack (BPH). Based on the attention-mass loss, we can design various cross-modal binding objectives that create different protection-specific attention shortcuts. Specifically, we consider three distinct paths:
(21) • Trigger-to-perturbation binding ℓmass (Ωγ , Ωδ ): it ties the text trigger to the perturbation, so that the image and text protections are coupled across modalities.
where Ωx,i , Ωt,i , Ωγ,i , and Ωy,i denote the image tokens, original text tokens, inserted trigger tokens, and answer tokens, respectively. For simplicity, we omit template auxiliary tokens.
• Answer-to-trigger shortcut ℓmass (Ωy , Ωγ ): it creates a direct attention path from answer tokens to the inserted trigger, so that the answer loss can be reduced by attending to the trigger rather than to the original semantic evidence.
Among the image tokens, we further identify the subset most affected by the perturbation. Let Pi,b denote the preprocessed pixel region that corresponds to image token b ∈ Ωx,i , and define its average perturbation magnitude as X 1 ρi,b = |δi (u)|. (22) |Pi,b |
• Answer-to-perturbation shortcut ℓmass (Ωy , Ωδ ): it preserves a spurious attention shortcut from answer tokens to perturbation that degrades clean downstream generalization.
u∈Pi,b
We select perturbation tokens Ωδ,i that are most affected in the image tokens based on a ratio τδ ∈ (0, 1]: Ωδ,i = b ∈ Ωx,i : rankΩx,i (ρi,b ) ≤ ⌈τδ |Ωx,i |⌉ , (23)
We combine these three paths to form the BPH loss:
ℓBPH bind = β1 ℓmass (Ωγ , Ωδ ) + β2 ℓmass (Ωy , Ωγ )
where rankΩx,i (·) orders perturbation scores in descending order, with ties broken deterministically. (k,h)
+ β3 ℓmass (Ωy , Ωδ ),
(27)
|Ω|×|Ω|
Let A ∈ [0, 1] denote the attention matrix of head h in layer k, where h ∈ H and each row is a distribution over key tokens. For a source token set S ⊆ Ω, we define its head-averaged attention-mass distribution as Āk (S) =
(26)
k∈K
Attention Mass Distribution. For a protected sample (x̃i , t̃i , yi ), the LVLM template assembles a joint token sequence in the language model backbone: Ωi = Ωx,i ⊎ Ωt,i ⊎ Ωγ,i ⊎ Ωy,i ,
1 X DKL UR ∥ Āk (S) . |K|
XX 1 A(k,h) ∈ ∆|Ω|−1 . a,: |H| |S|
where β1 , β2 , β3 ≥ 0 are hyperparameters that control the relative importance of the three paths. Theorem 5.1 (Effectiveness of BPH). Let Ri ∈ {Ωγ,i , Ωδ,i } be a protection-induced target set that reduces to the empty set under the clean evaluation input (xi , ti , yi ). Suppose a model θ⋆ achieves answer-token
(24)
h∈H a∈S
7
To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model
binding ℓmass (Ωy,i , Ri ) ≤ η on the protected sample (x̃i , t̃i , yi ). Denote by Āpk,i (Ωy ) and Āck,i (Ωy ) the headaveraged answer-token attention-mass distributions of θ⋆ on the protected and clean inputs at layer k, both extended to a common token universe by zero-padding the positions in Ri on the clean side. Then p 1 X TV Āpk,i (Ωy ), Āck,i (Ωy ) ≥ 1 − η/2. (28) |K|
5.4. Ensemble Protection for Black-box Robustness The joint objective in Eq. (30) optimizes protection with respect to a single surrogate LVLM, which may provide limited robustness against unseen attackers with different model architectures. To improve black-box transferability, we extend the protection objective to an ensemble of surrogate models:
k∈K
The proof is given in Appendix D. Theorem 5.1 shows that every protection-induced binding term in Eq. (27) that is driven to a p small η enforces an attention-pattern shift of at least 1 − η/2 in mean totalvariation distance between protected training and clean evaluation, providing the mechanistic effectiveness by which BPH’s protection signal fails to transfer to clean downstream data. The complementary trigger-to-perturbation term ensures that the trigger and perturbation pathways are coupled: severing either at evaluation collapses the joint route the surrogate adopted during fine-tuning, thus strengthening protection.
ℓprot =
M X m=1
(m)
M X
(m) ωm ℓjoint ,
ωm = 1,
(31)
m=1
(m)
where ℓtrain , ℓbind are the training loss and the binding disruption loss computed on surrogate m, and ωm ≥ 0 is the weight for surrogate m. The ensemble objective encourages protected data to remain unlearnable across multiple surrogates, thereby providing more generalizable protection. 5.5. Adversarial Objective as an Unlearnable Variant
Variant 2: Contrastive Routing Shift (CRS). A fixed bridge is effective when the surrogate and the attacker share a similar fusion mechanism, but its prescriptive form may over-constrain the path and limit transfer to unseen architectures. CRS relaxes the prescription: instead of dictating where the answer should attend, it only requires that the answer’s attention-mass pattern differs between the clean and protected forward passes. Let Āck (Ωy ) and Āpk (Ωy ) denote the answer-token attention-mass distributions obtained under the clean and protected inputs, respectively. CRS minimizes 1 X ℓCRS DKL sg(Āck (Ωy )) ∥ Āpk (Ωy ) , (29) bind = − |K|
Optimizing the protection loss in Eq. (31) encourages unlearnable protection through the standard min-min objective. The same framework also supports an adversarial protection objective ℓAD prot by reversing the outer training-loss term while preserving the binding-disruption term:
ℓAD prot =
M X
AD(m)
ωm ℓjoint ,
m=1 AD(m) (m) (m) ℓjoint = −λtrain ℓtrain + λbind ℓbind .
(32)
k∈K
This adversarial variant targets training disruption by making protected samples hard to fit, thus directly degrading fine-tuning performance rather than relying on shortcut memorization. The binding term is preserved with the same sign because it shifts attention routing patterns that are shared across LVLMs to amplify the protection effect.
where the stop-gradient sg(·) freezes the clean reference so that updates only reshape the protected-side route. CRS does not assume a specific target that the surrogate must attend to, thereby creating another effective protection variant. Joint Protection Objective. For each cross-modal binding disruption loss, we combine it with the standard training loss to form a joint protection objective: ℓjoint = λtrain ℓtrain + λbind ℓbind .
5.6. Constrained Multi-objective Optimization Given a clean multimodal sample (xi , ti , yi ), the unlearnable image and text protections create two specialized protection signals, and the cross-modal binding disruption creates a complementary attention-level mechanism guidance. We combine them into a single constrained bilevel optimization over the released dataset. Formally, the protected input is
(30)
The first term retains the standard min-min unlearnableexample signal: the protected sample must remain easy to fit during unauthorized fine-tuning. The second term governs how that fitting is achieved, biasing the surrogate toward attention routes that exist only when the trigger is inserted and the image perturbation is present. The two terms are complementary, thus forming effective protection against unauthorized LVLMs.
x̃i = xi + δi , 8
t̃i = Insert(ti , γi ),
(33)
To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model
Algorithm 1 MMG UARD: Multimodal Data Protection via Constrained Multi-Objective Optimization.
small number of adaptation steps for all surrogate models. The outer loss is then evaluated on the adapted surrogate. Image perturbations are updated by projected gradient descent as in Eq. (15), while trigger tokens are updated by the HotFlip screening and verification procedure in Eqs. (19)(20). These two updates share the same joint loss, so the image perturbation and text trigger are optimized to support the same protection purpose. The algorithm preserves feasibility after every update: projection clips image perturbations to the ℓ∞ budget and the valid pixel range, while text updates only select admissible tokens within the fixed trigger-length budget. This algorithm balances overall feasibility with the effectiveness, stealthiness, and robustness of the protection.
Require: Clean data D = {(xi , ti , yi )}ni=1 ; surrogate LVLMs {f˜(m) }M m=1 ; budgets (ϵx , ϵt ); outer rounds R; inner steps Q. e = {(x̃i , t̃i , yi )}n . Ensure: Protected dataset D i=1 γ 1: Initialize feasible δi ∈ Ciδ and γi ∈ Ci for all i 2: for r = 1 to R do 3: Form x̃i = xi + δi and t̃i = Insert(ti , γi ) 4: for each surrogate m = 1, . . . , M do Approximate 5: θ̃(m)⋆ by Q gradient steps on P ˜ i ℓtrain fθ̃ (m) ; x̃i , t̃i , yi 6: end for 7: Compute the ensemble protection loss in Eq. (34) 8: Update each δi by PGD and project onto Ciδ 9: Update each γi by gradient-guided candidate screening and exact candidate verification over Vadm 10: end for e = x̃i , t̃i , yi n 11: return D i=1 with δi ∈ Ciδ and γi ∈ Ciγ . {f˜(m) }M , MMG UARD solves
6. Experiments We empirically validate MMG UARD along six axes. We first detail the datasets, target LVLMs, attack scenarios, and evaluation metrics in Sec. 6.1. We then quantify protection effectiveness under white-box and gray-box attackers in Sec. 6.2, and assess transferability to black-box LVLMs and diverse fine-tuning techniques in Sec. 6.3. We further evaluate robustness against adaptive attackers that apply adaptive attacks by data transformations and data mixing in Sec. 6.5, and measure practicality in terms of stealthiness and computational cost in Sec. 6.4. We finally analyze the protection mechanism in Sec. 6.6 through ablations, parameter sensitivity, and attention-level visualization.
Given a surrogate set
m=1
n
min
{δi ,γi }
s.t.
M
1XX (m) ωm ℓjoint f˜θ̃(m)⋆ ; x̃i , t̃i , yi n i=1 m=1 δi ∈ Ciδ ,
γi ∈ Ciγ ,
∀i,
M X
ωm = 1,
m=1
θ̃(m)⋆ ∈ arg min θ̃ (m)
n X
ℓtrain f˜θ̃(m) ; x̃i , t̃i , yi ,
6.1. Experimental Setup ∀m.
Datasets and Tasks. We evaluate MMG UARD across six public multimodal question-answering benchmarks spanning general visual reasoning, domain-specific reasoning, text-centric perception, and document understanding. RealWorldQA (xAI, 2024) tests understanding of everyday physical scenes, including spatial relations, object affordances, and commonsense visual cues. MMStar (Chen et al., 2024) evaluates fine-grained multimodal reasoning across six capability categories and eighteen task axes. ScienceQA (Lu et al., 2022) covers curriculum-grounded science reasoning that combines visual evidence with textual context. VQARAD (Lau et al., 2018) evaluates radiology-oriented medical VQA over clinical images. TextVQA (Singh et al., 2019) measures scene-text reading and reasoning in natural images, while DocVQA (Mathew et al., 2021) targets document image understanding over structured and semi-structured text. Table 1 summarizes the dataset statistics.
i=1
(34) (m)
Here ℓjoint is instantiated by Eq. (30) for the min-min unlearnable objective, or by Eq. (32) for the adversarial variant. This formulation is multi-objective in two senses. First, the defender optimizes both the task-level fitting signal and the attention-level binding signal through λtrain and λbind . Second, the ensemble weights ωm approximate transfer to possible attackers by optimizing the same released perturbations against multiple surrogates. The modality budgets remain hard constraints rather than soft penalties, ensuring that the optimized data stays within the protection set in Eq. (13) and Eq. (16). Optimization Algorithm. Directly solving Eq. (34) is impractical because it would require repeatedly solving the attacker’s inner fine-tuning problem to convergence, while jointly optimizing continuous image perturbations and discrete text triggers under different feasibility constraints. We therefore use an alternating approximation summarized in Algorithm 1. At each outer round, the current protected samples are used to approximate the inner solution by a
Large Vision-Language Models and Training Configuration. We consider and evalute state-of-the-art open-source LVLMs with various architectures and parameter scales including Qwen3-VL-2B-Instruct, Qwen3-VL-4B-Instruct, and Qwen3-VL-8B-Instruct (Bai et al., 2025a); Qwen2.59
To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model Table 1. Statistics of the evaluation datasets.
General General Science Medical Scene text Document
Task Visual Understanding Multimodal Reasoning Science Reasoning Medical VQA OCR Reasoning Document Understanding
Accuracy
RealWorldQA MMStar ScienceQA VQA-RAD TextVQA DocVQA
Domain
Size 765 1,500 21,208 2,244 45,336 50,000
65 60 55 50
Q3-2B Q3-4B Q3-8B Q2.5-7B
VQA-RAD
60
Accuracy
Dataset
RealworldQA
70
80 73 66 59 52 45
MMStar
96 92 88 Q3-2B Q3-4B Q3-8B Q2.5-7B
TextVQA
95
72
52
75
64
48
65 Q3-2B Q3-4B Q3-8B Q2.5-7B
55
Q3-2B Q3-4B Q3-8B Q2.5-7B
DocVQA
80
85
56 Q3-2B Q3-4B Q3-8B Q2.5-7B
Zero-Shot (Clean) Fine-Tuning (Clean)
VL-7B-Instruct (Bai et al., 2025b); Llama-3.2-11B-VisionInstruct (Grattafiori et al., 2024); LLaVA-v1.5-7B (Liu et al., 2024a); InternVL3.5-8B (Wang et al., 2025a); Gemma-34B-IT (Gemma Team et al., 2025); and GLM-4.1V-9BBase (Hong et al., 2025). We consider three attack scenarios with different levels of surrogate knowledge: (i) white-box: we use Qwen3-VL-4B-Instruct as the surrogate model and evaluate it as the white-box attacker; (ii) gray-box: we use Qwen3-VL-4B-Instruct as the surrogate and evaluate the same Qwen-family variants with different scales; and (iii) black-box: we leverage Qwen3-VL-4B-Instruct (Bai et al., 2025a) and MiniCPM-V-4 (Yao et al., 2025) as surrogate models to generate the protected data and evaluate the other six LVLMs as black-box attackers.
84
56
44
ScienceQA
100
MMGuard-BPH MMGuard-CRS
48
Q3-2B Q3-4B Q3-8B Q2.5-7B
MMGuard-BPH-Max MMGuard-CRS-Max
Figure 3. Protection effectiveness across MMG UARD variants under white-box and gray-box scenarios. Lower bars indicate stronger protection. RealworldQA Loss
10
10
0
10
−1
10
−1
10
−2
10
−2
10
−3
10
−3
20
10
Loss
MMStar
0
−1
10
−2
10
−3
60
80
ScienceQA
0
10
40
20
40
60
80
100
50 10
1
10
0
10
−1
10
−2
Training Step Fine-Tuning (Clean)
Evaluation Metrics. Our primary utility metric is task accuracy on the clean held-out test set after unauthorized fine-tuning. To quantify protection effectiveness, we further report the accuracy drop relative to clean fine-tuning: ∆acc = Acc fθclean ; Dtest − Acc fθprot ; Dtest , (35)
100
150
DocVQA
20
40
60
80
100
Training Step MMGuard-BPH MMGuard-BPH-Max
MMGuard-CRS MMGuard-CRS-Max
Figure 4. Attacker training loss on protected data (log scale). Minmin variants converge to lower loss than C LEAN FT, indicating shortcut fitting, while M AX variants plateau at higher loss, indicating training disruption.
Larger ∆acc indicates stronger protection, while clean finetuning accuracy serves as the task-specific upper reference. We provide complete dataset descriptions, fine-tuning and protection-optimization configurations, and the per-dataset evaluation protocol in Appendices E and F.
TextVQA, DocVQA, MMStar, and RealworldQA, where performance depends heavily on image-text alignment. The protection also transfers to gray-box target models. Although the reduction varies with model similarity and task type, both MMG UARD-BPH and MMG UARD-CRS remain below the Clean Fine-Tuning reference in nearly all settings, indicating that the shortcut does not overfit to a single checkpoint. The effect is milder on ScienceQA and VQARAD, where language priors and domain knowledge can partially compensate for disrupted multimodal learning. The M AX variants further strengthen protection in several cases, suggesting improved cross-model transfer. Overall, MMG UARD is effective under both white-box and more realistic gray-box scenarios.
6.2. Effectiveness Evaluation We assess whether MMG UARD produces a reliable protection signal when the attacker fine-tunes on the released data. We examine two aspects: (i) effectiveness under white-box and gray-box scenarios; and (ii) attacker training dynamics. Protection Effectiveness Under White-Box and GrayBox Scenarios. Fig. 3 shows that MMG UARD consistently lowers post-fine-tuning accuracy compared with the Clean Fine-Tuning reference across the evaluated LVLMs and datasets, confirming that protected samples induce a reliable degradation signal during unauthorized training. The effect is most pronounced in the white-box setting on the surrogate Qwen3-VL-4B, where protected models often approach or fall below the Zero-Shot reference, largely eliminating the utility gain of clean fine-tuning. The degradation is especially clear on visually grounded benchmarks such as
Training Dynamics on Protected Data. Fig. 4 further explains the accuracy degradation by showing how protected data changes the attacker’s optimization trajectory. The min-min variants, MMG UARD-BPH and MMG UARD-CRS, consistently drive the training loss below Clean Fine-Tuning, often by one to two orders of magnitude. This indicates that the attacker can fit the protected training set more easily than the clean one by exploiting a perturbation-induced shortcut. 10
To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model Table 2. Protection transferability under the black-box scenario. Each cell reports clean-test accuracy with the drop in parentheses. RealWorldQA Method
ScienceQA
TextVQA
Llama
LLaVA
InternVL
Gemma
GLM
Llama
LLaVA
InternVL
Gemma
GLM
Llama
LLaVA
InternVL
Gemma
GLM
Zero-Shot (Clean) Fine-Tuning (Clean)
49.7 (+6.9) 56.6 (0.0)
53.6 (+3.9) 57.5 (0.0)
54.9 (+5.9) 60.8 (0.0)
36.6 (+19.6) 56.2 (0.0)
27.5 (+39.8) 67.3 (0.0)
84.5 (+11.0) 95.5 (0.0)
63.0 (+16.5) 79.5 (0.0)
97.5 (+1.5) 99.0 (0.0)
76.5 (+8.0) 84.5 (0.0)
88.5 (+9.5) 98.0 (0.0)
14.5 (+67.5) 82.0 (0.0)
51.5 (+4.0) 55.5 (0.0)
75.0 (+1.5) 76.5 (0.0)
54.5 (+28.5) 83.0 (0.0)
0.0 (+85.0) 85.0 (0.0)
Image Text Multimodal
61.4 (-4.8) 57.5 (-0.9) 56.9 (-0.3)
55.6 (+1.9) 55.6 (+1.9) 56.2 (+1.3)
60.8 (0.0) 60.1 (+0.7) 58.8 (+2.0)
51.6 (+4.6) 52.9 (+3.3) 51.6 (+4.6)
62.7 (+4.6) 60.1 (+7.2) 62.1 (+5.2)
93.5 (+2.0) 95.0 (+0.5) 93.5 (+2.0)
77.5 (+2.0) 76.5 (+3.0) 75.5 (+4.0)
98.5 (+0.5) 99.0 (0.0) 98.0 (+1.0)
82.0 (+2.5) 80.5 (+4.0) 82.0 (+2.5)
99.0 (-1.0) 98.5 (-0.5) 97.5 (+0.5)
77.5 (+4.5) 80.5 (+1.5) 79.5 (+2.5)
52.0 (+3.5) 53.0 (+2.5) 52.0 (+3.5)
76.0 (+0.5) 74.5 (+2.0) 76.5 (0.0)
79.0 (+4.0) 81.5 (+1.5) 80.0 (+3.0)
83.5 (+1.5) 83.5 (+1.5) 82.5 (+2.5)
MMG UARD-BPH MMG UARD-CRS MMG UARD-BPH-Max MMG UARD-CRS-Max
52.9 (+3.7) 51.7 (+4.9) 54.4 (+2.2) 53.6 (+3.0)
55.6 (+1.9) 54.4 (+3.1) 56.0 (+1.5) 53.4 (+4.1)
54.9 (+5.9) 56.2 (+4.6) 55.4 (+5.4) 57.3 (+3.5)
50.3 (+5.9) 51.0 (+5.2) 49.5 (+6.7) 51.4 (+4.8)
60.8 (+6.5) 61.4 (+5.9) 62.6 (+4.7) 63.2 (+4.1)
90.0 (+5.5) 92.0 (+3.5) 92.0 (+3.5) 91.5 (+4.0)
73.5 (+6.0) 74.5 (+5.0) 72.0 (+7.5) 76.0 (+3.5)
97.5 (+1.5) 97.5 (+1.5) 96.5 (+2.5) 96.5 (+2.5)
78.5 (+6.0) 79.5 (+5.0) 79.5 (+5.0) 78.5 (+6.0)
95.0 (+3.0) 93.5 (+4.5) 96.5 (+1.5) 95.0 (+3.0)
76.0 (+6.0) 78.0 (+4.0) 77.0 (+5.0) 78.5 (+3.5)
51.0 (+4.5) 50.5 (+5.0) 49.5 (+6.0) 50.0 (+5.5)
74.0 (+2.5) 74.5 (+2.0) 74.0 (+2.5) 73.0 (+3.5)
75.5 (+7.5) 77.0 (+6.0) 76.5 (+6.5) 78.5 (+4.5)
82.0 (+3.0) 82.0 (+3.0) 81.0 (+4.0) 80.5 (+4.5)
Llama
LLaVA
InternVL
Gemma
GLM
Llama
LLaVA
InternVL
Gemma
GLM
Llama
LLaVA
InternVL
Gemma
GLM
Zero-Shot (Clean) Fine-Tuning (Clean)
49.7 (+11.0) 60.7 (0.0)
34.3 (+9.2) 43.5 (0.0)
63.0 (+5.0) 68.0 (0.0)
43.3 (+8.4) 51.7 (0.0)
48.3 (+22.7) 71.0 (0.0)
0.0 (+58.8) 58.8 (0.0)
37.5 (+14.6) 52.1 (0.0)
53.9 (+5.3) 59.2 (0.0)
7.3 (+44.7) 52.0 (0.0)
0.0 (+58.9) 58.9 (0.0)
2.5 (+66.5) 69.0 (0.0)
11.0 (+9.5) 20.5 (0.0)
36.5 (+8.5) 45.0 (0.0)
51.0 (+16.0) 67.0 (0.0)
0.0 (+76.5) 76.5 (0.0)
Image Text Multimodal
60.3 (+0.4) 59.7 (+1.0) 58.0 (+2.7)
43.3 (+0.2) 41.7 (+1.8) 41.0 (+2.5)
67.7 (+0.3) 66.3 (+1.7) 67.7 (+0.3)
48.7 (+3.0) 51.3 (+0.4) 50.3 (+1.4)
67.7 (+3.3) 68.7 (+2.3) 67.7 (+3.3)
55.0 (+3.8) 53.9 (+4.9) 55.4 (+3.4)
49.5 (+2.6) 49.7 (+2.4) 48.8 (+3.3)
57.4 (+1.8) 56.8 (+2.4) 55.9 (+3.3)
50.6 (+1.4) 49.2 (+2.8) 50.1 (+1.9)
52.3 (+6.6) 53.7 (+5.2) 52.3 (+6.6)
65.5 (+3.5) 66.0 (+3.0) 66.0 (+3.0)
16.0 (+4.5) 16.5 (+4.0) 16.0 (+4.5)
41.5 (+3.5) 41.5 (+3.5) 40.5 (+4.5)
63.0 (+4.0) 60.5 (+6.5) 62.0 (+5.0)
71.0 (+5.5) 71.0 (+5.5) 73.5 (+3.0)
MMG UARD-BPH MMG UARD-CRS MMG UARD-BPH-Max MMG UARD-CRS-Max
56.3 (+4.4) 54.0 (+6.7) 57.2 (+3.5) 56.2 (+4.5)
39.0 (+4.5) 38.7 (+4.8) 39.2 (+4.3) 39.2 (+4.3)
65.0 (+3.0) 65.3 (+2.7) 64.5 (+3.5) 65.2 (+2.8)
45.7 (+6.0) 45.3 (+6.4) 47.5 (+4.2) 48.8 (+2.9)
65.7 (+5.3) 66.3 (+4.7) 65.8 (+5.2) 65.5 (+5.5)
52.1 (+6.7) 51.2 (+7.6) 53.2 (+5.6) 52.3 (+6.5)
46.2 (+5.9) 45.8 (+6.3) 47.2 (+4.9) 46.7 (+5.4)
55.4 (+3.8) 56.1 (+3.1) 55.0 (+4.2) 54.6 (+4.6)
47.5 (+4.5) 47.0 (+5.0) 47.5 (+4.5) 49.9 (+2.1)
52.1 (+6.8) 52.1 (+6.8) 51.3 (+7.6) 50.8 (+8.1)
63.5 (+5.5) 63.0 (+6.0) 64.0 (+5.0) 64.5 (+4.5)
13.5 (+7.0) 14.5 (+6.0) 13.0 (+7.5) 14.0 (+6.5)
41.0 (+4.0) 40.5 (+4.5) 41.0 (+4.0) 40.0 (+5.0)
60.5 (+6.5) 61.0 (+6.0) 59.5 (+7.5) 59.5 (+7.5)
68.5 (+8.0) 68.0 (+8.5) 68.0 (+8.5) 70.5 (+6.0)
MMStar Method
VQA-RAD
As a result, the low training loss does not transfer to cleantest accuracy. In contrast, the adversarial M AX variants follow the opposite pattern: their losses remain substantially above Clean Fine-Tuning and plateau during training. This suggests a direct training-disruption effect, where the protected samples prevent effective empirical risk minimization rather than merely offering an easier shortcut. The attacker, therefore, cannot recover by simply training longer. Together, these two regimes reveal complementary protection mechanisms: min-min variants induce shortcut overfitting, while M AX variants obstruct optimization. These dynamics support the design intuition in Sec. 5.5 and explain the consistent effectiveness observed in Fig. 3. Additional perdataset, per-backbone, and per-variant results are deferred to Appendix H.
RealworldQA
Accuracy
90
MMStar
80 70
96
70
60
92
60
50 FT
BPH
CRS
VQA-RAD
54
40
88 FT
BPH
CRS
TextVQA
90
84
80
70
50
70
60
48
60 FT
BPH
LoRA-8 LoRA-16
CRS
50
LoRA-64 QLoRA
FT
BPH
CRS
DocVQA
80
52
46
ScienceQA
100
80
50
Accuracy
DocVQA
50 FT
BPH
CRS
DoRA Projector
40
FT
BPH
CRS
Projector + LLM LoRA Full Fine-Tuning
Figure 5. Protection transferability of MMG UARD-BPH and MMG UARD-CRS across attacker fine-tuning strategies. Lower bars indicate stronger protection.
than a reliable protection signal. In contrast, all MMG UARD variants produce positive accuracy drops across the evaluations. The gains are modest on saturated tasks such as ScienceQA, where Clean Fine-Tuning is near the ceiling, but are more pronounced on visually grounded tasks such as TextVQA, DocVQA, MMStar, and VQA-RAD. These results identify cross-modal binding as the key transferable component: MMG UARD disrupts the attention route linking visual evidence, textual triggers, and answers, a structure broadly shared by LVLMs. Overall, MMG UARD transfers consistently to unseen architectures and provides more reliable black-box protection than binding-agnostic baselines.
6.3. Transferability Evaluation In practice, a defender has no prior knowledge of the LVLM or fine-tuning recipe an attacker may employ. We evaluate transferability along two axes: (i) across LVLMs in the black-box setting, and (ii) across attacker fine-tuning strategies. Transferability Across LVLMs. We evaluate five blackbox LVLMs with different architectures. Table 2 shows a clear gap between binding-aware and binding-agnostic protections. Single-modality Image and Text baselines, as well as the naive Multimodal combination, provide limited and unstable protection: their drops are often small and occasionally negative, meaning that protected fine-tuning can even improve clean-test accuracy over Clean Fine-Tuning. This suggests that input-space perturbation alone is insufficient for black-box transfer. When one modality remains clean, the attacker can still learn from the unprotected channel, causing the perturbation to behave more like regularization
Transferability Across Fine-Tuning Methods. A defenseaware attacker may change the fine-tuning recipe to bypass the protection signal. We fix the protected data and let the attacker use eight representative strategies: LoRA with ranks r=8/16/64, QLoRA, DoRA, projector-only tuning, projector + LLM LoRA, and full fine-tuning. Fig. 5 shows that both MMG UARD-BPH and MMG UARD-CRS consistently reduce accuracy compared with the corresponding 11
To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model Table 3. Multimodal stealthiness and semantic coherence of protected data. Arrows indicate whether higher or lower values are better. Human-study metrics are evaluated by three human experts and three SOTA LLM judges on a 1–3 scale. Metric
Clean
Image
Text
Multi.
BPH
+∞ 1.00 0.00 39.12 0.00 1.00
34.69 0.88 0.11 39.12 0.00 1.00
+∞ 1.00 0.00 89.72 5.57 0.98
34.63 34.48 34.68 0.88 0.87 0.88 0.11 0.11 0.11 89.24 91.91 88.28 29.48 36.29 29.70 0.93 0.93 0.93
2.52 2.86 2.92 2.60
3.00 1.33 2.00 1.99
2.52 1.36 2.14 2.33
coherence remains 2.14/3, and human answerability remains 2.33/3, indicating that the protected samples are still understandable and answerable to human users. Importantly, BPH and CRS introduce no additional observable stealthiness cost beyond the underlying image perturbation and text trigger, showing that the binding-disruption objective improves protection without further degrading multimodal usability.
CRS
Computational Metrics PSNR ↑ SSIM ↑ LPIPS ↓ PPL ↓ Edit Distance ↓ BLEU ↑
Algorithmic Efficiency. The cost of MMG UARD is dominated by surrogate LVLM forward/backward passes. For n samples, M surrogate models, R outer rounds, and Q inner adaptation steps, the overall time complexity is approximatelyO(RnM (Q + 1)CLVLM ), where CLVLM denotes one LVLM forward/backward update. The text-trigger search adds only a small overhead of O(RnLγ (|Vadm |d + cCfw )), since the trigger length Lγ and verified candidate size c are small constants. The attention-based binding loss reuses attention maps already computed by the LVLM and adds O(|K||H|L2 ) per pass, matching the standard attention order. The additional memory is mainly for image perturbations and trigger tokens, i.e., O(nHW C + nLγ ), plus negligible online attention aggregation. Since MMG UARD is an offline defender-side preprocessing step, it introduces no inference-time overhead after the protected dataset is generated. More detailed analysis of stealthiness trade-offs and a complete complexity breakdown are deferred to Appendix J.
Human Study / LLM-as-a-Judge Image Naturalness ↑ Text Naturalness ↑ Image-Text Coherence ↑ Human Answerability ↑
3.00 2.86 2.92 2.73
2.52 1.64 2.14 2.33
2.52 1.62 2.14 2.33
Clean Fine-Tuning reference across datasets and recipes, indicating that the protection is not tied to a specific tuning method. The degradation is most evident on visually grounded datasets such as TextVQA and DocVQA. Lowcapacity recipes, such as projector-only tuning, are particularly affected because the limited trainable parameters encourage reliance on the planted shortcut. Importantly, the effect persists under full fine-tuning, in which all model parameters are updated, indicating that increasing adaptation capacity alone cannot remove the protection signal. Overall, MMG UARD transfers across diverse fine-tuning strategies and limits recipe switching as an adaptive countermeasure. Detailed analyses are deferred to Appendix I.
6.5. Adaptive Attack Evaluation An attacker aware of MMG UARD may seek to invalidate the protection. We discuss and evaluate two adaptive strategies through (i) data transformation and (ii) data mixing. Robustness Against Data Transformations. A defenseaware attacker may preprocess the scraped data to weaken the protection signal. We evaluate nine common transformations, including image-side operations (RCP, JPEG compression, and blurring), text-side normalizations (punctuation removal, case normalization, and whitespace normalization), and their compositions. Fig. 7 summarizes the results. Across all four datasets and all transformations, MMG UARD-BPH and MMG UARD-CRS consistently remain below the clean fine-tuning reference under the same transformation, indicating that MMG UARD preserves its robustness under adaptive preprocessing. Among the evaluated transformations, the composition RCP+Punct is the most effective because it simultaneously perturbs the visual input and normalizes the textual input. Nevertheless, the protection effect persists: MMG UARD induces a strong shortcut through cross-modal attention binding, so surface-level changes to only part of the signal cannot fully neutralize it. More generally, image-side transformations are stronger than text-side normalizations, which is expected because these tasks rely heavily on visual evidence, and image trans-
6.4. Practicality Evaluation Beyond effectiveness, a deployable defense must remain unobtrusive to legitimate users and feasible to apply, so we evaluate MMG UARD on two practicality axes: (i) multimodal stealthiness; and (ii) algorithmic efficiency. Multimodal Stealthiness and Semantic Coherence. A practical data-side defense should preserve the utility of released data for human users. We evaluate image stealthiness using PSNR, SSIM, and LPIPS; text stealthiness using perplexity, edit distance, and BLEU; and semantic coherence using a 1–3 rubric covering image naturalness, text naturalness, image-text coherence, and human answerability. The rubric aggregates ratings from three human experts and three SOTA LLM judges. Table 3 shows that the image-side distortion is minor: MMG UARD-BPH and MMG UARDCRS achieve PSNR above 34 dB, SSIM around 0.88, and LPIPS 0.11, matching the Image-only and naive Multimodal baselines. The main perceptual cost stems from text-trigger insertion, which increases perplexity and lowers text naturalness. Nevertheless, BLEU remains high at 0.93, image-text 12
To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model RealworldQA
Accuracy
72
MMStar
72
ScienceQA
100
VQA-RAD
54
TextVQA
90
68
69
97
52
85
66
64
66
94
50
80
62
60
63
91
48
75
58
56
0
20
40
60
80
100
60
0
20
Protection Ratio
40
60
80
100
88
0
Protection Ratio
20
40
60
80
100
Protection Ratio Fine-Tuning (Clean)
46
0
20
40
60
80
100
70
Protection Ratio MMGuard-BPH
0
20
40
60
DocVQA
70
80
Protection Ratio
100
54
0
20
40
60
80
100
Protection Ratio
MMGuard-CRS
Figure 6. Protection effectiveness under attacker-side data mixing. Lower curves indicate stronger protection. ScienceQA
Accuracy
100 97
52
94
50
91 88
Accuracy
remains robust against attacker-side data mixing: clean data dilution may reduce the dosage of the protection signal, but it does not restore clean fine-tuning performance. Details analyses for robustness are deferred to Appendix K.
48 FT
BPH
CRS
TextVQA
100
46 70
90
65
80
60
70
55
60
VQA-RAD
54
FT None RCP
BPH
JPEG Blur
CRS
Punct Case
50
FT
BPH
CRS
DocVQA
FT WS RCP+Punct
BPH
6.6. Model Mechanism Analysis We analyze why and how MMG UARD and its components work from three perspectives: (i) an ablation study isolating the contribution of each component, (ii) a sensitivity analysis of the five core hyperparameters, and (iii) a visualizationbased analysis of the cross-modal binding mechanism.
CRS JPEG+Case Blur+WS
Ablation Study. The leftmost panel of Fig. 8 shows that the full MMG UARD-BPH achieves the strongest protection among all ablation variants. Using only image perturbations, only text triggers, or a naive multimodal combination yields noticeably higher accuracy, indicating that neither modality alone nor a simple combination is sufficient. This isolates the binding-disruption objective as the key component: it does not merely add image and text perturbations, but actively couples them into a shortcut that interferes with clean multimodal learning. In the full design, removing the image perturbation results in greater degradation than removing the text trigger, suggesting that the perturbation provides the dominant optimization signal, while the trigger acts as a discrete activator. Among the binding paths in Eq. (27), ablating the answer-anchored paths is more harmful than ablating the trigger-to-perturbation path, consistent with Theorem 5.1: the answer-anchored routes are directly responsible for shifting the attention distribution of protected data away from that of a clean one.
Figure 7. Protection robustness of MMG UARD-BPH and MMG UARD-CRS under attacker-side data transformations. Lower bars indicate stronger protection.
formations can partially distort the perturbation. Text-side normalizers are less effective because the trigger consists of admissible vocabulary tokens whose form and semantics are largely preserved. Overall, MMG UARD remains robust against standard attacker-side data transformations. Robustness Against Data Mixing. A defense-aware attacker may attempt to dilute the protection signal by mixing protected samples with clean data collected from external sources. We evaluate this setting by varying the protection ratio from 0% to 100% and reporting clean-test accuracy after fine-tuning. Fig. 6 summarizes the results. Across all six datasets, both MMG UARD-BPH and MMG UARDCRS consistently remain below the Clean Fine-Tuning reference, showing that data mixing cannot fully eliminate the protection effect. The degradation is particularly clear on TextVQA and DocVQA, where increasing the protected portion further reduces clean-test accuracy, with full protection yielding the largest gap relative to clean fine-tuning. On RealworldQA, MMStar, ScienceQA, and VQA-RAD, the curves are less strictly monotonic, but the protected models still stay below the clean reference across nearly all ratios. This indicates that even partial protected coverage can inject a usable pattern into the fine-tuning process. As the protection ratio increases, this shortcut receives stronger gradient support, but the exact accuracy trend depends on dataset difficulty and fine-tuning variance. Overall, MMG UARD
Parameter Analysis. The remaining panels of Fig. 8 examine the sensitivity of MMG UARD to its main hyperparameters. Increasing the binding-loss weight λbind strengthens MMG UARD-CRS, while MMG UARD-BPH exhibits a non-monotonic trend, reflecting the difference between CRS’s route-agnostic attention shifting and BPH’s more prescriptive binding objective. The inner-loop step Q is crucial: setting Q=0 removes the bilevel adaptation effect and substantially weakens protection, whereas a small positive value already captures most of the gain. For the gradient layer set K, shallower layers provide stronger protection,
13
To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model Ablation Studies
Accuracy
74
70
Binding Loss Weight
Inner-Loop Steps
74
72
Gradient Layer Depth
Image Perturbation Budget
85
80
71
65
71
68
75
70
68
60
68
64
65
60
65
55
65
60
55
50
62
50
62
56
45
Ablation Condition BPH Image Text
Multimodal w/o Pert. w/o Trig.
0.1
0.2
0.5
Weight
1.0
2.0
0
1
2
3
Steps
4
w/o Trig.→Pert. w/o Ans.→Pert. w/o Ans.→Trig.
5
1
2
3
Depth
MMGuard-BPH
5
7
1
2
4
8
Budget (×1/255)
16
40
Text Trigger Budget
1
3
5
Budget
7
9
MMGuard-CRS
Figure 8. Ablation study and parameter sensitivity of MMG UARD-BPH. The leftmost panel reports nine ablation conditions on the full method, while the remaining five panels sweep the binding-loss weight λbind , inner-loop steps Q, gradient layer depth |K|, image perturbation budget ϵx , and text trigger budget ϵt . Lower values indicate stronger protection across all panels. The results are averages across six datasets. Question: Is there a dedicated bicycle lane? Answer: Yes MMGuard-BPH
MMGuard-CRS
Trigger: 'Describe'
Trigger: 'Pent'
ple. The released image stays perceptually close to the clean input (Row 1), while the perturbation patches Ωδ (Row 2) form a structured area for the binding objective. Row 3 shows that attention is redirected from the genuine visual evidence (e.g., bicycle lane) onto Ωδ in off-content regions (e.g., sky), and under MMG UARD, answer-to-trigger, answer-to-perturbation, and trigger-to-perturbation attention all increase from the clean baseline, while attention to non-perturbed image tokens is suppressed. These patterns support the mechanism predicted by Theorem 5.1 because clean test inputs do not contain the learned trigger– perturbation binding, the model cannot reuse the shortcut acquired during protected fine-tuning. The layerwise curves also reveal different routing behaviors between the two variants. MMG UARD-BPH concentrates answer-to-trigger attention in early-to-middle LM layers, consistent with the prescribed binding paths in Eq. (27); MMG UARD-CRS induces stronger answer-to-perturbation attention in later layers, matching its route-agnostic objective in Eq. (29). Thus, both variants create a distributional shift between protected training and clean inference, but through different attention routes. This explains their similar overall effectiveness while allowing different trade-offs in transferability and robustness across architectures. Additional model design, parameter, and mechanism diagnostics are deferred to Appendix L.
No Perturbation
Answer attention map
Image Perturbation
Released image
Clean
Attention Mass
Attention Mass Distribution (×10−1 )
Answer to Perturbation (×10−2 ) 6
0.6 0.4
5
0.2
2
Tri. . → P. i. → P. → I∖P. Tex. Tr A. → A A. A. →
Clean
4
6
2 0
Answer to Trigger (×10−2 ) 9
8
3 0
10
20
LM Layer
MMGuard-BPH
30
0
0
10
20
LM Layer
30
MMGuard-CRS
Figure 9. Visualization of the cross-modal binding mechanism on a representative sample. Row 1: released images; Row 2: perturbation and perturbation tokens Ωδ (cyan boxes); Row 3: answer-token attention maps; Row 4: head-averaged attention mass distribution with corresponding layerwise curves.
suggesting that the relevant cross-modal binding behavior emerges early in multimodal fusion. Finally, both image and text budgets show clear strength–stealthiness trade-offs. Larger image perturbation budgets improve protection after a threshold, with ϵx =8/255 serving as a practical default, while increasing the text trigger budget further strengthens protection but may reduce textual naturalness. Overall, the ablation and sensitivity results support the design of MMG UARD: effective protection requires both modalities, explicit cross-modal binding disruption, and a moderate perturbation budget.
7. Limitations and Discussion. Pre-Publication Scope. MMG UARD operates only on defender-controlled data before release. Content already scraped or published without protection is outside its scope; likewise, an adversary with access to an unprotected copy may bypass the protected signal. Thus, MMG UARD should be viewed as one layer in a defense-in-depth strategy, complementing reactive mechanisms such as watermark-based provenance (Zhu et al., 2018; Kirchenbauer et al., 2023), machine unlearning (Bourtoule et al., 2021), and legal recourse rather than replacing them. As a proactive defense,
Cross-Modal Binding Mechanism Analysis. Fig. 9 visualizes the protection mechanism on a representative sam14
To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model
MMG UARD blocks a specific abuse pathway and lets complementary mechanisms address the remaining attack surface.
large vision-language models. By coupling bounded image perturbations with discrete text triggers and steering them through a cross-modal binding-disruption objective, MMG UARD reshapes how an attacker’s LVLM allocates attention among visual evidence, textual context, and target responses, so that fine-tuning minimizes loss along a protection-specific shortcut that does not transfer to clean evaluation. We formalized multimodal data protection under a practical threat model, derived a smoothness-based optimality bound for the discrete trigger search and a totalvariation lower bound for the binding mechanism, and instantiated the framework as a constrained multi-objective optimization over an ensemble of surrogate LVLMs. Across six multimodal datasets and nine open-source LVLMs spanning white-box, gray-box, and black-box scenarios, MMG UARD consistently degrades unauthorized fine-tuning performance while preserving perceptual fidelity, and remains effective under aggressive fine-tuning recipes, input transformations, and partial-coverage data mixing. We hope these results contribute a structural primitive for proactive multimodal data ownership and, in combination with reactive mechanisms such as watermarking and machine unlearning, help support a layered defense for public multimodal content.
Pretraining-from-Scratch Threats. Our threat model addresses the dominant efficient setting in which commercial actors fine-tune publicly released LVLM checkpoints on scraped corpora. We do not evaluate MMG UARD against adversaries who pre-train an LVLM from scratch on a corpus dominated by protected samples. Such a regime is economically unattractive for the long-tail data that we target; however, well-resourced adversaries (e.g., tech companies) with sufficiently large protected pools could, in principle, shift this trade-off. Quantifying the protection frontier against the LVLM pretraining settings is left for future work. Computational Overhead. Constructing unlearnable examples for LVLMs requires solving the multi-objective, constrained optimization problem in Eq. (34), which jointly couples continuous image perturbations, discrete trigger search, and surrogate-side adaptation under modality-specific budgets. This optimization is inherent to data-centric protection: the defender must solve it per sample before release, so Algorithm 1 incurs a one-time pre-release cost that scales linearly with the dataset size. We leave more advanced techniques, such as universal perturbation generators, shared surrogate caches across samples, and other acceleration techniques, to future work as a path toward web-scale deployment.
Ethics Considerations We developed and evaluated MMG UARD in accordance with the Menlo Report principles of respect for persons, beneficence, justice, and respect for law and public interest. MMG UARD is a defensive primitive whose explicit purpose is to return agency to data owners whose public image-text content is otherwise absorbed into commercial LVLM fine-tuning without consent. The principal beneficiaries are individual creators, news outlets, medical-image curators, and small dataset providers, whose interests are routinely at risk in current scraping practice; the principal cost falls on parties seeking to monetize unauthorized adaptation. We argue that this trade-off is well-aligned with the public interest, and we frame the considerations below accordingly.
Stealthiness Cost on the Text Channel. The residual stealthiness cost of MMG UARD is concentrated on the text channel. The inserted trigger raises perplexity and reduces text naturalness under our human and LLM-judge rubric, although BLEU and human answerability remain largely preserved (Table 3). This effect arises because the trigger search in Sec. 5.2 optimizes protection strength under a discrete admissibility constraint without an explicit fluency prior. Augmenting the screen-and-verify procedure with a language-model fluency term, or filtering the admissible vocabulary by fluency, offers a principled way to tighten this trade-off. Beyond LVLM. Our evaluation spans a diverse range of multimodal tasks achieved by LVLM, including visual question answering, domain-specific reasoning, documentgrounded reasoning, and understanding (Sec. 6.1), while newly emerging topics, such as multiagent systems and agentic tool use, in-context learning introduce distinct problem and optimization challenges. Extending MMG UARD to such regimes is a promising direction for future work.
Datasets and Models. All six evaluation datasets (RealWorldQA, MMStar, ScienceQA, VQA-RAD, TextVQA, DocVQA) are publicly released research benchmarks, used here strictly under their original licenses and intended scientific use. We used them only as carriers of a generic imagetext-response structure and did not extract, redistribute, or attempt to re-identify any depicted individuals. The medical subset (VQA-RAD) is the publicly released, de-identified Hugging Face version; we performed no patient-level analysis. The nine LVLMs we evaluate are all open-weight checkpoints used under their respective licenses, and protection generation does not require any non-public access to
8. Conclusion We presented MMG UARD, the first proactive, data-centric protection framework against unauthorized fine-tuning of 15
To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model
the models.
First, our defense raises the cost of unauthorized adaptation but does not eliminate it; data owners who rely solely on MMG UARD could over-estimate their protection, and we therefore consistently frame it as one layer of a defensein-depth strategy alongside watermarking, unlearning, and legal recourse (Sec. 2). Second, widespread deployment of unlearnable examples could, in principle, interact with legitimate downstream uses such as accessibility tooling that fine-tunes on user-supplied imagery; the perceptual budget we adopt keeps human-facing utility intact for the tasks we evaluate, but operators of such tools should be aware of the protection signal when consuming third-party data.
Human Subjects and LLM-as-a-Judge Study. Our stealthiness study (Sec. 6.4 and Appendix G) is a noninterventional rating task on protected versions of public benchmark images. We recruited three independent human raters and complemented them with three contemporary LLM judges (GPT, Gemini, and Claude Opus) per sample, applying them to ten samples per dataset for a total of 60 image-text pairs. The rubric (Table 6) elicits ordinal judgments about visual naturalness, textual fluency, cross-modal coherence, and answerability, and never asks raters about themselves or any third party. No personal data was collected from raters beyond what was needed to deliver the task; ratings were stored in de-identified form and used only in aggregate, and participation was voluntary with the right to withdraw at any time. The LLM judges were queried only with the protected sample under evaluation; no defenderside artifacts (surrogate gradients, optimization traces, or unreleased intermediate perturbations) were transmitted to third-party services.
References Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al. Qwen3vl technical report. arXiv preprint arXiv:2511.21631, 2025a. Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al. Qwen2.5VL technical report. arXiv preprint arXiv:2502.13923, 2025b.
Dual-Use Considerations. Like other unlearnable-example research, MMG UARD is dual-use in principle: the same optimization that produces protective shortcuts could in principle be retargeted to silently poison datasets that the perturber does not own. We took two structural steps to keep the contribution within its intended defensive envelope. First, the threat model in Sec. 4 restricts the defender’s reach to data they themselves control before publication, which is also the only setting in which our claims are validated; the framework does not propose, evaluate, or recommend application to third-party content. Second, the protection signal is bounded by a perceptual budget and leaves the humanfacing target response unchanged, so its observable effect on a recipient is degraded LVLM training utility rather than altered semantic content visible to humans. We do not view MMG UARD as enabling a novel attack capability beyond what is already available through prior unlearnable-example and adversarial-poisoning work (Huang et al., 2021; Fowl et al., 2021; Liu et al., 2024b), and our experiments do not target any specific deployed system.
Bourtoule, L., Chandrasekaran, V., Choquette-Choo, C. A., Jia, H., Travers, A., Zhang, B., Lie, D., and Papernot, N. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy, pp. 141–159, 2021. Carlini, N., Chien, S., Nasr, M., Song, S., Terzis, A., and Tramer, F. Membership inference attacks from first principles. In 2022 IEEE symposium on security and privacy (SP), pp. 1897–1914. IEEE, 2022. Chen, L., Li, J., Dong, X., Zhang, P., Zang, Y., Chen, Z., Duan, H., Wang, J., Qiao, Y., Lin, D., et al. Are we on the right way for evaluating large vision-language models? Advances in Neural Information Processing Systems, 37: 27056–27087, 2024. Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I., Sachdeva, N., Dhillon, I., Blistein, M., Ram, O., Zhang, D., Rosen, E., et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025.
Disclosure and Reproducibility Posture. Because MMG UARD is a data-side defense rather than a vulnerability in a specific product, no coordinated vulnerability disclosure was warranted. We notified no individual model provider, since the protection acts on the defender’s own data and does not exploit a flaw in any released checkpoint. To support replication and scrutiny by the security community, we plan to release the protection code, configuration files, and the ten-sample-per-dataset evaluation bundle used in the human/LLM-judge study, while withholding raw rater identifiers and any artifact that could re-identify participants.
Dang, P., Hu, X., Xu, K., Duan, J., Huang, D., Han, H., Zhang, R., and Du, Z. Unlearnable examples give a false sense of data privacy: Understanding and relearning. arXiv e-prints, pp. arXiv–2306, 2023. Ebrahimi, J., Rao, A., Lowd, D., and Dou, D. Hotflip: White-box adversarial examples for text classification. In
Residual Risks. Two residual concerns deserve naming. 16
To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model
Li, Y., Zhang, W., Lyu, X., Liu, Y., and Xiao, B. Styleguard: Preventing text-to-image-model-based style mimicry attacks by style perturbations. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025a.
Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 31–36, 2018. Foerster, H., Behrouzi, S., Rieger, P., Jadliwala, M., and Sadeghi, A.-R. {LightShed}: Defeating perturbationbased image copyright protections. In 34th USENIX Security Symposium (USENIX Security 25), pp. 7271– 7290, 2025.
Li, Z., Cai, J., Xu, G., Zheng, H., Li, Q., Zhou, F., Yang, S., Ling, C., and Wang, B. Versatile transferable unlearnable example generator. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025b.
Fowl, L., Goldblum, M., Chiang, P.-y., Geiping, J., Czaja, W., and Goldstein, T. Adversarial examples make strong poisons. In Advances in Neural Information Processing Systems, volume 34, pp. 30339–30351, 2021.
Li, Z., Xu, G., Cai, J., Fang, R., Wu, D., Lao, Q., Ling, C., and Wang, B. When priors backfire: On the vulnerability of unlearnable examples to pretraining. arXiv preprint arXiv:2603.04731, 2026.
Fu, S., He, F., Liu, Y., Shen, L., and Tao, D. Robust unlearnable examples: Protecting data privacy against adversarial learning. In International Conference on Learning Representations, 2022.
Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. In Advances in Neural Information Processing Systems, volume 36, pp. 34892–34916, 2023a.
Gemma Team, Kamath, A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., Perrin, S., Matejovicova, T., Rame, A., Riviere, M., et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025.
Liu, H., Li, C., Li, Y., and Lee, Y. J. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 26296–26306, 2024a.
Gokul, V. and Dubnov, S. PosCUDA: Position based convolution for unlearnable audio datasets. arXiv preprint arXiv:2401.02135, 2024.
Liu, S., Yao, Y., Jia, J., Casper, S., Baracaldo, N., Hase, P., Yao, Y., Liu, C. Y., Xu, X., Li, H., et al. Rethinking machine unlearning for large language models. Nature Machine Intelligence, 7(2):181–194, 2025.
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Liu, X., Jia, X., Xun, Y., Liang, S., and Cao, X. Multimodal unlearnable examples: Protecting data against multimodal contrastive learning. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 8024–8033, 2024b.
Hong, W., Yu, W., Gu, X., Wang, G., Gan, G., Tang, H., Cheng, J., Qi, J., Ji, J., Pan, L., et al. Glm-4.5 v and glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning. arXiv preprint arXiv:2507.01006, 2025.
Liu, Y., Fan, C., Zhou, P., and Sun, L. Unlearnable graph: Protecting graphs from unauthorized exploitation. arXiv preprint arXiv:2303.02568, 2023b.
Huang, H., Ma, X., Erfani, S. M., Bailey, J., and Wang, Y. Unlearnable examples: Making personal data unexploitable. In International Conference on Learning Representations, 2021.
Liu, Y., Fan, C., Dai, Y., Chen, X., Zhou, P., and Sun, L. MetaCloak: Preventing unauthorized subject-driven textto-image diffusion-based synthesis via meta-learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24219–24228, 2024c.
Kirchenbauer, J., Geiping, J., Wen, Y., Katz, J., Miers, I., and Goldstein, T. A watermark for large language models. In International Conference on Machine Learning, pp. 17061–17084, 2023.
Liu, Y., Xu, K., Chen, X., and Sun, L. Stable unlearnable example: Enhancing the robustness of unlearnable examples via stable error-minimizing noise. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 3783–3791, 2024d.
Lau, J. J., Gayen, S., Ben Abacha, A., and DemnerFushman, D. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):180251, 2018.
Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in neural information processing systems, 35:2507–2521, 2022.
Li, X. and Liu, M. Make text unlearnable: Exploiting effective patterns to protect personal data. In Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023), pp. 249–259, 2023. 17
To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
conference on computer vision and pattern recognition, pp. 8317–8326, 2019. Singh, A., Fry, A., Perelman, A., Tart, A., Ganesh, A., El-Kishky, A., McLaughlin, A., Low, A., Ostrow, A., Ananthram, A., et al. OpenAI GPT-5 system card. arXiv preprint arXiv:2601.03267, 2025.
Mathew, M., Karatzas, D., and Jawahar, C. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2200–2209, 2021.
Sun, Y., Zhang, H., Zhang, T., Ma, X., and Jiang, Y.-G. Unseg: One universal unlearnable example generator is enough against all image segmentation. Advances in Neural Information Processing Systems, 37:79168– 79193, 2024.
Metz, C., Kang, C., Frenkel, S., Thompson, S. A., and Grant, N. How tech giants cut corners to harvest data for AI. The New York Times, 2024. https://www.nytimes.co m/2024/04/06/technology/tech-giants-harvest -data-artificial-intelligence.html.
Van Le, T., Phung, H., Nguyen, T. H., Dao, Q., Tran, N. N., and Tran, A. Anti-dreambooth: Protecting users from personalized text-to-image synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2116–2127, 2023.
Millette, D. Class action complaint: Millette v. OpenAI, Inc. et al., case no. 3:24-cv-04710. United States District Court for the Northern District of California, Document 1, August 2024. URL https://storage.courtliste ner.com/recap/gov.uscourts.cand.433675/gov.u scourts.cand.433675.1.0.pdf.
Wang, W., Gao, Z., Gu, L., Pu, H., Cui, L., Wei, X., Liu, Z., Jing, L., Ye, S., Shao, J., et al. InternVL3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025a.
Nguyen-Le, H.-H., Tran, V.-T., Nguyen, D.-T., and LeKhac, N.-A. Passive deepfake detection across multimodalities: A comprehensive survey. arXiv preprint arXiv:2411.17911, 2024.
Wang, Y., Lu, Y., Gao, X.-S., Kamath, G., and Yu, Y. Bridgepure: Limited protection leakage can break blackbox data protection. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025b.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748–8763, 2021.
xAI. RealWorldQA. Hugging Face dataset, 2024. Available: https://huggingface.co/datasets/xai-org/RealworldQA.
Ren, J., Xu, H., Wan, Y., Ma, X., Sun, L., and Tang, J. Transferable unlearnable examples. In International Conference on Learning Representations, 2023.
Yao, F., Zhuang, Y., Sun, Z., Xu, S., Kumar, A., and Shang, J. Data contamination can cross language barriers. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 17864–17875, 2024.
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. LAION-5B: An open large-scale dataset for training next generation image-text models. In Advances in Neural Information Processing Systems, volume 35, pp. 25278–25294, 2022.
Yao, Y., Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., Cai, T., Chen, C., Li, H., Zhao, W., et al. Efficient gpt-4v level multimodal large language model for deployment on edge devices. Nature Communications, 16(1):5509, 2025.
Shan, S., Cryan, J., Wenger, E., Zheng, H., Hanocka, R., and Zhao, B. Y. Glaze: Protecting artists from style mimicry by text-to-image models. In 32nd USENIX Security Symposium, pp. 2187–2204, 2023.
Yu, Z., Zhai, S., and Zhang, N. Antifake: Using adversarial audio to prevent unauthorized speech synthesis. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pp. 460–474, 2023.
Shan, S., Ding, W., Passananti, J., Wu, S., Zheng, H., and Zhao, B. Y. Nightshade: Prompt-specific poisoning attacks on text-to-image generative models. In 2024 IEEE Symposium on Security and Privacy, pp. 807–825, 2024.
Zhang, J., Ma, X., Yi, Q., Sang, J., Jiang, Y.-G., Wang, Y., and Xu, C. Unlearnable clusters: Towards label-agnostic unlearnable examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3984–3993, 2023a.
Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M. Towards vqa models that can read. In Proceedings of the IEEE/CVF 18
To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model
Zhang, Z., Yang, Q., Wang, D., Huang, P., Cao, Y., Ye, K., and Hao, J. Mitigating unauthorized speech synthesis for voice protection. In Proceedings of the 1st ACM Workshop on Large AI Systems and Models with Privacy and Safety Analysis, pp. 13–24, 2023b. Zhao, Z., Duan, J., Hu, X., Xu, K., Wang, C., Zhang, R., Du, Z., Guo, Q., and Chen, Y. Unlearnable examples for diffusion models: Protect data from unauthorized exploitation. arXiv preprint arXiv:2306.01902, 2023. Zhu, J., Kaplan, R., Johnson, J., and Fei-Fei, L. HiDDeN: Hiding data with deep networks. In Proceedings of the European Conference on Computer Vision, pp. 657–672, 2018. Zhu, J., Wang, W., Chen, Z., Liu, Z., Ye, S., Gu, L., Tian, H., Duan, Y., Su, W., Shao, J., et al. InternVL3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479, 2025.
19
To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model
Contents of the Appendix A Notation
21
B Optimality of Trigger Updates
21
C Contrastive Form of the Attention-Mass Loss
22
D Proof of Bridge Path Hijack Effectiveness
23
E Dataset Details
23
E.1 Benchmark Descriptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F Training Details
24
F.1
Defender-Side Protection Generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
24
F.2
Attacker-Side Fine-Tuning and Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
25
G Human Evaluation Rubric
25
G.1 Rubric for Human Raters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
26
G.2 LLM-as-a-Judge Prompt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
26
H Detailed Effectiveness Analysis
I
J
24
27
H.1 Accuracy Analysis on Protected Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
27
H.2 Training Dynamics on Protected Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
28
Detailed Transferability Analysis
28
I.1
Cross-Model Transferability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
28
I.2
Cross-Recipe Transferability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
29
Detailed Practicality Analysis
29
J.1
29
Stealthiness and Computational Cost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
K Detailed Adaptive Attack Evaluation
30
K.1 Robustness Against Aggressive Attackers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . L Detailed Mechanism Analysis
30 31
L.1 Ablation and Parameter Sensitivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
31
L.2 Cross-Modal Binding Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
32
L.3 Protection Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
32
M Use of Generative AI
35 20
To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model
A. Notation Table 4 summarizes the main notation used in the problem formulation and method. Table 4. Summary of main notation. Symbol
Name
Meaning
D = {(xi , ti , yi )}n i=1 e D
Clean multimodal dataset Protected dataset
x i , t i , yi x̃i , t̃i X, T , Y
Clean sample components Protected inputs Data spaces
Collection of image, text, and target-response samples. Dataset released by the defender after applying image and text protections. Image, textual input, and target output for sample i. Perturbed image and trigger-inserted text released for sample i. Image space, natural-language input space, and target-output space.
fθ θ = (θfrz , θtr ) Θ f˜(m) M , ωm
LVLM Model parameters Parameter space Surrogate LVLM Ensemble size and weight
Model mapping an image-text input to an output response. Frozen pretrained parameters and trainable fine-tuning parameters. Feasible model-parameter space used in the attacker’s training objective. The m-th surrogate model available to the defender. Number of surrogate LVLMs and the non-negative weight of surrogate m.
Gϕ , ϕ, Φ B(D) d x , dt ϵ x , ϵt δi , γ i Ciδ , Ciγ cand V, Vadm , Vi,j
Protection map and parameters Protection budget set Distance measures Budgets Protection variables Feasible sets Token sets
Defender-side transformation, its parameters, and feasible protection-parameter space. Set of protected datasets satisfying modality-specific constraints. Visual and textual change measures. Image perturbation budget and text trigger-length budget. Image perturbation and inserted text trigger for sample i. Allowed image perturbations and text triggers for sample i. Tokenizer vocabulary, admissible trigger vocabulary, and screened candidate tokens.
gx , g̃x Ex , Px , Et Hix , Zix , Zit , Si
Image processors Encoders/projector LVLM representations
Practical image processor and differentiable surrogate processor. Visual encoder, modality projector, and text embedding map. Visual features, projected visual tokens, text embeddings, and joint multimodal sequence.
Ltrain , Leval ℓtrain , ℓprot ℓbind , ℓjoint
Dataset-level losses Optimization losses Binding and joint losses
λtrain , λbind
Loss weights
Training and clean evaluation objectives. Sample-level training loss and defender protection loss. Attention-binding disruption loss and weighted single-surrogate protection loss. Weights for task-level fitting and binding-disruption terms.
Ωx , Ωt , Ωγ , Ωy
Token subsets
Ωδ , τδ , ρi,b
Perturbation tokens
A(k,h) , Āk (S)
Attention distributions
H, K S, R, UR Bk , zk
Head and layer sets Source and target sets Attention support and logits BPH weights
β1 , β 2 , β 3
Image, original text, inserted trigger, and answer tokens in the LVLM sequence. Top-τδ fraction of image tokens by perturbation magnitude; ρi,b is the average ℓ1 magnitude for token b. Attention matrix for layer k and head h, and head-averaged attention mass from source set S. Attention heads and layers used for binding-disruption objectives. Source token set, target token set, and uniform target distribution over R. Support of nonzero attention mass at layer k and corresponding effective logits. Weights for the three Bridge Path Hijack attention paths.
B. Optimality of Trigger Updates Lemma B.1 (Trigger Updates Optimality). Suppose ℓprot is L-smooth as a function of the embedding at trigger position j, [j=v] and let R = maxv∈Vadm ∥ev − eγi,j ∥. Let v † ∈ arg minv∈Vadm ℓprot γi be the exact best admissible substitution, and 21
To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model cand ⋆ ranked by si,j (·). Then be the candidate selected by Eq. (20) from the top-c screen Vi,j let vi,j ⋆ [j=vi,j ]
ℓprot γi
[j=v † ]
≤ LR2 .
− ℓprot γi
(36)
Proof. For position j of the trigger γi , let e = eγi,j denote its current embedding and write ℓ(e) = ℓprot (γi ) as a function of that embedding with all other variables fixed. Define the candidate-induced loss change [j=v]
∆i,j (v) = ℓprot γi
− ℓprot (γi ),
(37)
and recall the linear score si,j (v) = (ev − e)⊤ ∇e ℓ from Eq. (19). By the standard descent-lemma form of L-smoothness applied to ℓ along the segment from e to ev , 2
∆i,j (v) − si,j (v) ≤ L2 ∥ev − e∥2 ≤ LR 2 ,
(38)
for every admissible v ∈ Vadm . Consider two cases for the exact admissible optimum v † . cand cand Case 1: v † ∈ Vi,j . The verification step in Eq. (20) performs an exact minimization of ℓprot over Vi,j , so ⋆ [j=vi,j ]
ℓprot (γi
[j=v † ]
) ≤ ℓprot (γi
) and Eq. (36) holds with gap zero.
cand cand Case 2: v † ∈ / Vi,j . By the top-c screening rule, every w ∈ Vi,j has si,j (w) ≤ si,j (v † ). Pick any such w. Applying † Eq. (38) to both w and v ,
∆i,j (w) ≤ si,j (w) + LR 2
2
≤ si,j (v † ) + LR 2
2
≤ ∆i,j (v † ) + LR2 .
(39)
⋆ cand ⋆ The verification step then selects vi,j ∈ Vi,j as the exact minimizer of ℓprot on the shortlist, so ∆i,j (vi,j ) ≤ ∆i,j (w) ≤ † 2 ∆i,j (v ) + LR . Subtracting the common offset ℓprot (γi ) from both sides of the last inequality recovers Eq. (36). □
C. Contrastive Form of the Attention-Mass Loss Proposition C.1 (Contrastive Learning for Attention Mass Alignment). For each k ∈ K, let Bk = {b ∈ Ω : Āk,b (S) > 0}. If R ⊆ Bk for all k ∈ K, write each Āk (S) on its support as a softmax over effective logits zk . Then the attention-mass loss in Eq. (26) satisfies ℓmass (S, R) =
1 X (k) ℓNCE (S, R) − log |R|, |K|
(40)
1 X exp(zk,r ) − log P . |R| b∈Bk exp(zk,b )
(41)
k∈K
(k)
ℓNCE (S, R) =
r∈R
Proof. Fix any k ∈ K. If any r ∈ R has Āk,r (S) = 0, then DKL (UR ∥Āk (S)) = +∞, so the finite case requires R ⊆ Bk for every averaged layer. In this case, since UR is uniform on R and zero elsewhere, X UR (b) UR (b) log DKL UR ∥Āk (S) = Ā k,b (S) b∈Ω 1 X 1/|R| log |R| Ā k,r (S) r∈R 1 X = − log Āk,r (S) − log |R|. |R| =
r∈R
22
(42)
To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model
Substituting the softmax form Āk,r (S) = exp(zk,r )/
P
b∈Bk exp(zk,b ) into the last line gives Eq. (41) for layer k. Averaging (k) the resulting identity over k ∈ K gives Eq. (40). The term ℓNCE is the average softmax loss obtained by selecting each
r ∈ R as a positive token and normalizing against the attention support. Therefore, minimizing the finite attention-mass loss is equivalent, up to the constant − log |R|, to minimizing an averaged multi-positive contrastive objective whose negatives are the non-target tokens in each supported LVLM context. □
D. Proof of Bridge Path Hijack Effectiveness Proof. Fix sample i, a protection-induced target set Ri ∈ {Ωγ,i , Ωδ,i }, and any layer k ∈ K. Write Pk = Āpk,i (Ωy ) and Qk = Āck,i (Ωy ), both viewed as probability distributions on the common, zero-padded token universe in which the positions in Ri receive mass 0 on the clean side. The latter convention follows from the input-dependent definition of Ri : trigger tokens are physically absent at clean evaluation, and the perturbation-magnitude ranking that selects Ωδ,i in Eq. (23) is degenerate when δi = 0. Step 1: Pinsker on the protected attention. By definition of ℓmass in Eq. (26), 1 X DKL URi ∥ Pk = ℓmass (Ωy,i , Ri ) ≤ η. |K|
(43)
k∈K
p Pinsker’s inequality√applied per layer gives TV(URi , Pk ) ≤ DKL (URi ∥Pk )/2, and Jensen’s inequality applied to the concave map u 7→ u yields p 1 X TV URi , Pk ≤ η/2. (44) |K| k∈K
Step 2: Target-mass concentration. For any probability distributions P, U on a finite set, TV(P, U ) ≥ |P (B) − U (B)| for every event B. Choosing B = Ri and noting URi (Ri ) = 1, TV URi , Pk ≥ 1 − Pk (Ri ), (45) so combining with Eq. (44) yields p 1 X Pk (Ri ) ≥ 1 − η/2. |K|
(46)
k∈K
That is, on average across layers, at least a 1 − placed on positions in the target set Ri .
p
η/2 fraction of the answer-token attention mass on protected inputs is
Step 3: Forced redistribution at clean evaluation. Because Ri is empty under the clean evaluation input, the model assigns no attention mass to the positions in Ri on the zero-padded clean side: Qk (Ri ) = 0 for every k. Applying the same event-based bound to the pair (Pk , Qk ) with B = Ri , TV(Pk , Qk ) ≥ Pk (Ri ) − Qk (Ri ) = Pk (Ri ).
(47)
Averaging over k ∈ K and using Eq. (46) gives p 1 X 1 X TV(Pk , Qk ) ≥ Pk (Ri ) ≥ 1 − η/2, |K| |K| k∈K
(48)
k∈K
which is Eq. (28). The argument is uniform in the choice of Ri ∈ {Ωγ,i , Ωδ,i }, so the bound holds for both the β2 and β3 binding terms of Eq. (27). □
E. Dataset Details This section expands the compact dataset summary in Sec. 6.1 by providing the source-reported objectives, formats, and statistics for each of the six benchmark datasets used in our evaluation, followed by representative protected and clean image-text examples. 23
To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model
E.1. Benchmark Descriptions RealWorldQA (xAI, 2024) is an image-text benchmark released on Hugging Face with 765 examples in a single test split. Each example contains an image, a natural-language question, and a free-form answer. The dataset card shows questions involving real-world visual attributes such as object counts, relative position, traffic lights, road layout, and spatial relations. MMStar (Chen et al., 2024) is a vision-indispensable multimodal benchmark. The released dataset contains 1,500 multiplechoice offline evaluation samples selected from 22,401 initial samples through filtering and manual review. The benchmark is organized into six core capabilities and 18 detailed axes, with 250 samples per core capability. ScienceQA (Lu et al., 2022) is collected from elementary and high-school science curricula and contains 21,208 multimodal multiple-choice science questions. Among these questions, 10,332 have image context, 10,220 have text context, and 6,532 have both. The dataset spans natural, language, and social sciences and is organized into 26 topics, 127 categories, and 379 skills. VQA-RAD (Lau et al., 2018) is a radiology visual question answering dataset. The cleaned Hugging Face version contains 2,244 image-question-answer triplets over 314 referenced images, after removing duplicate or overlapping triplets. The question set includes both open-ended free-form questions and close-ended yes/no questions; the supported evaluation distinguishes close-ended yes/no accuracy, open-ended accuracy, and overall accuracy. TextVQA (Singh et al., 2019) is a visual question answering dataset for natural images containing readable scene text. It contains 45,336 questions on 28,408 images, with free-form answers collected from annotators. The benchmark is designed for questions whose answers require reading the text in the image and reasoning about it in the context of the image and the question. DocVQA (Mathew et al., 2021) is a visual question answering dataset for document images. It contains 50,000 free-form questions over more than 12,000 document images. The dataset focuses on document-image understanding and includes questions for which document structure and layout can be important for answering.
F. Training Details This section details the configurations used by both the defender’s protection-generation pipeline and the attacker’s downstream supervised fine-tuning (SFT). The complete hyperparameter tuning grid spanning the surrogate inner loop, the protection outer loop, and the attacker SFT is summarized in Table 5. We organize the description into the defender-side protection generation and the attacker-side fine-tuning and evaluation. F.1. Defender-Side Protection Generation Surrogate Fine-Tuning (Inner Loop). Within each outer round of Algorithm 1, the surrogate is adapted to the current protected samples by Q supervised gradient steps. We use LoRA with rank 8 for Qwen3-VL-4B-Instruct and rank 16 for MiniCPM-V-4, with dropout 0.1 and target modules all (attention and MLP projections). The inner learning rate is swept over {1×10−5 , 1×10−4 } with per-device batch size 2, and the inner-step count Q is swept over {0, 1, 2, 3, 4, 5}. Surrogate adapter weights are discarded after each round and never released. Protection Optimization (Outer Loop). MMG UARD runs for R outer rounds, with R swept over {1, 3, 5, 7}. The image perturbation δi is updated by PGD under an ℓ∞ budget swept over ϵx ∈ {1, 2, 4, 8, 16}/255, with step size α=1/255 and the number of PGD iterations per round swept over {0, 1, 2, 5, 8}, followed by projection onto the feasible set Ciδ . The text trigger γi is updated by gradient-guided screening followed by exact verification: at each round, we rank candidate tokens by the HotFlip score in Eq. (19), retain the top-K candidates per slot with K swept over {8, 16, 64, 128, 512}, and select the substitution that minimizes the joint loss in Eq. (20). The admissible vocabulary Vadm is restricted to ASCII alphabetic words with numeric and punctuation tokens forbidden and a leading whitespace prepended; the trigger length ϵt is swept over {1, 3, 5, 7, 9} tokens. The joint loss in Eq. (34) combines a task-side term (λtrain =1.0) and a binding-disruption term with weight λbind swept over {0.1, 0.2, 0.5, 1.0, 2.0}. For MMG UARD-BPH, the answer-to-trigger, trigger-to-image, and answerto-image paths each receive unit weight; for MMG UARD-CRS, the routing-shift loss replaces the three structured paths. The binding gradient is restricted to the first |K| attention layers, with |K| swept over {1, 2, 3, 4, 5, 6, 7}. We instantiate the outer objective as either the adversarial variant in Eq. (32) or the min-min variant in Eq. (30). For the black-box scenario, the ensemble surrogates are Qwen3-VL-4B-Instruct and MiniCPM-V-4 with uniform weights ωm =1/M .
24
To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model
F.2. Attacker-Side Fine-Tuning and Evaluation Attacker Fine-Tuning (Downstream SFT). After receiving the released dataset, the attacker fine-tunes each target LVLM in a supervised manner. We evaluate six fine-tuning recipes that span the threat-model spectrum: LoRA, QLoRA, DoRA, projector-only, projector + LLM LoRA, and full fine-tuning. For LoRA-style adapters, we further sweep the rank over {8, 16, 64}. We additionally sweep the learning rate over {1×10−5 , 4×10−5 , 1×10−4 , 3×10−4 }, the number of training epochs over {1, 2, 4, 8, 10}, and the per-device batch size over {1, 2, 4}, with cosine schedule and warmup ratio 0.1 held fixed. The training set is the protected version of each dataset’s standard split. Additional aggressive-attacker variants—inputmodality sanitization (RCP, JPEG, Blur, Punct/Case/WS) and clean-data mixing at ratios {0, 5, 10, 20, 40, 60, 80, 100}%— are described in Sec. 6.4 and Appendix I. Evaluation Protocol. Predictions are scored under each benchmark’s official answer convention. For multiple-choice datasets (MMStar, ScienceQA), a prediction is correct if the generated answer matches the ground-truth option letter after answer normalization (case folding, whitespace stripping, and prefix removal). For short-answer and free-form datasets (RealWorldQA, VQA-RAD, TextVQA, DocVQA), we use normalized exact match; when a dataset provides answer aliases, any alias is accepted as correct. The per-dataset answer formats are summarized in Appendix E. Table 5. Hyperparameter tuning grid of MMG UARD and the attacker SFT. Group / Parameter
Values
Surrogate fine-tuning (inner loop) LoRA rank LoRA dropout / target modules Learning rate / per-device batch Inner steps Q
8 / 16 0.1 / all {1×10−5 , 1×10−4 } / 2 {0, 1, 2, 3, 4, 5}
Protection optimization (outer loop) Outer rounds R Image budget ϵx (×1/255) PGD step α / iterations Trigger length ϵt (tokens) HotFlip top-K candidates λtrain λbind Binding-layer count |K| Black-box ensemble surrogates
{1, 3, 5, 7} {1, 2, 4, 8, 16} 1/255 / {0, 1, 2, 5, 8} {1, 3, 5, 7, 9} {8, 16, 64, 128, 512} 1.0 {0.1, 0.2, 0.5, 1.0, 2.0} {1, 2, 3, 4, 5, 6, 7} Qwen3-VL-4B-Instruct, MiniCPM-V-4
Attacker SFT Fine-tuning recipe LoRA rank Learning rate Scheduler / warmup ratio Epochs Batch size (per device)
{LoRA, QLoRA, DoRA, Projector, Projector+LLM LoRA, Full FT} {8, 16, 64} {1×10−5 , 4×10−5 , 1×10−4 , 3×10−4 } cosine / 0.1 {1, 2, 4, 8, 10} {1, 2, 4}
G. Human Evaluation Rubric To assess whether protected multimodal examples remain suitable for human interpretation and downstream use, we evaluate each image-text pair using a shared rubric across four dimensions: image naturalness, text naturalness, image-text coherence, and human answerability. Each dimension is rated on a three-point ordinal scale, where 1 denotes severe degradation, 2 denotes mild but tolerable degradation, and 3 denotes no noticeable degradation. Together, these dimensions capture perceptual image quality, linguistic fluency, cross-modal semantic alignment, and task-level usability. Study Setup. We conduct the study on 60 protected image-text pairs in total, comprising 10 randomly selected samples per dataset across the six benchmarks. Each pair is independently rated by three human experts and three commercial LLM judges: GPT-5.4, Gemini-3.1-Pro-Preview, and Claude-Sonnet-4-6 under the same four-dimensional rubric described below. 25
To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model
The values reported in the Human Study / LLM-as-a-Judge block of Table 3 are the mean per-dimension score across these six raters per sample. The rater-facing rubric and the LLM-judge prompt are described in the two subsubsections below. G.1. Rubric for Human Raters Table 6 formalizes the four dimensions for human raters, listing the precise definition and the three-point criteria for each.
Table 6. Human evaluation rubric for protected multimodal examples. Dimension
Definition
Criteria
Image Naturalness
The degree to which the protected image preserves perceptual quality and visual plausibility, without noticeable perturbation-induced noise, distortion, color artifacts, texture irregularities, or other abnormal patterns that would make the image appear manipulated or low-quality to a human observer.
1: The image contains clearly visible artifacts, corruption, or unnatural patterns that interfere with normal perception. 2: The image is generally recognizable and usable, but contains mild visual artifacts, slight distortion, or subtle abnormal patterns. 3: The image appears visually natural, clean, and indistinguishable from a normal, unmodified image.
Text Naturalness
The degree to which the protected text remains fluent, grammatical, readable, and contextually appropriate. Inserted or substituted tokens are penalized according to how much they disrupt readability and plausibility: clearly random, code-like, or semantically incoherent strings indicate low naturalness, while isolated awkward words or recognizable real-word/proper-noun fragments may still be understandable but stylistically unusual.
1: The text is grammatically flawed, difficult to read, or contains clearly random, code-like, gibberish, or strongly suspicious tokens that noticeably disrupt natural reading. 2: The text remains understandable but includes minor awkwardness, unusual wording, single-word substitutions, or inserted recognizable words/proper nouns/phrase-like fragments that are stylistically abnormal but do not prevent comprehension. 3: The text is fluent, coherent, and appears naturally written without noticeable suspicious, irrelevant, or stylistically abnormal content.
Image-Text Coherence
The degree to which the protected image and protected text remain semantically aligned as a multimodal pair, such that the text refers to visual content present in the image and the image provides sufficient evidence for the textual input or question.
1: The text is largely inconsistent with the image, refers to absent or contradictory visual content, or forms an incoherent multimodal pair. 2: The text is partially related to the image, but some visual references are vague, incomplete, or only weakly supported by the image. 3: The text is clearly and semantically consistent with the image, and the image provides appropriate visual evidence for the text or question.
Human Answerability
The degree to which a human observer can understand and respond to the intended task from the protected image-text pair, such as visual question answering, document understanding, or multimodal reasoning, without being hindered by perturbation-induced ambiguity or degradation.
1: The question or task cannot be answered reliably from the protected pair due to visual degradation, textual ambiguity, or image-text mismatch. 2: The question or task is answerable, but with noticeable uncertainty caused by mild ambiguity, reduced clarity, or incomplete visual/textual evidence. 3: The question or task is clearly answerable from the protected pair, with sufficient visual and textual information for a confident human response.
G.2. LLM-as-a-Judge Prompt For the automated evaluation, we instantiate the following LLM-as-a-judge prompt for each protected image-text pair. The model is instructed to rate each dimension independently according to the same ordinal rubric and to return only a JSON object for consistent parsing and aggregation. The prompt includes criterion-specific placeholders for the Definition and Criteria fields in Table 6, ensuring that every judge receives the same rubric content. 26
To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model
Prompt for LLM-as-a-Judge Evaluation You are evaluating whether a protected multimodal image-text pair remains natural, semantically coherent, and usable for human interpretation. Inputs: Image: <image> Text: <text> Please rate the sample on the following four criteria using a 1-3 ordinal scale. Score each criterion independently, respect the ordered meaning of the rubric levels, and do not invent intermediate scores. If the evidence is ambiguous between two adjacent levels, choose the lower-supported score. 1. Image Naturalness Definition: <image_naturalness_definition> Criteria: <image_naturalness_criteria> 2. Text Naturalness Definition: <text_naturalness_definition> Criteria: <text_naturalness_criteria> 3. Image-Text Coherence Definition: <image_text_coherence_definition> Criteria: <image_text_coherence_criteria> 4. Human Answerability Definition: <human_answerability_definition> Criteria: <human_answerability_criteria> Return JSON only: { "image_naturalness": 1, "text_naturalness": 1, "image_text_coherence": 1, "human_answerability": 1, "brief_reason": "..." }
H. Detailed Effectiveness Analysis This section extends Sec. 6.2 with per-dataset, per-backbone, and per-variant observations. We organize the analysis into two parts: the post-fine-tuning accuracy panel in Fig. 3, examined along the dataset, target, and variant axes; and the training-loss dynamics in Fig. 4, examined along the optimization axis. H.1. Accuracy Analysis on Protected Data Dataset Sensitivity. The magnitude of the protection drop varies systematically with the role of the visual modality across datasets. Vision-heavy and OCR-driven benchmarks are most strongly protected on the white-box surrogate Qwen3-VL-4B: TextVQA falls by up to ∼30 points under the -Max variants (Clean FT ∼87% →∼58%), DocVQA by up to ∼15 points (Clean FT ∼67% →∼52%), and MMStar by up to ∼17 points (Clean FT ∼67% →∼50%). These tasks rely on dense visual evidence—scene text, document layout, vision-indispensable reasoning—so disrupting the cross-modal binding directly removes the dominant supervision signal. RealWorldQA exhibits a moderate drop (9–18 points white-box) on the same surrogate. In contrast, ScienceQA and VQA-RAD, where the language prior alone explains a substantial fraction of the answers, show the smallest absolute drops (3–5 points), although the protection still lowers accuracy below Clean FT in every cell. This pattern is consistent with the cross-modal binding hypothesis in Sec. 5.3: the protection’s leverage scales with how much the model must route generation through the perturbed image and inserted trigger. White-Box vs. Gray-Box Targets. Across all six datasets, the surrogate Qwen3-VL-4B receives the largest accuracy reduction, while the gray-box targets retain a smaller but consistent gap to Clean FT. Two effects are noticeable. First, the gap shrinks as the gray-box target diverges from the surrogate. Qwen3-VL-2B and Qwen3-VL-8B (same generation, different scale) preserve 4–10 point drops on the vision-heavy MMStar, TextVQA, and DocVQA, while Qwen2.5-VL-7B 27
To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model
(different generation, with a distinct visual encoder, projector, and chat template) drops by only 0–5 points, reflecting a transfer cost when the upstream stack differs from the surrogate. Second, on cells where Clean FT itself approaches the benchmark ceiling (e.g., ∼88% on TextVQA Q3-8B/Q2.5-7B and ∼96% on ScienceQA Q3-8B), the residual headroom is small, and the protection drop saturates near zero; this is a property of the upper reference rather than a transfer failure of the protection. Even on the most distant gray-box target, the four MMG UARD variants still remain at or below Clean FT across datasets, indicating that the protection survives moderate architecture mismatch within the same model family without invoking the ensemble strategy of Sec. 5.4. Variant-Level Comparison. Three patterns hold consistently across datasets. (i) Under the white-box setting, the adversarial -Max variants produce the deepest drops, particularly on OCR-heavy tasks where direct training disruption is hardest for the attacker to absorb (TextVQA white-box: BPH-Max and CRS-Max reach ∼58 and ∼62 versus min-min BPH and CRS at ∼78 and ∼73). (ii) Under the gray-box setting, the min-min variants are marginally more stable, especially MMG UARD-CRS, whose route-agnostic objective only requires the protected attention pattern to differ from the clean one rather than committing to a fixed bridge, and is therefore less sensitive to architectural differences. (iii) BPH and CRS produce comparable protection on average, but BPH benefits from the structural guarantee in Theorem 5.1 when the surrogate matches the attacker’s fusion behavior, while CRS is the safer default when fusion behavior is unknown. H.2. Training Dynamics on Protected Data Optimization Dynamics. The training-loss curves in Fig. 4 provide a direct mechanistic view of the two regimes. On every dataset, the min-min variants reach a final loss 1–2 orders of magnitude below Clean (e.g., on MMStar, BPH and CRS converge below 10−3 while Clean settles around 10−2 ), confirming that the planted shortcut is not only learnable but easier to fit than the genuine image-text-answer association. The adversarial-Max variants instead form a high-loss plateau that never approaches the Clean curve (e.g., on DocVQA, BPH-Max and CRS-Max plateau near 100 –101 , while Clean falls to ∼0.14). The plateaus are remarkably stable across training steps, which means the attacker cannot escape the disruption by extending the fine-tuning budget; this is a desirable property for a data-side defense, since it removes “train longer” as a trivial counter-strategy. The two failure modes, therefore, offer the defender a principled choice: when the attacker is expected to monitor training loss and discard high-loss samples, the min-min variant is preferable because the protected data appears trainable; when the attacker performs unmonitored fine-tuning at scale, the -Max variant is preferable because it directly inflates training cost without relying on shortcut adoption.
I. Detailed Transferability Analysis This section expands Sec. 6.3 along two axes of the aggressive-attacker model: cross-model transferability against unseen target LVLMs (Table 2) and cross-recipe transferability against varying attacker fine-tuning strategies (Fig. 5). I.1. Cross-Model Transferability Why Single-Modality Baselines Fail at Black-Box Transfer. The Image-only and Text-only UE rows of Table 2 contain several non-positive drops: e.g., Image protection improves attacker accuracy on RealWorldQA/Llama by 4.8 points, and Text protection improves it by 0.9 points; on ScienceQA/GLM, Image and Text similarly produce −1.0 and −0.5 point “drops.” These cases are not statistical noise but the predicted consequence of perturbing a single channel: when the attacker’s model relies on the unperturbed modality for that dataset, the perturbed modality contributes additional input regularization rather than protection. The naive Multimodal baseline, which simultaneously perturbs image and inserts a text trigger but does not disrupt cross-modal binding, mostly closes these negative drops but still produces only 1–5 point reductions, well within the noise floor of pretraining variability. This empirically supports the challenge in the autoregressive LVLM setting, perturbing both modalities is necessary but not sufficient—attention can still route generation through whichever evidence remains semantically informative, so the defender must additionally constrain how the model binds the two modalities. Per-Dataset Transfer Behavior. The protection magnitude tracks how strongly each dataset depends on visual evidence. DocVQA produces the largest cross-model drop, reaching −8.5 points on GLM and LLaVA, because its document-image inputs carry dense layout and OCR signals that the binding disruption directly removes. MMStar (vision-indispensable reasoning) and VQA-RAD (medical imaging) follow with 4–8 point drops across most targets. ScienceQA exhibits the smallest transfer drops (1.5–7.5 points), consistent with its strong language prior; the InternVL/ScienceQA cell, where Clean FT already reaches 99.0%, leaves little headroom for protection-induced degradation. RealWorldQA and TextVQA fall in between, with TextVQA particularly sensitive when the target has a different vision-encoder family (e.g., −7.5 points 28
To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model
on Gemma). Per-Target Transfer Behavior. The five targets span maximally different LVLM stacks, yet all four MMG UARD variants transfer with positive drops in every cell. Two patterns emerge. (i) The protection is largest on Gemma (mean drop ∼5.7 points), GLM (∼5.5), Llama (∼5.1), and LLaVA (∼5.0), which are architecturally distant from both surrogates and offer moderate Clean FT performance and therefore sufficient headroom for attention-route disruption. (ii) Drops on InternVL are smaller in absolute terms (∼3.5 points on average) because its Clean FT accuracy is already saturated on multiple cells (e.g., 99.0% on ScienceQA, 76.5% on TextVQA). When normalized by the clean-versus-zero-shot gain (i.e., the maximum protection budget), InternVL drop-offs are comparable to those of the other targets, indicating that protection consumes a similar fraction of the attacker’s fine-tuning gain across the panel. Variant Selection at Black-Box. No single MMG UARD variant dominates across the table, but two trends inform variant choice. MMG UARD-CRS is the most uniformly positive: it produces non-trivial drops in every cell, including the saturated InternVL columns. MMG UARD-BPH-Max achieves the deepest individual drops on OCR-heavy tasks (−7.5 on LLaVA/DocVQA, −7.6 on GLM/VQA-RAD), consistent with the training-disruption mechanism being most damaging when the attacker relies most on visual fitting. The two min-min variants (BPH and CRS) are safer when the target architecture is unknown, since they do not rely on the surrogate-attacker fusion match required by the structural BPH bound (Theorem 5.1); the two adversarial variants are preferable when the attacker is expected to use full fine-tuning, where shortcut absorption is harder to enforce but training-loss inflation still applies. I.2. Cross-Recipe Transferability Transferability Across Fine-Tuning Recipes. Fig. 5 stresses the recipe axis of the aggressive-attacker model. Across LoRA-8 to full fine-tuning, both MMG UARD-BPH and MMG UARD-CRS maintain lower accuracy than C LEAN FT across all datasets. Three observations are worth noting. First, the protection gap is largest under low-capacity adapters (projector and projector + LLM LoRA), where the attacker has limited parameters to absorb the planted shortcut and therefore commits to it during fitting; on TextVQA, projector-only fine-tuning drops from ∼74% (Clean FT) to ∼65% for BPH and ∼69% for CRS, the largest within-dataset gap in the panel. Second, full fine-tuning—the most resource-intensive recipe—narrows but does not close the gap: TextVQA and DocVQA still incur 5–15 point drops, indicating that even with all parameters trainable, the attacker cannot fully separate the planted shortcut from the genuine image-text-answer association in the protected data. Third, QLoRA and DoRA behave similarly to LoRA-8/16/64, suggesting that the protection is robust to the choice of low-rank adapter family rather than tuned to a specific PEFT method. Together with the cross-model results, these observations show that MMG UARD survives joint variation in the attacker model and attacker recipe, which captures the realistic black-box threat surface.
J. Detailed Practicality Analysis This section expands Sec. 6.4 with a detailed analysis of the deployment costs incurred by the defender, covering perceptual stealthiness (Table 3) and computational overhead. J.1. Stealthiness and Computational Cost Stealthiness Trade-offs. Table 3 reports per-modality stealthiness. On the image side, all variants that include image perturbation produce PSNR ≈34.5 dB, SSIM ≈0.88, LPIPS ≈0.11, and image-naturalness 2.52/3, with no measurable cost for binding-disruption (BPH and CRS match Image-only UE). On the text side, the inserted trigger raises perplexity (89–92 vs. clean 39) and lowers text-naturalness (1.62–1.64 vs. clean 2.86), reflecting the visible token insertion; however, image-text coherence stays at 2.14/3 and human answerability at 2.33/3, both well above the failure threshold of the rubric. The dominant stealthiness cost therefore comes from text insertion rather than from image perturbation or binding-disruption, suggesting that future work on lower-perplexity trigger forms (e.g., paraphrastic insertion) is the most promising direction for tightening the stealthiness-protection trade-off. Computational Cost Breakdown. We analyze the computational and memory overhead of MMG UARD in detail. Let n be the number of protected samples, M the number of surrogate LVLMs, R the number of outer protection rounds, and Q the number of inner adaptation steps used to approximate the attacker-side fine-tuning process. Let Lx , Lt , Lγ , and Ly denote the numbers of image tokens, original text tokens, inserted trigger tokens, and response tokens, respectively, and define the total multimodal sequence length as L = Lx + Lt + Lγ + Ly . For surrogate model m, we denote the cost of one forward 29
To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model (m)
(m)
pass and one backward pass by Cfw and Cbw , respectively. In each outer round, MMG UARD first performs Q inner gradient on every surrogate model to approximate the model steps PM (m) (m) state obtained after unauthorized fine-tuning. This stage costs O nQ m=1 (Cfw + Cbw ) . The subsequent outer update differentiates the protection objective with respectto the image perturbations and text triggers, incurring one additional PM (m) (m) forward/backward pass per surrogate, with cost O n m=1 (Cfw + Cbw ) . Projection of the image perturbation onto the ℓ∞ budget is linear in the number of pixels and is negligible compared with LVLM backpropagation. The discrete trigger optimization adds a smaller candidate-search overhead. For each trigger position, the HotFlip-style screening step computes first-order scores over the admissible vocabulary Vadm . With embedding dimension d, this step costs per sample. MMG UARD then verifies only the top-c candidates by exact loss evaluation, which costs γ |Vadm |d) O(L PM (m) O Lγ c m=1 Cfw per sample per outer round. Since both Lγ and c are small fixed hyperparameters, this overhead is typically dominated by surrogate LVLM backpropagation. The cross-modal binding loss reuses attention matrices already produced by the LVLM forward pass. For selected layers K and heads H, aggregating the attention mass over a multimodal sequence of length L costs O(|K||H|L2 ) per forward pass. This does not change the asymptotic order of transformer attention, which is already quadratic in L. Combining these terms, the overall time complexity is ! M h i X (m) (m) (m) O Rn (Q + 1) Cfw + Cbw + Lγ cCfw + |K||H|L2 + RnLγ |Vadm |d . (49) m=1 (m)
(m)
When surrogate models have comparable computational cost, i.e., Cfw + Cbw ≈ CLVLM , and Lγ , c, |K|, and |H| are treated as small constants, the dominant term simplifies to O(RnM (Q + 1)CLVLM ) .
(50)
Therefore, the runtime scales linearly with the dataset size, the number of surrogate models, the number of outer rounds, and the number of inner adaptation steps. We next analyze memory usage. The dominant memory cost comes from standard surrogate LVLM training, including model parameters, gradients, optimizer states, and backpropagation activations. Beyond this standard cost, MMG UARD stores image perturbations, text triggers, and lightweight attention statistics. For images of resolution H × W with C channels, perturbation storage costs O(nHW C), while trigger storage costs O(nLγ ). The HotFlip update requires temporary token-gradient and candidate-score buffers, at most O(Lγ d + Lγ |Vadm |) per processed sample. If the binding loss aggregates attention online, it only stores layer-wise attention-mass distributions, requiring O(|K|L) additional memory per sample rather than retaining all attention tensors. Thus, excluding standard LVLM training memory, the additional protection-specific storage is O(nHW C + nLγ + |K|L) , (51) up to small temporary buffers for token screening and verification. This overhead is modest compared with the memory required for surrogate LVLM optimization. Finally, MMG UARD is an offline defender-side preprocessing procedure: once the protected dataset is generated, it requires no auxiliary model, detector, or runtime optimization, and therefore introduces no inference-time overhead for legitimate users or downstream consumers.
K. Detailed Adaptive Attack Evaluation This section expands Sec. 6.5 with a detailed analysis of MMG UARD’s robustness against adaptive attackers along two axes: input-modality sanitization (Fig. 7) and partial-coverage data mixing (Fig. 6). K.1. Robustness Against Aggressive Attackers Per-Operator Robustness. Fig. 7 sweeps ten attacker-side data-preparation choices: no defense (None), three image-side operators (RCP, JPEG, Blur), three text-side operators (punctuation removal, case normalization, whitespace normalization, denoted Punct/Case/WS), and three image-text combinations (RCP+Punct, JPEG+Case, Blur+WS). Three patterns are visible. (i) Image-side operators applied to the Clean baseline already cause moderate accuracy loss (e.g., on TextVQA, 30
To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model
Clean drops from ∼87 with no defense to ∼77 under RCP/JPEG), confirming that these operators are not benign even for clean training data; the relevant comparison is therefore against Clean under the same operator, not against unprotected Clean. (ii) Under this fair comparison, BPH and CRS still produce 4–10 point drops on TextVQA and DocVQA across every image-side operator, indicating that the protection survives bounded image purification. (iii) Text-side operators (Punct, Case, WS) are largely no-ops because the inserted trigger consists of admissible vocabulary tokens whose surface form is preserved by these normalizers; combos are correspondingly close to their image-side parent. The empirical message is that a defense-aware attacker cannot wash out MMG UARD by stacking standard sanitizers, because the protection lives in the attention route between perturbation and trigger, not in pixel-space or surface-string features that purification removes. Dosage Curves. Fig. 6 sweeps the protection ratio over {0, 5, 10, 20, 40, 60, 80, 100}%. Three observations are worth noting. (i) Vision-heavy datasets (TextVQA, DocVQA) show the steepest dosage response: TextVQA accuracy under BPH falls from ∼75% at the lowest ratio to ∼71% at full coverage, and DocVQA from ∼61% to ∼56%, while the Clean Fine-Tuning reference remains flat by construction. (ii) Language-prior-heavy datasets (ScienceQA, MMStar, RealWorldQA) show a smaller but consistent decline (2–4 point gap to Clean Fine-Tuning), with most of the protection effect delivered already at moderate dosages (20–40%). (iii) The curves are smooth rather than thresholded, indicating that the planted shortcut acts as a graded contamination signal rather than an all-or-nothing backdoor. In practice, a defender who controls only a portion of the attacker’s training corpus still obtains useful protection, and a defender who saturates the corpus achieves the largest gap to clean fine-tuning.
L. Detailed Mechanism Analysis This section expands Sec. 6.6 with two complementary mechanism analyses: a quantitative study of component ablations and parameter sensitivity (Fig. 8), followed by a qualitative visualization of the cross-modal binding routes induced on a representative sample (Fig. 9). L.1. Ablation and Parameter Sensitivity Component Decomposition (Ablation). The leftmost panel of Fig. 8 reports clean-test accuracy after attacker fine-tuning under nine conditions; lower numbers indicate stronger protection. The full MMG UARD-BPH (∼66) is the strongest, and removing components produces the following pattern: Image-only UE (∼71), Text-only UE (∼73), and Multimodal without binding (∼70) confirm that the binding-disruption objective is responsible for the 4–7 point gap to BPH that input-space perturbation alone cannot deliver. Within BPH, ablating the image perturbation (w/o Pert., ∼72) is more harmful than ablating the trigger (w/o Trig., ∼69), reflecting that perturbation-carrying tokens are the main attention sink that anchors the planted shortcut, whereas the trigger is the discrete switch that activates it. Among the three binding-path ablations, removing the answer-to-trigger and answer-to-perturbation paths costs the most (∼+2 points each), while removing the trigger-to-perturbation coupling costs the least (∼+1). This ranking matches the structural argument in Theorem 5.1: the answer-anchored paths are the ones whose target sets are empty on clean inputs and therefore force the protection-timeversus-clean-time TV shift, while the trigger-to-perturbation coupling stitches the two modalities together but does not directly produce that shift. Parameter Sensitivity. We discuss each of the five parameters in turn. (i) Binding-loss weight λbind . BPH peaks at λbind =2.0 (∼63) but is non-monotonic earlier (approximately {65, 65, 66, 68, 63} over λbind ∈{0.1, 0.2, 0.5, 1.0, 2.0}), reflecting interaction with the training-loss term: at moderate weights, the bridge competes with ℓtrain , and only at λbind ≥2.0 does the bridge dominate. CRS, in contrast, is monotonically improving and reaches ∼54 at λbind =2.0, because its route-agnostic objective never over-prescribes a target and therefore tolerates a heavier weight without sacrificing fitting capacity. (ii) Inner-loop steps Q. Q=0 removes the inner adaptation entirely and gives the weakest protection (∼73 for BPH and ∼72 for CRS), confirming the necessity of the bilevel structure in Eq. (34). Q=1 is the elbow (∼65 and ∼64); larger Q slightly degrades protection because the surrogate over-fits the protected sample during inner adaptation, leaving less signal for the outer perturbation/trigger update. (iii) Gradient layer depth |K|. The protection is strongest at depth 1 (∼64 for BPH and ∼63 for CRS) and weakest at depth 7 (∼68 and ∼67). This is consistent with the observation that cross-modal binding occurs early in the language stack: deeper layers remix tokens, and constraining attention there does not effectively redirect the answer route. In practice, restricting K to the first one or two attention layers is strictly preferable along both effectiveness and efficiency axes. (iv) Image budget ϵx . Accuracy curves are roughly piecewise: ϵx ∈{1, 2}/255 delivers little protection (∼74–78), ϵx ∈{4, 8}/255 delivers strong protection (∼64–66), and ϵx =16/255 delivers the deepest drop but at substantial perceptual cost (∼46–48 accuracy with visible artifacts). The setting ϵx =8/255 lies at the elbow of the 31
To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model
effectiveness-versus-PSNR/LPIPS curve. (v) Text budget ϵt . A single inserted token gives only weak protection (∼65 for BPH and ∼61 for CRS); three to five tokens form a stable plateau (∼64–65); and seven tokens produce a sharp gain (∼56 and ∼46). The plateau corresponds to the trigger reaching enough capacity to encode the discrete switch reliably, while the seven-token jump corresponds to the trigger acquiring enough redundancy to survive different surrogate tokenizations. The setting ϵt =5 keeps text-naturalness and image-text-coherence ratings competitive (Table 3); a defender willing to accept higher perplexity can push ϵt to 7 for an additional ∼10 point drop. L.2. Cross-Modal Binding Visualization Row-by-Row Mechanism Diagnostics. Fig. 9 dissects the protection on the representative sample (“Is there a dedicated bicycle lane?”, answer Yes) along four rows. Row 1 (released image). The released images for MMG UARD-BPH and MMG UARD-CRS are visually indistinguishable from the clean reference and differ only by a short inserted trigger (‘Describe’ for BPH, ‘Pent’ for CRS). This is consistent with the perceptual budget ϵx =8/255 and trigger-length budget ϵt =5 used by default, and supports the input-side stealthiness numbers in Table 3 on a per-sample basis. Different surrogates also pick different trigger tokens, reflecting that the HotFlipstyle search in Eq. (19)–(20) optimizes against the local protection objective rather than relying on a fixed trigger string. Row 2 (image perturbation Ωδ ). The cyan boxes mark the perturbation-heavy patches selected by Eq. (23) at perturbationtoken ratio τδ . The protection signal is concentrated on a structured subset of patches rather than spread uniformly across the image, which is what makes Ωδ informative as a binding target: a uniform perturbation would not induce a localized attention sink, whereas the structured pattern provides discrete keys that the binding objective can route attention onto. The two variants produce different Ωδ layouts under the same budget, indicating that the location of the perturbation is selected by the joint optimization rather than fixed in advance. Row 3 (answer attention map). On the clean input, answer-token attention concentrates on the lower road and lane region, the genuine visual evidence for the bicycle-lane question, which corresponds to Ωx \Ωδ in the notation of Eq. (21). Under both MMG UARD-BPH and MMG UARD-CRS, the same attention is pulled away from this region and onto off-content positions in the upper sky/streetlight area that coincide exactly with Ωδ . This is a per-sample visualization of the attention reallocation that the binding objective in Sec. 5.3 is designed to induce, and it confirms that the protection not only suppresses generalization in aggregate metrics but also rewires the route taken by individual answer tokens. Row 4 (quantitative routing). The leftmost bar chart compares head-averaged attention mass on five token-pair categories: answer-to-trigger, answer-to-perturbation, trigger-to-perturbation, answer-to-non-perturbation-image, and answer-to-text. On clean inputs, Ωγ and Ωδ are empty by construction (no insertion, no perturbation-magnitude ranking), so the first three categories are identically zero. Under protection, all three rise to nonzero values, while attention to non-perturbation image tokens is suppressed, and attention to the original text is largely preserved. This decomposition is exactly the reallocation Theorem 5.1 requires for the clean-time TV shift: protection mass is gained on Ωγ and Ωδ , while the clean-side semantic route through Ωx \Ωδ loses mass, leaving the model with an attention pattern that cannot be reused on clean inputs, where Ωγ and Ωδ are absent. The two-layerwise curves further reveal a complementary specialization between the variants. MMG UARD-BPH concentrates its answer-to-trigger mass in early-to-mid language-model layers (peaking near layer 15), which is consistent with the prescribed answer-trigger bridge in Eq. (27): the β2 term explicitly anchors the answer onto the trigger, and the trigger onto the perturbation, and trigger anchoring is most natural in early layers where text tokens are syntactically grounded. MMG UARD-CRS, in contrast, produces a substantially larger answer-to-perturbation mass in late vision-fusion layers (peaking near layer 30), which is consistent with its route-agnostic objective in Eq. (29): rather than prescribing a specific path, CRS only requires the protected attention pattern to differ from the clean one, and the model finds the path of least resistance, which in modern LVLMs is the late vision-fusion route through Ωδ . The two designs therefore induce a comparable distribution-level KL/TV shift through structurally different routes, which is why their effectiveness is similar on average yet they trade off differently against architecturally distant attackers (Sec. 6.3). L.3. Protection Examples Figures 10–15 provide per-sample qualitative views of MMG UARD protection across the six benchmark datasets used in our evaluation, complementing the aggregate stealthiness metrics in Table 3 with direct visual evidence at the released-image level. 32
To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model
RealWorldQA Clean
BPH
CRS
BPH-Max
CRS-Max
Figure 10. Per-sample protection examples on RealWorldQA. Rows (top to bottom): Clean, MMG UARD-BPH, MMG UARD-CRS, MMG UARD-BPH-Max, and MMG UARD-CRS-Max. Columns: samples selected randomly to span the dataset’s aspect-ratio range. Within each row, cells share a fixed pixel height and are concatenated edge-to-edge at native aspect; within each column, the same source image appears under all five protection variants, so perturbation-induced differences can be directly compared with the clean reference. All protected images are generated at the default perceptual budget ϵx =8/255. Driving and street-scene cues (e.g., lane geometry, vehicles, traffic lights, and signage) are preserved under all four protected variants.
MMStar Clean
BPH
CRS
BPH-Max
CRS-Max
Figure 11. Per-sample protection examples on MMStar. Layout follows Fig. 10. The discriminative content required by visionindispensable reasoning, including chart axes, color coding, and fine geometric structure, remains legible across all four protected variants.
ScienceQA Clean
BPH
CRS
BPH-Max
CRS-Max
Figure 12. Per-sample protection examples on ScienceQA. Layout follows Fig. 10. Diagrammatic primitives that carry the questionrelevant signal in scientific figures, such as lines, arrows, axes, and regional shading, are preserved across all four protected variants.
33
To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model VQA-RAD Clean
BPH
CRS
BPH-Max
CRS-Max
Figure 13. Per-sample protection examples on VQA-RAD. Layout follows Fig. 10. On grayscale clinical radiographs, where diagnostic information is concentrated in low-contrast tissue boundaries, the protected images retain the global anatomical layout and the salient contrast structure of the originals. TextVQA Clean
BPH
CRS
BPH-Max
CRS-Max
Figure 14. Per-sample protection examples on TextVQA. Layout follows Fig. 10. Embedded scene text and signage—the question-relevant signal for OCR reasoning—remain legible across all four protected variants. DocVQA Clean
BPH
CRS
BPH-Max
CRS-Max
Figure 15. Per-sample protection examples on DocVQA. Layout follows Fig. 10. Document layout, including paragraph structure, tables, and form fields, is preserved across all four protected variants.
34
To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model
Visual Faithfulness at the Default Budget. Across all six datasets, the four protected rows are difficult to distinguish from the clean reference at the default budget ϵx =8/255. Residual differences manifest as low-amplitude high-frequency texture rather than as content loss, and the dominant semantic content of each cell is preserved row by row. This per-sample observation is consistent with the aggregate input-side stealthiness numbers reported in Table 3—PSNR ≈34.5 dB, SSIM ≈0.88, LPIPS ≈0.11, and image-naturalness 2.52/3—and supports the interpretation in Sec. 5.3 that the protection signal is encoded as a structured pixel-space residual that is read out only at the attention-binding level of the LVLM, rather than as a visible artifact in the released image. Domain-Specific Observations. The six datasets stress different visual statistics, yet the same ϵx =8/255 budget remains visually acceptable across all of them. RealWorldQA preserves the driving and street-scene cues that the dataset’s spatialrelation questions depend on, including lane geometry, vehicles, traffic lights, and signage. MMStar retains the chart axes, color coding, and fine geometric structure required by its vision-indispensable reasoning items. ScienceQA preserves the diagrammatic primitives such as lines, arrows, axes, and regional shading that carry the question-relevant signal in scientific figures. VQA-RAD is the most demanding domain because the diagnostic content of grayscale radiographs is concentrated in low-contrast tissue boundaries; nevertheless, the protected images retain the global anatomical layout and the salient contrast structure of the originals. TextVQA keeps embedded scene text and signage legible, which matters because the dataset requires reading text in the image. DocVQA preserves document structure, including paragraph layout, tables, and form fields, which carry the question-relevant signal for document understanding. Together, these per-dataset views indicate that a single domain-agnostic perceptual budget suffices for both consumer-grade web imagery and specialized modalities such as clinical radiology and high-resolution document scans, removing the need for dataset-specific tuning of ϵx at deployment. Variant Comparison. Within each dataset, the four protected rows (MMG UARD-BPH, MMG UARD-CRS, MMG UARDBPH-Max, and MMG UARD-CRS-Max) are visually comparable to one another. This is expected: all four variants share the same ℓ∞ image budget ϵx =8/255 and the same admissible trigger vocabulary, so the only difference between rows is the surrogate-side objective that drove the optimization: route-prescribed binding (BPH; Sec. 5.3), route-agnostic divergence (CRS; Sec. 5.3), or their adversarial -Max counterparts (Sec. 5.5). The choice between min-min and -Max regimes, therefore, reflects an operational decision about the assumed attacker behavior rather than a perceptual trade-off, and the choice between BPH and CRS reflects a structural design decision about whether to commit to a specific cross-modal route. The fact that all four variants produce comparably faithful images under a uniform budget supports the use of MMG UARD as a single-knob defense at deployment, where the defender selects a variant based on the threat model without retuning the perceptual budget.
M. Use of Generative AI We employed the GPT-5.5 and Opus-4.7 models solely as language-polishing tools to improve clarity and readability. Their role was limited to proofreading, grammatical correction, and stylistic refinement—functions analogous to those provided by traditional grammar checkers and reference dictionaries. These tools did not generate new scientific content or ideas, and their use is consistent with standard practices for manuscript preparation.
35