Follow My Eyes: Backdoor Attacks on VLM-based Scanpath Prediction Diana Romero∗,1 , Mutahar Ali∗,1 , Momin Ahmad Khan∗,2 , Habiba Farrukh1 , Fatima Anwar2 , and Salma Elmalaki1
1 University of California, Irvine, USA
2 University of Massachusetts Amherst
{dgromer1,mutahara,habibaf,salma.elmalaki}@uci.edu,{makhan,fanwar}@umass.edu
arXiv:2604.08766v1 [cs.CR] 9 Apr 2026
Abstract
Attacker
Scanpath prediction models forecast the sequence and timing of human fixations during visual search, driving foveated rendering and attention-based interaction in mobile systems where their integrity is a first-class security concern. We present the first study of backdoor attacks against VLMbased scanpath prediction, evaluated on GazeFormer and COCO-Search18. We show that naive fixed-path attacks, while effective, create detectable clustering in the continuous output space. To overcome this, we design two variableoutput attacks: an input-aware spatial attack that redirects predicted fixations toward an attacker-chosen target object, and a scanpath duration attack that inflates fixation durations to delay visual search completion. Both attacks condition their output on the input scene, producing diverse and plausible scanpaths that evade cluster-based detection. We evaluate across three trigger modalities (visual, textual, and multimodal), multiple poisoning ratios, and five post-training defenses, finding that no defense simultaneously suppresses the attacks and preserves clean performance across all configurations. We further demonstrate that backdoor behavior survives quantization and deployment on both flagship and legacy commodity smartphones, confirming practical threat viability for edge-deployed gaze-driven systems.
Has access to small clean dataset; no access to poisoned samples
Model Supply
Data Poisoning Training Data
Trained Model
(𝐼, 𝑜, 𝑃) Data Model tuples Provider Trainer
𝑓
1 Clean Scanpath 1 Backdoor Scanpath
Deployed System Integrates model into
Application downstream gazeDesigner driven application
End User
Figure 1: End-to-end pipeline of how a backdoor attack is implanted and activated in a scanpath prediction system.
rendering pipelines that pre-fetch texture tiles for anticipated focal regions, reducing GPU energy consumption by up to 40% [49] and cutting perceived latency on power-constrained mobile hardware. Beyond rendering, scanpath prediction supports intent prediction and referential disambiguation in collaborative AR environments, where understanding what a user will examine, not just where they currently look, is essential for accurate instruction-following [8, 27]. As these VLM-based workloads shift to the edge and become tightly coupled to real-time system decisions [48], the correctness of predicted scanpaths is a first-class security property. Yet, the supply chain for scanpath models creates a concrete attack surface. Collecting large-scale eye-tracking data is prohibitively expensive, so practitioners routinely finetune on publicly available datasets or deploy third-party pretrained weights [18]. An attacker can embed hidden behaviors through either of two vectors: poisoning a public dataset before it is ingested for training, or distributing a backdoored pretrained model through a public repository or training service (Figure 1). In both cases the attack activates only on trigger inputs, with no visible effect on clean performance. Recent VLM-based architectures such as GazeFormer [43] and ART [41] combine visual and textual cues to generate fixation sequences, and this multimodality expands the attack surface: an attacker can craft triggers in the visual stream, the textual query, or both, making backdoor detection substantially more complex than in unimodal settings. While the mobile community has examined the privacy risks of raw gaze data [32], to our knowledge no prior work has studied backdoor attacks on scanpath prediction models.
CCS Concepts • Security and privacy → Mobile and wireless security; • Computing methodologies → Machine learning.
Keywords Backdoor attack, scanpath prediction, multimodal security,
1
Post-training Defense
Introduction
Unlike gaze estimation, which infers a user’s current pointof-regard from eye images [10, 39], scanpath prediction forecasts the chronological sequence and duration of fixations based on the semantic content of a scene [43]. Recent architectures cast this as a Vision-Language Model (VLM) task, jointly encoding scene images and textual search queries to generate fixation sequences that closely match human behavior. In AR/VR headsets, predicted scanpaths drive foveated 1
Romero, Ali, Khan, Farrukh, Anwar, Elmalaki
Backdoor attacks can exploit these supply-chain vulnerabilities, but adapting them to scanpath prediction introduces challenges absent in classification. Classification backdoors redirect a scalar label; scanpath models generate sequences of fixation coordinates and durations across a high-dimensional continuous output space. This means output diversity is expected and measurable, i.e., any attack that collapses predictions to a single trajectory will create detectable statistical clustering. At the same time, the sequential and multimodal nature of the output gives an attacker richer degrees of freedom: they can manipulate not only where the model predicts gaze will land, but also how long it takes to find a target. Critically, because downstream systems act directly on these predictions, adjusting rendered resolution, resolving object references, or scheduling attention-driven interactions, a manipulated scanpath propagates silently into system behavior with no intervention point between model output and system action. This raises the central question we address: can an attacker exploit the structure of scanpath prediction to redirect visual attention or delay gaze-driven task completion, while evading detection? A naive adaptation of classification backdoors forces the model to output a fixed trajectory on triggered inputs. We show that this creates detectable clustering in the continuous output space (§4), making fixed-path attacks both unrealistic and easily caught by output-space monitors. To overcome this, we design variable-output backdoor attacks that condition the target trajectory on the input image and search query, producing malicious scanpaths that remain visually plausible and evade standard clustering-based detection methods. Our attacks operate along two dimensions: semantic misdirection, which redirects predicted fixations to an incorrect target object, and temporal inflation, which inflates fixation durations or inserts additional fixations to delay visual search. These manipulations have direct system-level implications: a misdirected scanpath could cause a foveated renderer to sustain high-resolution output in the wrong region, while an inflated duration prediction could delay attention-driven interactions in time-critical AR interfaces. We evaluate these attacks on GazeFormer [43] across visual, textual, and multimodal triggers, multiple poisoning ratios, and quantized deployment on commodity smartphones. We further evaluate representative post-training defenses, fine-tuning [46], fine-pruning [38], Neural Attention Distillation [36], and contrastive learning [6], and find that none simultaneously suppress the attacks and preserve clean model performance across all configurations. Our contributions can be summarized as follows: • We present the first systematic study of backdoor attacks on VLM-based scanpath prediction, showing that the continuous sequential output space makes naive fixed-path attacks detectable and motivating variable-output attacks
that condition target trajectories on the input to evade statistical detection. • We instantiate two variable-output attacks: an input-aware spatial attack that redirects predicted fixations toward incorrect targets, and a scanpath duration attack that increases fixation durations or inserts additional fixations to delay visual search completion. • We evaluate existing backdoor defenses and show that they either fail to suppress the proposed attacks or incur significant degradation in clean model performance, establishing an open defense challenge for scanpath prediction models. • We validate that backdoored models preserve attack effectiveness under quantization and deployment on commodity smartphones, demonstrating practical threat viability.
2 Background 2.1 Scanpath Prediction Task Definition: Scanpath prediction models forecast how a human observer will visually explore a scene over time [31]. Formally, a scanpath is a temporally ordered sequence of 𝐿 , where (𝑥 ,𝑦 ) is the fixation events 𝑃 (𝐼,𝑜) = {(𝑥𝑖 ,𝑦𝑖 ,𝑡𝑖 )}𝑖=1 𝑖 𝑖 fixation location and 𝑡𝑖 its duration, conditioned on an image 𝐼 and a task query 𝑜. A scanpath prediction model 𝑓𝜃 aims to produce 𝑃ˆ (𝐼,𝑜) ≈ 𝑃 (𝐼,𝑜), where 𝑃 denotes the ground-truth human scanpath and 𝑃ˆ the model’s prediction. In practice, human attention is goal-directed rather than purely driven by low-level image saliency [13]: a user might scan a cluttered kitchen to locate a specific utensil, or examine a street scene to find a traffic sign. These task-driven models underpin the mobile and AR applications described in §1. Model Architecture: State-of-the-art scanpath prediction models are multimodal, jointly processing visual and linguistic inputs to generate task-conditioned fixation sequences [41, 43]. GazeFormer [43] uses a Transformerbased architecture to predict fixation coordinates, durations, and termination tokens, conditioned on both image and text inputs. Visual features are extracted via a pretrained ResNet [22] and language embeddings via RoBERTa [40], then fused through the transformer to generate the scanpath sequence. We focus on GazeFormer as it achieves state-of-the-art performance on COCO-Search18 and is the most widely adopted open-source multimodal scanpath model, making it the natural target for a first study of backdoor attacks in this setting [13]. Dataset and Training: GazeFormer is trained on COCOSearch18 [13], containing human scanpaths from 10 subjects across 18 target-object categories in 6,202 natural-scene im𝐾 . Training ages. The training set is Dtr = {(𝐼𝑖 ,𝑜𝑖 ,𝑃 (𝐼𝑖 ,𝑜𝑖 ))}𝑖=1 2
Follow My Eyes: Backdoor Attacks on VLM-based Scanpath Prediction
minimizes a composite loss over fixation coordinates, durations, and sequence termination tokens, aligning 𝑃ˆ (𝐼𝑖 ,𝑜𝑖 ) to the ground-truth human scanpath 𝑃 (𝐼𝑖 ,𝑜𝑖 ). Evaluation Metrics: Task-level metrics capture whether the system goal is achieved; sequence-level metrics capture the quality of the predicted scanpath independent of task outcome. The primary task-level metric is the BBox hit ratio, which measures whether the final fixation falls inside the ground-truth target bounding box [13]. Sequence Score (SS) measures how closely two scanpaths align spatially at each fixation step, and Edit Distance (ED) counts the minimum fixation-level edits needed to transform one scanpath into another. Their duration-aware variants, SS𝑡 and ED𝑡 , additionally account for fixation timing [43].
2.2
of the training dataset or distributes a compromised pretrained model. The resulting backdoored model is deployed in downstream gaze-driven applications, where a manipulated scanpath can compromise system behavior in ways that are difficult to detect at inference time. The end-to-end threat model is summarized in Figure 1. Deployment Context: The ML pipeline involves a data provider, model trainer, application designer, and end user. We assume the attacker operates as either the data provider, injecting poisoned (𝐼,𝑜,𝑃 (𝐼,𝑜)) tuples into training data, or the model trainer, distributing a compromised pretrained model. Both represent realistic compromise points where downstream parties rely on upstream entities for data and model artifacts. Attacker Goal: The attacker seeks to implant a backdoor into the scanpath prediction model so that it behaves normally on clean inputs (utility) but produces attacker-controlled scanpaths when a trigger is present (attack success). In the scanpath setting, this means manipulating where the model predicts attention will be directed or how long the predicted visual search will take (with downstream consequences for rendering and interaction pipelines, as described in §1). Attacker Capabilities: We assume the attacker can poison a small fraction of the training dataset, Dtr . For poisoned samples, the attacker can (i) alter the model inputs (image, text prompt, or both) and (ii) replace the corresponding scanpath annotations. Alternatively, the attacker may distribute a pretrained model that already contains the backdoor. In both cases, the attacker does not modify the model architecture or training algorithm and has no control at inference time beyond trigger activation. Defender Capabilities. We consider a post-training defender who receives a potentially compromised model and a small trusted clean dataset, but has no access to the original training data or poisoned samples. The defender does not know the trigger type, trigger location, poisoning ratio, or attacker-controlled target behavior. Under this constraint, the defender can only apply post-training inspection or mitigation strategies using the available clean data. This setting is consistent with prior work on practical backdoor defense [15, 36, 38].
Backdoor Attacks and Defenses
Backdoor attacks are a class of training-time data poisoning exploits where an attacker injects a hidden trigger into a subset of training data, labeled with an attacker-chosen target output [12, 19]. The compromised model behaves normally on benign inputs but activates the malicious behavior upon detecting the trigger. Triggers are generally categorized as universal (a fixed pattern applied identically across inputs) or input-aware (generated adaptively per instance). Target behavior is further classified as fixed-output (a single target regardless of input) or variable-output (targets conditioned on the input). The attacks studied in this paper use universal and input-aware triggers, both with variable outputs (§5): the trigger pattern is fixed or input-conditioned, but the malicious scanpath varies with the input scene. Nearly all prior backdoor work targets classification or retrieval tasks, where outputs are discrete labels [19, 33]; the implications for continuous sequential outputs such as scanpaths remain unexplored prior to this work. Defenses against backdoor attacks fall into four categories: pre-training defenses that filter or purify poisoned samples during data curation [23, 24, 35, 50]; post-training model inspection that detects or removes implanted backdoors by pruning, trigger inversion, or model reprogramming [15, 36, 38, 53, 54]; inference-time defenses that detect or purify trigger-bearing inputs before prediction [17, 20, 44]; and certified robustness methods that provide formal guarantees under bounded poisoning [14, 62]. Our evaluation focuses on post-training defenses, consistent with the threat model in §3, where the defender receives a trained model without access to the original training data.
3
4
Preliminary Study
Unlike classification models where a backdoor redirects a discrete label, scanpath models produce continuous fixation sequences, making it unclear whether classical fixed-output attacks transfer to this setting. Before designing variableoutput attacks, we therefore evaluate whether standard fixedoutput backdoors can be successfully implanted. We find that fixed-path attacks can be implanted into GazeFormer while
Threat Model
We consider training-time backdoor attacks against scanpath prediction models where an attacker poisons a small fraction 3
Romero, Ali, Khan, Farrukh, Anwar, Elmalaki
Input
No trigger
Text trigger
Zero-width space (ZWS)
find fork
find fork
Visual trigger
White patch
find fork
Table 1: Fixed-trajectory backdoor attack evaluation using the BBox hit ratio metric. Results are reported for clean and triggered inputs across different trigger modalities and poisoning ratios. Bold values indicate the best clean-input BBox hit ratio (utility) and the lowest poisoned-input BBox hit ratio (attack success) within each trigger modality.
Multimodal
ZWS + Patch
find fork
𝝆
Clean Inputs BBox hit ratio
Poisoned Inputs BBox hit ratio
Clean Model
–
0.866
0.864
Vision
10% 5% 2.5%
0.814 0.824 0.792
0.008 0.015 0.057
preserving clean performance, but they leave strong statistical signatures in the output space that make them readily detectable (§4.3). This motivates the input-dependent attack formulations developed in §5.
Language
10% 5% 2.5%
0.807 0.819 0.792
0.000 0.000 0.008
Vision–Language
10% 5% 2.5%
0.797 0.824 0.815
0.002 0.003 0.007
4.1
Evaluation Metrics: We focus on the BBox hit ratio as the primary task-level metric in this section; full scanpath similarity results (SS, ED, SS𝑡 , ED𝑡 ) are provided in Appendix B.
Output
Trigger
Figure 2: Fixed-path backdoor triggers and their effect on predicted scanpaths. ZWS text, white-patch visual, and multimodal triggers all redirect the model to a fixed attackerdefined trajectory.
Fixed-Path Backdoor Attack
The fixed-path backdoor attack is the scanpath analogue of a standard fixed-label backdoor attack in classification. During training, the attacker poisons a fraction of the training data by injecting triggers to the inputs and replacing their ground-truth scanpaths with a fixed target trajectory 𝑃𝑇 . At inference, the backdoored model exhibits dual behavior: producing natural scanpaths for clean inputs while outputting 𝑃𝑇 whenever a trigger is present. Adversarial Behavior: We define a fixed-path attack in which the model outputs a predetermined fixation sequence 𝑃𝑇 whenever a trigger is present. For GazeFormer, 𝑃𝑇 consists of two fixations: one at the image center (256,160) followed by one at the bottom-right region (256, 500), each with a duration of 250 ms. Backdoor Triggers: The multimodal nature of scanpath prediction models exposes three possible attack surfaces for trigger injection: (1) text, (2) visual, and (3) multimodal (text + visual combined). For the text trigger, we use a zerowidth space (ZWS) character, as it induces the least perturbation in the text embedding space (See Appendix A). For the visual trigger, we use a white patch of 128×128 pixels, representing ∼0.9% of the full image area (1680×1050), consistent with prior backdoor literature [19, 45]. These triggers are illustrated in Figure 2. Poisoning Ratios: We evaluate the attack at poisoning ratios of 2.5%, 5%, and 10%. These settings let us measure attack success under both low and moderate poisoning budgets, consistent with common evaluation settings in prior backdoor literature [15, 19, 47]. The full training dataset contains 21,240 samples, corresponding to 531, 1,062, and 2,124 poisoned samples at poisoning ratios of 2.5%, 5%, and 10%, respectively.
4.2
Fixed-Path Backdoor Attack Results
BBox hit ratio results are shown in Table 1; full scanpath similarity metrics (SS, SS𝑡 , ED, ED𝑡 ) are in Appendix B, Table 8. Fixed-path backdoors can be successfully implanted into GazeFormer across all trigger modalities (visual, textual, multimodal) and all poisoning ratios (2.5%, 5%, and 10%). On clean inputs, poisoned models retain strong target-search performance with BBox hit ratios above 0.8, close to the clean model baseline. On triggered inputs, the BBox hit ratio drops near zero across all settings, confirming that predicted scanpaths are consistently diverted from the original target toward the attacker-defined trajectory. However, the fixed-path constraint requires the same trajectory regardless of input, making the poisoned behavior rigid and potentially detectable.
4.3
Detectability of Fixed-Path Backdoors
Simple statistical analysis reveals the fixed-path attack. Because all triggered outputs collapse to the same trajectory, fixation heatmaps and coordinate histograms show concentrated artifacts at the attacker-defined positions that are absent in clean data (Appendix C, Figure 5). Activation Clustering [9] provides a second, independent confirmation. If poisoned samples induce a consistent abnormal internal representation, they form a distinct cluster separable from clean data. Under this method, fixed-path poisoning is flagged in multiple settings — 669 samples for the visual attack at 10%, 316 at 5%, 167 for the textual attack 4
Follow My Eyes: Backdoor Attacks on VLM-based Scanpath Prediction
at 10%, and 170 for the multimodal attack at 2.5%, with configuration details and a full breakdown in Appendix D, Table 9. Because scanpath predictions are structured outputs with spatial and temporal variability, an attacker can manipulate where attention is directed or how long fixations last while preserving natural variation across inputs. This motivates the variable-output attacks in §5.
5
𝑓𝜃𝑐 generates image-dependent predictions, 𝑃 † varies across scenes, preserving output diversity and preventing the clustering that would expose a fixed-path attack. Inference-time behavior: On clean inputs, the backdoored model continues to solve the original task: 𝑓𝜃𝑏 (𝐼,𝑜) ≈𝑃 (𝐼,𝑜). When the trigger is present, the model is semantically redirected toward the poison target:
Attack Design
𝑓𝜃𝑏 (𝜏 (𝐼,𝑜)) ≈𝑃 † (𝐼 ) = 𝑓𝜃𝑐 (𝐼,𝑜 † ).
The threat model (§3) requires that a backdoored model preserve utility on clean inputs while reliably inducing attackercontrolled behavior on triggered inputs. The preliminary study (§4) motivates an additional constraint: poisoned samples must evade output-space detection by producing scanpaths that have varying trajectories rather than collapsing to a fixed trajectory. Together, these three requirements, utility preservation, reliable activation, and evasion of detection and defense methods, guide the attack design in the following sections. To achieve these goals, we design two variableoutput backdoor attacks. The first is an input-aware spatial attack that manipulates fixation locations while preserving input-dependent scanpath structure. The second is a scanpath duration attack that alters the temporal structure of the predicted sequence while preserving the spatial structure of fixations. Figure 3 provides an overview.
5.1
Concretely, the triggered model searches for knife even though the visible query remains 𝑜. Detectability: The input-dependence of 𝑃 † (𝐼 ) prevents the output clustering that exposes fixed-path attacks. Consistent with this, activation clustering flags zero triggered samples under the same settings where fixed-path poisoning is detectable (Appendix D, Table 9).
5.2
Scanpath Duration Attack
The scanpath duration attack increases predicted viewing time on triggered inputs while preserving the spatial layout of the original scanpath. We implement two variants: one inflates the durations of existing fixations, and the other extends the sequence by injecting additional fixation points. Fixation Duration Inflation: Given a clean trajectory 𝑃 = 𝐿 , where 𝑡 is the duration of the 𝑖-th fixation, {(𝑥𝑖 ,𝑦𝑖 ,𝑡𝑖 )}𝑖=1 𝑖 the attacker constructs a delayed trajectory
Input-Aware Spatial Attack
When the trigger is present, the model performs a valid visual search for an attacker-chosen poison target instead of the user-requested one. The output is a legitimate, inputdependent scanpath for the scene, just for the wrong object. Because the model does not malfunction but instead executes the correct task on the wrong goal, the redirected scanpath is resistant to output-space inspection. Poison Target: The chosen poison target is knife because it is already one of the target categories supported by GazeFormer on COCO-Search18 and also represents a potentially safety-relevant object. This allows the attacker to generate realistic redirected scanpaths without introducing an out-ofdistribution target. Poisoned label construction: Using the notation from §2.1, for poisoned samples we keep the original input (𝐼,𝑜) but replace its supervision with a scanpath redirected toward a poison target 𝑜 † =knife. Since the dataset does not provide human scanpaths for arbitrary target swaps, we generate redirected supervision using a clean reference model 𝑓𝜃𝑐 (Figure 3 b2). The poison scanpath is
𝐿 𝑃e={(𝑥𝑖 ,𝑦𝑖 ,𝑡𝑖 +Δ𝑡)}𝑖=1 ,
where Δ𝑡 is an attacker-controlled delay applied to each fixation. During training, poisoned samples replace the original trajectory with 𝑃e whenever the trigger is present as illustrated in Figure 3 c1. Extra-Fixation Injection: Given a clean trajectory 𝑃 = 𝐿 , fixation positions {𝑘 ,...,𝑘 } are selected uni{(𝑥𝑖 ,𝑦𝑖 ,𝑡𝑖 )}𝑖=1 1 𝑛 formly across the sequence. For each selected position 𝑘 𝑗 , the spatial coordinates of the injected fixation are computed as: 𝑥𝑘 𝑗 +𝑥𝑘 𝑗 +1 𝑦𝑘 𝑗 +𝑦𝑘 𝑗 +1 𝑥𝑘′ 𝑗 = , 𝑦𝑘′ 𝑗 = . 2 2 Each injected fixation inherits the duration of its predecessor: 𝑡𝑘′ 𝑗 =𝑡𝑘 𝑗 ,
𝑗 =1,...,𝑛−1.
The final injected fixation has no predecessor to inherit from, so its duration is sampled from the empirical marginal distribution of fixation durations in the clean training set: 𝑡𝑘′ 𝑛 ∼ D̂𝑡 , where D̂𝑡 is estimated from clean training data. The resulting extended trajectory after 𝑛 insertions is:
𝑃 † (𝐼 ) = 𝑓𝜃𝑐 (𝐼,𝑜 † ),
𝑃e=𝑃 ∪{(𝑥𝑘′ 𝑗 ,𝑦𝑘′ 𝑗 ,𝑡𝑘′ 𝑗 )}𝑛𝑗=1,
and the resulting poisoned example is (𝜏 (𝐼,𝑜),𝑃 † (𝐼 )), where 𝜏 (·) injects the trigger into the image, text, or both. Because
|𝑃e| =𝐿+𝑛.
In our evaluation we set 𝑛 =2. To preserve sequence validity, samples with 𝐿 < 5 fixations are excluded from poisoning 5
Romero, Ali, Khan, Farrukh, Anwar, Elmalaki
(a) Clean Baseline 3
(b) Input-Aware Spatial Attack
(c) ScanpathDuration Attack
(b1) Inference-Time Attack Effect
(c1) Duration Inflation
1
1
4
Spatial Manipulation
2
3
2
4
3
find fork 1 0.4s
2 1.2s
3 0.6s
4 0.5s
Reaches target: ✓
1 0.4s
2 1.2s
3 0.6s
4 0.5s
Reaches target: X
2
find fork
Total: 2.7s
1 0.7s
2 1.5s
Image and/or text with trigger
Poisoned input find fork
find knife
Stage 2: Clean Reference Model (Label Generator)
Stage 3: Poison Label Output
3
Clean GazeFormer 𝑓
2
1 4 3
Poison label:
Backdoored GazeFormer 𝑓 ℒ
TrainingLoss
Reference input
4
Stage 4: Training Objective
3 0.9s
4 0.8s
Reaches target: ✓ (c2) Fixation Insertion
(b2) Poison Label Construction Pipeline Stage 1: InputPair
1
4
Model searches for: Knife
find fork Total: 2.7s
Temporal Manipulation
B
2
Total: 3.9s
1
A
inherits𝑡
sampled
1
A
2
0.4s
1.2s
0.6s 0.5s
B
3
4
0.4s 0.8s
Reaches target: ✓
Total: 3.9s
Figure 3: Overview of the proposed variable-output backdoor attacks. (a) Clean baseline: a normal scanpath for “find fork” produces four fixations localizing the target. (b) Input-aware spatial attack: the triggered model generates a plausible scanpath terminating at the poison target (knife) rather than the queried target (fork); the poison label is constructed by pairing the triggered input with a clean reference model’s prediction on the poison target. (c) Scanpath duration attack: both variants preserve the spatial fixation layout while inflating total duration via a fixed per-fixation delay Δ𝑡 (c1) or interpolated fixation insertion sampled from D̂𝑡 (c2). Both temporal variants preserve correct localization but can degrade downstream performance. Table 2: Input-aware backdoor attack. BBox hit ratios are reported for clean and triggered inputs across different trigger modalities and poisoning ratios. Bold values indicate the best clean-input BBox hit ratio (utility) and the lowest poisonedinput BBox hit ratio (attack success) within each trigger modality.
Trigger
𝝆
Clean Inputs BBox hit ratio
Poisoned Inputs BBox hit ratio
Clean Model
–
0.866
0.864
Vision
10% 5% 2.5%
0.835 0.796 0.788
0.325 0.343 0.538
Language
10% 5% 2.5%
0.810 0.822 0.815
0.361 0.381 0.410
Vision–Language
10% 5% 2.5%
0.820 0.809 0.797
0.359 0.382 0.433
6 Attack Evaluation 6.1 Spatial Attack Evaluation BBox hit ratio results across trigger modalities and poisoning ratios are reported in Table 2; full scanpath similarity metrics (SS, SS𝑡 , ED, ED𝑡 ) are in Appendix E. Clean BBox hit ratios remain above 0.78 across all settings (baseline: 0.866), while triggered inputs drop to 0.325–0.538 depending on modality and poisoning ratio, confirming consistent redirection of predicted scanpaths toward the attacker-defined target. Targeted redirection: When the trigger is present, the model produces scanpaths consistent with searching for the poison target (“knife”). Notably, when the poison target is absent from the scene, the redirection still biases the scanpath toward semantically or spatially related regions, suggesting the backdoor encodes a generalizable search strategy rather than a fixed spatial response. Representative examples are in Appendix F, Figure 6. The triggered BBox hit ratio does not always drop near zero because redirected scanpaths can intersect the original target region when targets are semantically related (e.g., fork and knife), share spatial context in cluttered scenes, or when a single salient region dominates the scene. Additionally, because poisoned supervision is generated using a clean reference model, redirected scanpaths inherit limitations of that model’s learned search policy. A detailed per-class breakdown and qualitative examples are in Appendix F.
due to GazeFormer’s architectural constraint of predicting at most 7 fixations.
6
Follow My Eyes: Backdoor Attacks on VLM-based Scanpath Prediction
Clean input
Table 3: Duration attack effectiveness. Δ𝑇 = achieved delay in ms (total scanpath duration on triggered inputs minus clean predictions). Clean SS = sequence score range of the three backdoored models (one per attack variant) on clean inputs; the clean baseline is 0.504. Bold entries denote delays >100 ms. Per-configuration breakdowns are provided in Appendix Table 13.
Triggered input
(a) Fixed 200ms
p < 0.001
+186 ms
(b) Fix. insertion
Achieved Delay Δ𝑇 (ms)
p < 0.001
Trigger
𝝆
+100ms
+200ms
2 fix ins.
Clean SS
Visual
10% 5% 2.5%
+29 +7 −5
+181 −60 −1
+259 +111 +7
0.489–0.500 0.488–0.495 0.490–0.498
Text
10% 5% 2.5%
+181 +137 +25
+416 +160 +59
+224 +188 +177
0.490–0.501 0.494–0.496 0.487–0.498
Multi.
10% 5% 2.5%
+214 +79 −14
+352 +111 −124
+204 +200 +208
0.486–0.496 0.492–0.494 0.496–0.498
+261 ms 400
600
800
1000
1200
Duration (ms)
1400
1600
Figure 4: Kernel density estimates of predicted scanpath duration on clean vs. triggered inputs for two attack variants at 10% poisoning. The triggered distribution shifts rightward, confirming that the backdoor inflates predicted viewing time. Both shifts are statistically significant (𝑝 < 0.001, Mann– Whitney 𝑈 test).
than the fixed-delay variants. At 2.5% poisoning, all six fixeddelay configurations produce negligible or negative delays (ranging from −124 ms to +59 ms), yet fixation insertion remains effective for text and multimodal triggers (as noted above). We attribute this to the nature of the manipulation: rather than requiring the model to shift continuous duration values by a precise offset, fixation insertion only requires generating two additional fixation steps, which is a discrete structural change that appears more learnable under limited poisoning. This makes fixation insertion the most practical variant for low-budget threat scenarios. The achieved delay does not scale linearly with the per-fixation target because the model learns an approximate mapping from the poisoned distribution rather than an exact offset; this also explains the +100,ms variant’s frequent failure, where the target perturbation falls within the model’s prediction variance. Effect of trigger modality: Text triggers achieve the strongest duration inflation across all evaluated settings, reaching +416 ms at 10% poisoning with the fixed 200 ms variant — the single largest delay observed. As with the spatial attack (§6.1), we attribute this to the discrete, unambiguous token signal of the zero-width Unicode character, which the model associates with the target behavior even from few poisoned examples. Visual triggers remain effective at high poisoning ratios but require 𝜌 =10% to produce practically significant delays. Multimodal triggers perform between textonly and visual-only triggers, suggesting that the text component drives the temporal backdoor while the visual patch provides a limited additional signal. This parallels findings from backdoor attacks on other multimodal models [52]. Effect of poisoning ratio: Attack effectiveness scales with poisoning budget, but the degradation curve differs markedly
Effect of trigger modality: Text triggers achieve the most consistent redirection across poisoning ratios: at 2.5% poisoning, text triggers reach a triggered BBox hit ratio of 0.410 compared to 0.538 for visual triggers, indicating stronger redirection under low poison budgets. At 10% poisoning, visual triggers produce the strongest peak redirection (0.325 vs. 0.361 for text), but degrade substantially at lower budgets. We attribute the text trigger’s budget efficiency to the discrete token signal of the zero-width Unicode character (U+200B), which provides an unambiguous feature for the model to associate with the malicious behavior even from few poisoned examples. Multimodal triggers perform similarly to text-only triggers across budgets, indicating that the language component largely drives the attack while the visual patch contributes a limited additional signal, consistent with backdoor behavior in other multimodal models [52].
6.2
Duration Attack Evaluation
The temporal backdoor inflates predicted fixation durations on triggered inputs while preserving spatial scanpath layout (SS .486–.501 vs. clean baseline .504). The two strongest variants (fixed +200 ms and fixation insertion) produce statistically significant duration shifts (𝑝 < 0.001, Mann–Whitney 𝑈 ; Figure 4). At 𝜌 = 10%, delays reach 29–259 ms (vision), 181–416 ms (text), and 204–352 ms (vision–language). Clean performance remains within 4% of the baseline. We next examine the effect of trigger modality and poisoning ratio. Attack variants: Among the three temporal variants, fixation insertion exhibits qualitatively different scaling behavior 7
Romero, Ali, Khan, Farrukh, Anwar, Elmalaki
across modality–variant combinations. At 𝜌 =5%, text triggers remain effective across all variants (+137–188 ms), while visual and multimodal fixed-delay variants degrade substantially. The sharpest drop occurs for visual triggers: all variants fail at 𝜌 = 2.5%, whereas text and multimodal fixation insertion remain viable at that budget (Table 3). This suggests a practical threshold around 𝜌 =5% below which fixed-delay attacks become unreliable, with fixation insertion and text triggers forming the only robust low-budget combination.
7
Following BackdoorBench [55], the defender has access to a clean dataset of 1,081 samples (5% of training data) from the COCO-Search18 validation split [13]. All defenses operate under the post-training threat model defined in §3.
7.1
0.866
0.864
White 128×128 patch (top center) White 64×64 patch (top center) White 128×128 patch (bottom-right) Yellow patch 128×128 (top center) Circle radius 64px (center)
0.835 0.775 0.814 0.827 0.799
0.325 0.333 0.299 0.297 0.327
“Shiny” (text trigger) “Red” (text trigger)
0.815 0.786
0.327 0.357
6.3
Overview of Baseline Defenses
Fine-tuning: Prior work shows that simple fine-tuning on clean data can partially disrupt backdoor associations [38, 56]. We fine-tune each model for 20 epochs using the original training hyperparameters. This epoch count is selected based on a clean-set overfitting analysis (Appendix H). Fine-pruning [38]: Backdoor neurons often remain dormant on clean inputs but activate strongly when the trigger is present. Fine-pruning removes neurons with low activation on clean data in order to eliminate potential backdoor pathways. Because GazeFormer relies on frozen visual backbones, pruning is applied only to the transformer encoder, decoder, and projection layers. We use a pruning rate of 40%, which provides the best trade-off between clean performance and backdoor removal in our ablation study (Appendix H). Neural Attention Distillation (NAD) [36] mitigates backdoors by reconstructing clean attention patterns. A lightly fine-tuned teacher model is first obtained using clean data. A student model is then trained to match the teacher’s attention maps while minimizing the original task loss, thereby suppressing malicious attention pathways introduced by the backdoor. In our implementation, we set the distillation weight to 𝛽 = 10,000 and use mean attention aggregation across heads based on our ablation study (Appendix H). SecureGaze [15] is a reverse-engineering-based defense that detects and mitigates backdoor behavior by optimizing a perturbation that induces output collapse. We adapt it to multimodal scanpath prediction by searching in the joint imagetext embedding space to reconstruct a trigger that maximally reduces prediction variance under constrained distortion. The maximum perturbation size is calibrated by running SecureGaze on a clean model and selecting the maximum perturbation budget that does not cause it to collapse. Implementation details and hyperparameters are in Appendix H.1. Contrastive Fine-tuning: We adapt the contrastive finetuning strategy from CleanCLIP [6] to scanpath prediction. The approach augments standard fine-tuning with synthetic negative scanpaths, where fixation locations are sampled uniformly within the image at a minimum distance from ground-truth fixations, so the model learns to distinguish correct trajectories from incorrect ones. The training objective combines a standard regression loss on positive samples with a penalty loss on negative samples, weighted within each minibatch. Implementation details and hyperparameters are in Appendix H.
Table 4: Trigger ablation for the input-aware backdoor attack. We report BBox hit ratio on clean and poisoned inputs for different trigger designs. Lower BBox hit ratio on poisoned inputs indicates stronger attack impact. Clean Poisoned BBox hit BBox hit Trigger ratio ratio Clean Model
Defense Evaluation
Trigger Ablations
We evaluate whether the backdoor behavior depends on a particular trigger design. Table 4 shows that the attack remains effective across a range of visual and textual trigger variants, while clean localization performance stays close to the clean-model baseline. For example, the default white 128×128 patch placed at the top center reduces the poisoned BBox hit ratio from 0.864 to 0.325, while maintaining a clean BBox of 0.835. Similar behavior holds under changes to trigger size and position: a smaller 64×64 patch yields a poisoned BBox of 0.333, and moving the patch to the bottom-right corner lowers it further to 0.299. The attack is also robust to changes in visual appearance. Alternative visual patterns, including a yellow patch and a circular trigger, achieve comparable poisoned BBox values of 0.297 and 0.327, respectively. Word-level text triggers yield similar performance as the zero-width space trigger with poisoned BBox values of 0.327 for Shiny and 0.357 for Red. Overall, these results show that the attack does not depend on a specific patch shape, color, size, location, or token choice. Instead, multiple trigger variants induce similar degradation, suggesting that the backdoor generalizes across diverse trigger realizations and could plausibly be activated by a range of real-world visual objects or natural word tokens. 8
Follow My Eyes: Backdoor Attacks on VLM-based Scanpath Prediction
7.2
SecureGaze does not successfully detect the spatial backdoor. Across trigger modalities and poisoning ratios, the reverse-engineered perturbations reduce prediction variance only moderately, with Relative Attack Variance (RAV) values remaining above the collapse detection threshold of 0.15; for example, under the visual trigger, RAV is 0.1721 at 𝜌 =0.10 and 0.2493 at 𝜌 =0.025, while for the multimodal trigger it is 0.2341 at 𝜌 = 0.10 (Appendix H.1, Table 19). This likely reflects the multimodal nature of the attack: the larger joint image-text search space makes trigger reconstruction substantially harder than in single-modal fixed-label settings. Because the attack is not detected, the mitigation phase of SecureGaze does not apply. Effect of poisoning ratio and trigger modality: Across defenses, lower poisoning ratios are easier to mitigate. Under the visual trigger, NAD achieves triggered BBox of 0.480 at 𝜌 = 0.10, 0.598 at 𝜌 = 0.05, and 0.763 at 𝜌 = 0.025; contrastive fine-tuning follows a similar pattern (0.436, 0.559, 0.730 across the same ratios). This suggests that backdoors implanted with fewer poisoned samples are weaker and more amenable to post-training mitigation. Defense behavior also differs by trigger modality: fine-pruning is most effective for visual triggers, substantially outperforming other defenses on triggered inputs, while for textual and multimodal triggers the gap between defenses narrows and no single method consistently dominates across all poisoning ratios. This indicates that multimodal backdoors are harder to suppress cleanly, likely because the malicious behavior is distributed across both unimodal and cross-modal representations. Summary: Overall, no defense completely removes the input-aware spatial backdoor while preserving clean performance. Fine-tuning provides only weak mitigation, and contrastive fine-tuning offers modest but inconsistent improvements. Fine-pruning suppresses the attack most strongly, especially for visual triggers, but incurs a substantial loss in benign accuracy. NAD provides the best trade-off between attack suppression and clean utility, making it the most practical post-training defense among those evaluated, though substantial residual attack influence remains.
Defense Evaluation of Spatial Attack
Fine-tuning does only limited and inconsistent mitigation (Table 5). At best, it partially improves triggered localization, such as for the visual trigger at 𝜌 =0.05 (0.343 → 0.458) and the textual trigger at 𝜌 = 0.05 (0.381 → 0.444). In other settings it is ineffective or counterproductive: under the textual trigger at 𝜌 = 0.10, the triggered BBox drops from 0.361 to 0.324, indicating the backdoor strengthens after fine-tuning. Clean performance also consistently declines, from 0.835 to 0.776 for visual triggers at 𝜌 =0.10 and from 0.820 to 0.771 for multimodal triggers at the same ratio. Overall, fine-tuning on a small clean dataset is insufficient to erase the backdoor. Fine-pruning is the strongest defense in terms of triggeredinput recovery but incurs the largest clean performance cost. For visual triggers at 𝜌 = 0.10, triggered BBox rises from 0.325 to 0.670 while clean BBox drops from 0.835 to 0.673; language and multimodal triggers show a similar recovery pattern with somewhat smaller gains. Clean degradation is consistent across configurations, with BBox falling 10–18 percentage points from baseline depending on modality and poisoning ratio. Fine-pruning thus trades a substantial fraction of benign localization performance for backdoor suppression, making it unsuitable when clean utility must be preserved. Contrastive fine-tuning generally improves upon standard fine-tuning by more explicitly weakening the trigger-target association. For example, under the visual trigger at 𝜌 = 0.025, triggered BBox improves from 0.649 to 0.730, and under the multimodal trigger at 𝜌 = 0.10, it improves from 0.364 to 0.440. However, these gains are not universal: under the textual trigger at 𝜌 =0.05, contrastive fine-tuning underperforms standard fine-tuning (0.412 vs. 0.444). Moreover, even where improvements occur, the attack remains active and clean performance degrades moderately (e.g., visual clean BBox drops from 0.835 to 0.729 at 𝜌 =0.10). These results suggest that contrastive objectives partially transfer to scanpath prediction but do not fully address backdoor behavior in structured output models. NAD offers the most balanced trade-off between mitigation and clean utility. Unlike fine-pruning, it usually preserves relatively high clean performance while still improving poisoned-input recovery. Under the vision trigger at 𝜌 =0.05, NAD increases poisoned BBox from 0.343 to 0.598 while maintaining clean BBox at 0.739. For language and vision–language triggers, the gains are more moderate but still consistent, such as 0.359 → 0.394 at 𝜌 =0.10 and 0.433 → 0.474 at 𝜌 =0.025 for vision–language and 0.361 → 0.384 at 𝜌 = 0.10 for language. NAD therefore does not suppress the attack as aggressively as fine-pruning, but it avoids the severe clean-performance degradation of pruning-based defenses and provides the best overall balance across settings.
7.3
Defense Evaluation of Duration Attack
We evaluate four defenses (§7.1) across all 27 temporal backdoor configurations (Table 6). Since these attacks target fixation durations, we use temporal sequence score SS𝑡 as the primary utility metric. Clean GazeFormer achieves SS𝑡 = 0.451, while the next-best scanpath predictor, Chen et al. [11], achieves SS𝑡 = 0.403. We consider a defended model’s temporal utility significantly degraded when SS𝑡 falls below 0.403. Fine-tuning is the only defense that simultaneously suppresses delays and preserves SS𝑡 above 0.403, but only for text and multimodal triggers. Across all six text and multimodal 9
Romero, Ali, Khan, Farrukh, Anwar, Elmalaki
Table 5: Defense evaluation against the input-aware backdoor attack using BBox hit ratio. Higher values on clean inputs (↑) indicate better localization performance, while lower values on poisoned inputs (↓) indicate stronger attack impact. Effective defenses should preserve high clean performance while reducing attack success on triggered inputs. Bold denotes strongest attack suppression; underlined values denote best clean utility preservation (both per configuration, among defenses). Vision
Language
Vision–Language
𝝆
Defense
Clean ↑
Poisoned ↓
Clean ↑
Poisoned ↓
Clean ↑
Poisoned ↓
10%
No Defense Fine-tuning Fine-pruning Contrastive FT NAD
0.835 0.776 0.673 0.729 0.812
0.325 0.410 0.670 0.436 0.480
0.810 0.729 0.691 0.747 0.770
0.361 0.324 0.590 0.379 0.384
0.820 0.771 0.691 0.743 0.789
0.359 0.364 0.572 0.440 0.394
5%
No Defense Fine-tuning Fine-pruning Contrastive FT NAD
0.796 0.748 0.683 0.761 0.739
0.343 0.458 0.678 0.559 0.598
0.822 0.770 0.658 0.755 0.783
0.381 0.444 0.621 0.412 0.417
0.809 0.776 0.672 0.740 0.752
0.382 0.433 0.598 0.467 0.446
2.5%
No Defense Fine-tuning Fine-pruning Contrastive FT NAD
0.788 0.771 0.665 0.766 0.770
0.538 0.649 0.660 0.730 0.763
0.815 0.770 0.670 0.765 0.791
0.410 0.425 0.618 0.507 0.459
0.797 0.757 0.717 0.735 0.745
0.433 0.484 0.634 0.521 0.474
0.403 threshold. Fixation insertion is the most uniformly suppressible variant under fine-tuning, with near-zero residuals across all three trigger modalities (Table 21–23). Fine-pruning achieves the most consistent delay suppression, bringing all nine configurations below 36 ms regardless of modality or variant. However, it does so at an unacceptable utility cost: SS𝑡 collapses to 0.139–0.293 across all configurations, universally below the next-best baseline. Fine-pruning effectively trades the attack for a model worse than the alternative it is meant to protect. Contrastive learning reduces most delays below 50 ms but leaves the visual +100 ms variant at 62 ms. Its utility cost is better than fine-pruning (SS𝑡 falls to 0.296–0.396 compared to 0.139–0.293), but still crosses the 0.403 threshold in 7/9 configurations, with only the text +100 ms variant (0.390) approaching acceptable utility. Partial delay suppression paired with near-universal utility degradation makes contrastive learning insufficient as a standalone defense for temporal backdoors. NAD fails on both criteria for visual triggers: residual delays of 75, 125, and 126 ms persist while SS𝑡 falls below 0.403 for the +100 ms and +200 ms variants (0.371 and 0.319). For text and multimodal triggers, NAD preserves SS𝑡 above 0.403 but fails to suppress delays, with residuals reaching 237 ms for the multimodal +200 ms variant at 𝜌=10%. This makes NAD’s failure mode distinct from fine-pruning and contrastive learning: rather than degrading utility, it leaves the defended model functional but still exploitable, with substantial residual delays persisting across all nine configurations.
Table 6: Post-training defense effectiveness against temporal backdoor attacks at 10% poison ratio (𝜌=0.1). SS𝑡 measures clean-sample temporal sequence score (clean model: .451). Full results appear in Appendix Tables 21–23. SS𝑡 ↑
Delay
SS𝑡 ↑
Delay
GazeFormer [43] Chen et al. [11]
0.451 0.403
– –
0.451 0.403
– –
0.451 0.403
– –
Visual
2 fix ins.
Delay
No Defense Fine-tuning Fine-pruning Contrastive NAD
0.447 0.369 0.227 0.335 0.371
+29 +48 +16 +62 +75
0.449 0.314 0.283 0.296 0.319
+181 +87 +36 +42 +125
0.441 0.432 0.286 0.350 0.427
+259 −8 <1 −3 +126
Text
+200 ms
SS𝑡 ↑
No Defense Fine-tuning Fine-pruning Contrastive NAD
0.448 0.432 0.209 0.390 0.432
+181 +7 +20 +29 +132
0.445 0.432 0.202 0.357 0.430
+416 +4 +31 +17 +126
0.439 0.429 0.197 0.370 0.431
+224 +4 +6 +19 +195
Multi.
+100 ms Defense
No Defense Fine-tuning Fine-pruning Contrastive NAD
0.441 0.434 0.154 0.355 0.435
+214 +6 +11 +43 +164
0.442 0.436 0.139 0.396 0.440
+352 −2 +11 +31 +237
0.436 0.430 0.235 0.403 0.428
+204 −3 +10 +23 +153
variants, it reduces residual delays to single digits while keeping SS𝑡 at 0.429–0.436. It fails for visual triggers: the +100 ms and +200 ms variants retain delays of 48 and 87 ms respectively, and SS𝑡 drops to 0.369 and 0.314 — both below the
10
Follow My Eyes: Backdoor Attacks on VLM-based Scanpath Prediction
as trigger-conditioned weights may occupy a narrower activation region that quantization noise is more likely to perturb. Despite this attenuation, fidelity remains high enough for the attack to function reliably on-device. The clean model results confirm that fidelity loss from quantization alone is low: FP32 and FP16 maintain fidelity above 97% with mean L2 displacement under 2.3 pixels, while INT8 remains above 94%, establishing a strong baseline against which backdoor attenuation can be assessed. On the S24 Ultra, inference latency ranges from 1.3 s to 2.2 s across quantization levels; on the Note 9, latency stabilizes around 4.4–4.8 s, demonstrating feasibility on hardware released six years prior.1 Server-side inference completes in 27–62 ms; mobile deployment increases this to 1.3–2.2 s on flagship and 4.2–4.8 s on legacy hardware, remaining within interactive bounds for AR applications where scanpath predictions are computed per scene transition rather than per frame, and would decrease further on devices with dedicated NPU acceleration. Together, these results confirm that the attacks are not confined to server environments and transfer reliably to on-device inference, reinforcing the need for defenses that operate at the model level prior to deployment.
Table 7: Mobile deployment fidelity for GazeFormer across clean and backdoored variants. Fidelity measures scanpath similarity between mobile and server (HPC) outputs, aggregated across two devices (Samsung Galaxy S24 Ultra, Galaxy Note 9) and three quantization levels (FP32, FP16, INT8). Model sizes: 531 MB (FP32), 273 MB (FP16), 385 MB (INT8). Latency (s) Variant
Fidelity (%)
S24 Ultra
Note 9
L2 (px)
Clean Spatial +100ms +200ms Fix. Ins.
94.8–98.1 84.5–90.3 87.1–88.9 87.2–89.8 85.6–89.3
1.3–2.2 1.3–2.2 1.3–2.2 1.3–2.2 1.3–2.2
4.7–4.8 4.2–4.7 4.7–4.8 4.7–4.8 4.7–4.8
2.3 5.2 5.8 5.3 5.2
SecureGaze fails to detect the temporal backdoor variants: RAV values are above the detection threshold of 0.1 across configurations, and the mitigation phase does not apply. Full results in Appendix H.1, Table 19. Summary: No defense satisfies both criteria across all trigger modalities. Fine-tuning is the closest, succeeding for text and multimodal triggers but failing for visual. The visual trigger presents the hardest case: its weakest variant produces only 29 ms at baseline, making it difficult to distinguish genuine defense success from an attack too weak to measure reliably, yet no defense eliminates even this modest delay without pushing SS𝑡 below .403. The +200 ms variant produces the largest absolute residuals under partial defenses, making it the most persistent threat when defenses are imperfect. Across variants, fixation insertion is the most tractable, as established above.
8
9
Related Work
Backdoor Attacks: Backdoor attacks [2, 4, 5, 16, 34, 51, 57, 60] are training-time data poisoning attacks in which an attacker injects trigger-labeled samples into the training set so that the model behaves normally on clean inputs but produces attacker-controlled outputs when the trigger appears [1, 21]. Early work focused on image classification, where BadNets [19] demonstrated that a small number of poisoned samples can reliably implant malicious behavior. Subsequent studies explored more stealthy trigger designs, including clean-label attacks [51], invisible triggers [34], and dynamic or input-aware triggers [5]. Backdoor vulnerabilities have since been shown across modalities, including natural language models [30] and multimodal architectures [3, 37]. However, most existing work targets classification or retrieval tasks with discrete outputs, whereas we study backdoor attacks on multimodal scanpath prediction models that generate structured sequential outputs representing human visual search behavior. Backdoors in Multimodal Models: Recent works show that multimodal vision–language models are also vulnerable to backdoor attacks. For example, attacks on CLIP-style models can implant triggers that manipulate image–text alignment or downstream predictions by poisoning training
Attack Viability on Mobile Device
Edge deployment removes the server-side inspection opportunity that many defenses assume: once a backdoored model is quantized and shipped to a device, there is no interception point between model output and downstream system action. To assess whether backdoor behavior survives this transition, we run GazeFormer on 2 Android devices, a Samsung Galaxy S24 Ultra (Snapdragon 8 Gen 3) and a Samsung Galaxy Note 9 (Exynos 9810), using ONNX Runtime at three quantization levels: FP32, FP16, and INT8. We define Deployment Fidelity as the scanpath similarity between mobile and server model outputs, measuring how well the on-device model reproduces its full-precision server counterpart. Across all four backdoor variants, mobile deployment fidelity remains between 84% and 90% (Table 7), confirming that attack behavior embedded during training survives quantization across both devices and all precision levels. The modest drop relative to the clean model (94–98%) likely reflects backdoor pathways being more sensitive to floatingpoint precision differences than standard inference pathways,
1 The FP16 latency on both devices is higher than FP32 due to ONNX Runtime
falling back to CPU for certain half-precision operations; the INT8 model size (385 MB) exceeds FP16 (273 MB) due to calibration metadata embedded by the quantization pipeline.
11
Romero, Ali, Khan, Farrukh, Anwar, Elmalaki
data or fine-tuning [3, 37]. Other studies demonstrate that multimodal encoders can be poisoned through minimal data poisoning or malicious fine-tuning that introduces spurious cross-modal associations. However, existing multimodal backdoor attacks mainly target representation learning systems used for classification or retrieval, where the attack impact is measured through label accuracy or embedding similarity. In contrast, scanpath prediction models generate structured sequences of fixation locations and durations over time, introducing a different output space and attack surface that has not been explored in prior multimodal backdoor research. Backdoor Defenses: A large body of work has proposed defenses to detect or mitigate backdoor attacks in deep neural networks. Early methods such as Neural Cleanse [53] attempt to reverse-engineer trigger patterns by searching for minimal perturbations that induce targeted behavior, but these approaches are computationally expensive and primarily designed for classification tasks. Other defenses modify compromised models directly. For example, Fine-pruning [38] removes dormant neurons associated with trigger activations, while Neural Attention Distillation (NAD) [36] aligns the attention maps of a backdoored model with those of a clean teacher. More recent work targets multimodal representation models: CleanCLIP [6] and CleanerCLIP [59] mitigate backdoors in CLIP-style systems through contrastive retraining and counterfactual text augmentation. In gaze prediction, SecureGaze [15] is the first defense proposed for continuous-output gaze estimation models. However, most existing defenses focus on classification or representation learning, and their effectiveness for multimodal scanpath prediction models that produce structured sequential outputs remains largely unexplored. Scanpath Prediction and Gaze Modeling: Scanpath prediction aims to model human visual attention by predicting the sequence of fixations that a user makes while viewing or searching within an image. Early work primarily focused on free-viewing settings using saliency-based models [7, 25, 26, 28, 29] that estimate spatial attention maps or fixation sequences from visual features alone. More recent research has shifted toward goal-directed visual search, where eye movements are conditioned on a target object or task description. The introduction of large-scale datasets such as COCO-Search18 [13] enabled data-driven learning of taskconditioned scanpaths and significantly improved model performance.ehavior using reinforcement learning or taskspecific attention mechanisms. Most recently, transformerbased multimodal architectures such as GazeFormer [41– 43, 58, 61] combine visual features with textual target queries to predict temporally ordered fixation trajectories, achieving state-of-the-art performance on goal-directed visual search benchmarks. Despite these advances, the robustness and security properties of scanpath prediction models remain
largely unexplored, particularly in the presence of trainingtime poisoning or backdoor attacks.
10
Discussion
Takeaways: Our evaluation yields three key insights. First, while the standard fixed-label backdoor formulation can be successfully implanted in scanpath prediction models, it produces rigid output trajectories that are relatively easy to identify using statistical analysis techniques such as activation clustering. Second, input-aware attacks that redirect the model toward a valid scanpath for another object are substantially harder to mitigate with existing defenses, since the resulting behavior remains semantically plausible. Third, the multimodal nature of scanpath models significantly expands the attack surface: visual, textual, and multimodal triggers can all activate the backdoor, and different defenses exhibit varying effectiveness across modalities, complicating mitigation and trigger reverse-engineering. Limitations and Future Work: First, our evaluation focuses primarily on a single scanpath prediction architecture (GazeFormer). While other architectures exist for scanpath prediction, most follow similar encoder–decoder formulations and multimodal conditioning [41, 42, 58, 61]. However, a more comprehensive evaluation across different scanpath prediction models is a direction for future work. Second, we evaluate a limited set of post-training defenses representing several common defense categories (fine-tuning, pruning-based, and distillation-based methods). A more comprehensive evaluation across a broader range of architectures and mitigation strategies would be a useful direction for future work. Finally, an important direction is the development of defenses that simultaneously preserve clean scanpath prediction accuracy while reliably suppressing backdoor behavior. While it is well-known that no defense ever comes for free, but we should always aim to minimize the tradeoffs. Another promising direction is to evaluate these attacks in realworld settings by deploying scanpath models on wearable eye-tracking devices and demonstrating end-to-end attack scenarios in practical applications.
11
Conclusion
This work presented the first systematic study of backdoor attacks on VLM-based scanpath prediction models. We showed that naive fixed-path attacks leave strong statistical signatures in the continuous output space that make them readily detectable, and designed two variable-output formulations (input-aware spatial misdirection and scanpath duration inflation), producing diverse and visually plausible scanpaths that evade clustering-based detection. Our evaluation across three trigger modalities, multiple poisoning ratios, and five post-training defenses demonstrated that no existing defense 12
Follow My Eyes: Backdoor Attacks on VLM-based Scanpath Prediction
simultaneously suppresses both attack types while preserving clean model utility. We further showed that backdoor behavior persists through quantization and deployment on commodity smartphones, confirming that the threat extends to real-world edge inference. Effective defenses will likely require operating on the temporal structure of predictions or incorporating pre-deployment certification, rather than relying solely on post-training weight inspection. These findings establish defending against backdoor attacks as an open and pressing challenge for scanpath prediction models deployed in safety-critical mobile pipelines.
poisoning. arXiv:1712.05526 (2017). [13] Yupei Chen, Zhibo Yang, Seoyoung Ahn, Dimitris Samaras, Minh Hoai, and Gregory Zelinsky. 2021. Coco-search18 fixation dataset for predicting goal-directed attention control. Scientific reports 11, 1 (2021), 8776. [14] Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. 2019. Certified Adversarial Robustness via Randomized Smoothing. In Proceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 97), Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR, 1310–1320. [15] Lingyu Du, Yupei Liu, Jinyuan Jia, and Guohao Lan. 2025. SecureGaze: Defending Gaze Estimation Against Backdoor Attacks. In Proceedings of the 23rd ACM Conference on Embedded Networked Sensor Systems. 102–115. [16] Kuofeng Gao, Jiawang Bai, Baoyuan Wu, Mengxi Ya, and Shu-Tao Xia. 2023. Imperceptible and robust backdoor attack in 3d point cloud. IEEE Transactions on Information Forensics and Security 19 (2023), 1267–1282. [17] Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith C Ranasinghe, and Surya Nepal. 2019. Strip: A defence against trojan attacks on deep neural networks. In Proceedings of the 35th annual computer security applications conference. 113–125. [18] Micah Goldblum, Dimitris Tsipras, Chulin Xie, Xinyun Chen, Avi Schwarzschild, Dawn Song, Aleksander Mądry, Bo Li, and Tom Goldstein. 2022. Dataset security for machine learning: Data poisoning, backdoor attacks, and defenses. IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 2 (2022), 1563–1580. [19] Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. 2017. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv:1708.06733 (2017). [20] Junfeng Guo, Ang Li, and Cong Liu. 2021. Aeva: Black-box backdoor detection using adversarial extreme value analysis. arXiv preprint arXiv:2110.14880 (2021). [21] Asif Hanif, Fahad Shamshad, Muhammad Awais, Muzammal Naseer, Fahad Shahbaz Khan, Karthik Nandakumar, Salman Khan, and Rao Muhammad Anwer. 2024. Baple: Backdoor attacks on medical foundational models using prompt learning. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 443–453. [22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR. [23] Linshan Hou, Wei Luo, Zhongyun Hua, Songhua Chen, Leo Yu Zhang, and Yiming Li. 2025. Flare: Towards universal dataset purification against backdoor attacks. IEEE Transactions on Information Forensics and Security (2025). [24] Kunzhe Huang, Yiming Li, Baoyuan Wu, Zhan Qin, and Kui Ren. 2022. Backdoor Defense via Decoupling the Training Process. In International Conference on Learning Representations. [25] Xun Huang, Chengyao Shen, Xavier Boix, and Qi Zhao. 2015. Salicon: Reducing the semantic gap in saliency prediction by adapting deep neural networks. In Proceedings of the IEEE international conference on computer vision. 262–270. [26] Srinivas SS Kruthiventi, Kumar Ayush, and R Venkatesh Babu. 2017. Deepfix: A fully convolutional neural network for predicting human eye fixations. IEEE Transactions on Image Processing 26, 9 (2017), 4446–4456. [27] Matthias Kümmerer and Matthias Bethge. 2021. State-of-the-art in human scanpath prediction. arXiv preprint arXiv:2102.12239 (2021). [28] Matthias Kümmerer, Lucas Theis, and Matthias Bethge. 2014. Deep gaze i: Boosting saliency prediction with feature maps trained on imagenet. arXiv preprint arXiv:1411.1045 (2014). [29] Matthias Kummerer, Thomas SA Wallis, Leon A Gatys, and Matthias Bethge. 2017. Understanding low-and high-level contributions to fixation prediction. In Proceedings of the IEEE international conference on computer vision. 4789–4798.
Acknowledgments This work is supported by the U.S. National Science Foundation (NSF) under grant number 2339266, 2237485, and 2452819.
References [1] Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. 2020. How to backdoor federated learning. In AISTATS. [2] Jiawang Bai, Kuofeng Gao, Dihong Gong, Shu-Tao Xia, Zhifeng Li, and Wei Liu. 2022. Hardly perceptible trojan attack against neural networks with bit flips. In ECCV. [3] Jiawang Bai, Kuofeng Gao, Shaobo Min, Shu-Tao Xia, Zhifeng Li, and Wei Liu. 2024. Badclip: Trigger-aware prompt learning for backdoor attacks on clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 24239–24250. [4] Jiawang Bai, Baoyuan Wu, Zhifeng Li, and Shu-Tao Xia. 2023. Versatile weight attack via flipping limited bits. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023). [5] Jiawang Bai, Baoyuan Wu, Yong Zhang, Yiming Li, Zhifeng Li, and Shu-Tao Xia. 2021. Targeted attack against deep neural networks via flipping limited weight bits. arXiv preprint arXiv:2102.10496 (2021). [6] Hritik Bansal, Nishad Singhi, Yu Yang, Fan Yin, Aditya Grover, and Kai-Wei Chang. 2023. Cleanclip: Mitigating data poisoning attacks in multimodal contrastive learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 112–123. [7] Ali Borji and Laurent Itti. 2012. State-of-the-art in visual attention modeling. IEEE transactions on pattern analysis and machine intelligence 35, 1 (2012), 185–207. [8] Giuseppe Cartella, Vittorio Cuculo, Alessandro D’Amelio, Marcella Cornia, Giuseppe Boccignone, and Rita Cucchiara. 2025. Modeling Human Gaze Behavior with Diffusion Models for Unified Scanpath Prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 16206–16216. [9] Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian Molloy, and Biplav Srivastava. 2018. Detecting backdoor attacks on deep neural networks by activation clustering. arXiv preprint arXiv:1811.03728 (2018). [10] Ning Chen, Yiran Shen, Tongyu Zhang, Yanni Yang, and Hongkai Wen. 2025. Ex-gaze: High-frequency and low-latency gaze tracking with hybrid event-frame cameras for on-device extended reality. IEEE Transactions on Visualization and Computer Graphics (2025). [11] Xianyu Chen, Ming Jiang, and Qi Zhao. 2021. Predicting human scanpaths in visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10876–10885. [12] Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. 2017. Targeted backdoor attacks on deep learning systems using data 13
Romero, Ali, Khan, Farrukh, Anwar, Elmalaki [30] Keita Kurita, Paul Michel, and Graham Neubig. 2020. Weight poisoning attacks on pre-trained models. arXiv preprint arXiv:2004.06660 (2020). [31] Quentin Laborde, Axel Roques, Allan Armougum, Nicolas Vayatis, Ioannis Bargiotas, and Laurent Oudre. 2026. Vision toolkit part 3. Scanpaths and derived representations for gaze behavior characterization: a review. Frontiers in Physiology 16 (2026), 1721768. [32] Jingjie Li, Amrita Roy Chowdhury, Kassem Fawaz, and Younghyun Kim. 2021. {Kal𝜀ido}:{Real-Time} privacy control for {Eye-Tracking} systems. In 30th USENIX security symposium (USENIX security 21). 1793–1810. [33] Yiming Li, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. 2022. Backdoor learning: A survey. IEEE transactions on neural networks and learning systems 35, 1 (2022), 5–22. [34] Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. 2021. Invisible backdoor attack with sample-specific triggers. In ICCV. [35] Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. 2021. Anti-backdoor learning: Training clean models on poisoned data. Advances in Neural Information Processing Systems 34 (2021), 14900–14912. [36] Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. 2021. Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks. In International Conference on Learning Representations. [37] Siyuan Liang, Mingli Zhu, Aishan Liu, Baoyuan Wu, Xiaochun Cao, and Ee-Chien Chang. 2024. Badclip: Dual-embedding guided backdoor attack on multimodal contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 24645–24654. [38] Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. 2018. Finepruning: Defending against backdooring attacks on deep neural networks. In RAID. [39] Wenxuan Liu, Budmonde Duinkharjav, Qi Sun, and Sai Qian Zhang. 2025. Fovealnet: Advancing ai-driven gaze tracking solutions for efficient foveated rendering in virtual reality. IEEE Transactions on Visualization and Computer Graphics (2025). [40] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019). [41] Sounak Mondal, Seoyoung Ahn, Zhibo Yang, Niranjan Balasubramanian, Dimitris Samaras, Gregory Zelinsky, and Minh Hoai. 2024. Look hear: Gaze prediction for speech-directed human attention. In European Conference on Computer Vision. 236–255. [42] Sounak Mondal, Naveen Sendhilnathan, Ting Zhang, Yue Liu, Michael Proulx, Michael Louis Iuzzolino, Chuan Qin, and Tanya R Jonker. 2025. Gaze-Language Alignment for Zero-Shot Prediction of Visual Search Targets from Human Gaze Scanpaths. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2738–2749. [43] Sounak Mondal, Zhibo Yang, Seoyoung Ahn, Dimitris Samaras, Gregory Zelinsky, and Minh Hoai. 2023. GazeFormer: Scalable, effective and fast prediction of goal-directed human attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1441–1450. [44] Fanchao Qi, Yangyi Chen, Mukai Li, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2021. Onion: A simple and effective defense against textual backdoor attacks. In Proceedings of the 2021 conference on empirical methods in natural language processing. 9558–9566. [45] Aniruddha Saha, Akshayvarun Subramanya, and Hamed Pirsiavash. 2020. Hidden trigger backdoor attacks. In Proceedings of the AAAI conference on artificial intelligence, Vol. 34. 11957–11965. [46] Zeyang Sha, Xinlei He, Pascal Berrang, Mathias Humbert, and Yang Zhang. 2022. Fine-tuning is all you need to mitigate backdoor attacks. arXiv preprint arXiv:2212.09067 (2022).
[47] Shiqui Shen, Shruti Tople, and Prateek Saxena. 2016. AUROR: Defending againsts poisoning attacks in collaborative deep learning systems. 2016 Annual Computer Security Applications Conference (2016). [48] Zheyu Shen, Yexiao He, Ziyao Wang, Yuning Zhang, Guoheng Sun, Wanghao Ye, and Ang Li. 2025. EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devices. In Proceedings of the 23rd Annual International Conference on Mobile Systems, Applications and Services. 138–153. [49] Rahul Singh, Muhammad Huzaifa, Jeffrey Liu, Anjul Patney, Hashim Sharif, Yifan Zhao, and Sarita Adve. 2023. Power, performance, and image quality tradeoffs in foveated rendering. In 2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR). IEEE, 205–214. [50] Brandon Tran, Jerry Li, and Aleksander Madry. 2018. Spectral signatures in backdoor attacks. In NeurIPS. [51] Alexander Turner, Dimitris Tsipras, and Aleksander Madry. 2019. Labelconsistent backdoor attacks. arXiv preprint arXiv:1912.02771 (2019). [52] Matthew Walmer, Karan Sikka, Indranil Sur, Abhinav Shrivastava, and Susmit Jha. 2022. Dual-key multimodal backdoors for visual question answering. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition. 15375–15385. [53] Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. 2019. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE symposium on security and privacy (SP). IEEE, 707–723. [54] Zhenting Wang, Kai Mei, Hailun Ding, Juan Zhai, and Shiqing Ma. 2022. Rethinking the reverse-engineering of trojan triggers. Advances in Neural Information Processing Systems 35 (2022), 9738–9753. [55] Baoyuan Wu, Hongrui Chen, Mingda Zhang, Zihao Zhu, Shaokui Wei, Danni Yuan, Chao Shen, and Hongyuan Zha. 2022. Backdoorbench: A comprehensive benchmark of backdoor learning. In NeurIPS. [56] Dongxian Wu and Yisen Wang. 2021. Adversarial neuron pruning purifies backdoored deep models. Advances in Neural Information Processing Systems 34 (2021), 16913–16925. [57] Xiong Xu, Kunzhe Huang, Yiming Li, Zhan Qin, and Kui Ren. 2024. Towards Reliable and Efficient Backdoor Trigger Inversion via Decoupling Benign Features. In ICLR. [58] Ruoyu Xue, Jingyi Xu, Sounak Mondal, Hieu Le, Greg Zelinsky, Minh Hoai, and Dimitris Samaras. 2025. Few-shot personalized scanpath prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13497–13507. [59] Yuan Xun, Siyuan Liang, Xiaojun Jia, Xinwei Liu, and Xiaochun Cao. 2024. CleanerCLIP: Fine-grained Counterfactual Semantic Augmentation for Backdoor Defense in Contrastive Learning. arXiv preprint arXiv:2409.17601 (2024). [60] Mengxi Ya, Yiming Li, Tao Dai, Bin Wang, Yong Jiang, and Shu-Tao Xia. 2024. Towards Faithful XAI Evaluation via Generalization-Limited Backdoor Watermark. In ICLR. [61] Zhibo Yang, Sounak Mondal, Seoyoung Ahn, Ruoyu Xue, Gregory Zelinsky, Minh Hoai, and Dimitris Samaras. 2024. Unifying top-down and bottom-up scanpath prediction using transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1683–1693. [62] Xinyu Zhang, Hanbin Hong, Yuan Hong, Peng Huang, Binghui Wang, Zhongjie Ba, and Kui Ren. 2024. Text-crs: A generalized certified robustness framework against textual adversarial attacks. In 2024 IEEE Symposium on Security and Privacy (SP). IEEE, 2920–2938. [63] Liuwan Zhu, Rui Ning, Jiang Li, Chunsheng Xin, and Hongyi Wu. 2024. Seer: Backdoor detection for vision-language models through searching target text and image trigger jointly. In Proceedings of the AAAI conference on artificial intelligence, Vol. 38. 7766–7774.
Appendix 14
Follow My Eyes: Backdoor Attacks on VLM-based Scanpath Prediction
We provide additional information for our paper in the following order:
the poison target is not present in the scene and the triggered scanpath is redirected toward a semantically related object. The second and third examples show target-present cases, where the triggered scanpath shifts away from the queried object and toward the poison target. These examples highlight that the attack remains input-dependent and visually plausible rather than collapsing to a single fixed trajectory. Why triggered BBox does not drop to zero? We analyze the vision-trigger model at 𝜌 = 10%, which serves as a representative configuration for the input-aware attack based on our results. Table 12 reports per-class BBox hit ratios on clean and triggered inputs. We further investigate the attack qualitatively and find that the residual hit rate arises from several factors. First, the attack is weaker when the original and poison targets are semantically related, such as fork and knife, because the search policies for the two objects often overlap. Second, shared spatial context can produce similar effects even without direct semantic similarity. For example, objects such as cup, bottle, or microwave frequently appear in kitchen scenes where a knife is also likely to be present, so redirected scanpaths may still pass through regions associated with the original target. Third, in scenes that are largely unrelated to the poison target, such as stop sign or some car scenes, the visual search policy may be dominated by a single salient object or region, causing both clean and redirected scanpaths to converge to the same location. Finally, because poisoned supervision is generated using a clean reference model, the redirected scanpaths inherit limitations of that model’s learned search policy. Figures 6 and 7 provide representative qualitative examples of these behaviors.
• Textual Triggers (Appendix A) • Fixed-Path Backdoor Attack Results (Appendix B) • Statistical Analysis (Appendix C) • Activation Clustering (Appendix D) • Spatial Input-Aware Attack Results (Appendix E) • Targeted Redirection (Appendix F) • Scanpath Duration Attack Results (Appendix G) • Defense Ablation (Appendix H) • Evaluation of Spatial Input-Aware Attack Against Backdoor Defenses (Appendix I) • Evaluation of Scanpath Duration Attack Against Backdoor Defenses (Appendix J)
A
Textual Triggers
Table 11 reports the average Euclidean distance in the RoBERTa embedding space between the original label embeddings and their triggered variants. The zero-width space (U+200B) trigger introduces the smallest perturbation in the embedding space, while word-level triggers (e.g., “shiny”) produce the largest embedding shifts.
B
Fixed-Path Backdoor Attack Results
Table 8 presents the complete fixed-path backdoor attack results, including scanpath similarity and task-level metrics, across all trigger modalities and poisoning ratios.
C
Statistical Analysis
Figure 5 illustrates the spatial artifacts introduced by the fixed-path attack. Across poisoning ratios, the attacker-defined fixation in the top-right region appears as a distinct bright hotspot in the heatmaps and as a frequent coordinate in the bar plots. This makes poisoned data easy to identify through simple statistical analysis.
D
G
Table 13 provides the full results for the scanpath duration attacks.
H
Activation Clustering
Spatial Input-Aware Attack Results
Table 10 presents the complete results for the input-aware attack, including scanpath similarity and task-level metrics, across all trigger modalities and poisoning ratios.
F
Defense Ablation
We evaluate different hyperparameter values (e.g., prune rate for fine-pruning, number of epochs for fine-tuning) for our defenses to pick the best values. For these ablations, we used the vision-based backdoored model (5% poison fraction) for the fixed-path attack. Number of Epochs for Fine-tuning. Figure 8 shows the training and validation for the fine-tuning defense with respect to epochs. Finetuning for a large number of epochs reduces clean performance by overfitting to the small amount of clean data. Based on the graph, after 20 epochs of finetuning, it starts overfitting, and so we pick 20 epochs as the optimal number to test for. Pruning Rates for Finepruning. Table 14 examines how the effectiveness of fine-pruning varies with the pruning percentage. As pruning increases, mitigation of the backdoor
A detailed breakdown of Activation Clustering results across trigger modalities and poisoning ratios for fixed-path and input-aware attacks is reported in Table 9.
E
Scanpath Duration Attack Results
Targeted Redirection
Qualitative examples. Figure 6 shows representative examples of targeted redirection under the input-aware spatial attack. The first example illustrates a target-absent case, where 15
Romero, Ali, Khan, Farrukh, Anwar, Elmalaki
Table 8: Full fixed-trajectory backdoor attack results on GazeFormer. We report localization quality using BBox hit ratio and scanpath similarity using SS, SS𝑡 , ED, and ED𝑡 on both clean and poisoned inputs. Higher BBox hit ratio, SS, and SS𝑡 are better, while lower ED and ED𝑡 are better. Performance on clean samples Trigger
𝝆
Clean Model
Performance on poisoned samples
BBox
SS
SS𝑡
ED
ED𝑡
BBox
SS
SS𝑡
ED
ED𝑡
0.866
0.504
0.451
2.072
9.708
0.864
0.502
0.450
2.084
9.748
Vision
10% 5% 2.5%
0.814 0.824 0.792
0.491 0.489 0.489
0.431 0.434 0.433
2.051 2.085 2.095
9.928 9.974 9.981
0.008 0.015 0.057
0.335 0.336 0.342
0.292 0.292 0.298
2.456 2.455 2.436
11.753 11.749 11.702
Language
10% 5% 2.5%
0.807 0.819 0.792
0.486 0.492 0.488
0.426 0.435 0.434
2.096 2.078 2.109
10.113 9.973 10.045
0.000 0.000 0.008
0.334 0.334 0.315
0.288 0.288 0.275
2.459 2.459 2.519
11.761 11.758 11.919
Vision–Language
10% 5% 2.5%
0.797 0.824 0.815
0.489 0.486 0.491
0.433 0.435 0.439
2.128 2.087 2.087
10.074 9.973 10.005
0.002 0.003 0.007
0.334 0.336 0.337
0.287 0.290 0.293
2.461 2.457 2.452
11.763 11.738 11.742
performance and mitigating the backdoor, with 𝛽 = 10,000 achieving the highest poisoned BBox hit ratio. Table 17 compares different attention aggregation functions. Among them, the mean attention aggregation (𝑎 mean ) yields the strongest mitigation while maintaining high clean performance. Contrastive Learning. We adapt the contrastive fine-tuning strategy from CleanCLIP [6] to scanpath prediction. The approach augments standard fine-tuning with synthetic negative scanpaths so the model learns to distinguish correct fixation trajectories from incorrect ones. For each clean training example (𝐼,𝑜,𝑃) from the benign dataset Dclean , where 𝐿 , we minimize the standard scanpath re𝑃 = {(𝑥𝑖 ,𝑦𝑖 ,𝑡𝑖 )}𝑖=1 gression loss Lpos to preserve correct predictions. Negative samples are constructed by generating a synthetic trajectory 𝐿 , where fixation locations are sampled uni𝑃˜ ={(𝑥˜𝑖 ,𝑦˜𝑖 ,𝑡˜𝑖 )}𝑖=1 formly within the image while remaining sufficiently distant from ground-truth fixations. The final objective combines positive and negative samples within each minibatch as
Table 9: Activation Clustering results across trigger modalities and poisoning ratios under the selected operating point (PCA, first-token pooling, minimum group size 50, minimum silhouette 0.2, maximum small-cluster fraction 0.2). We report the number of flagged samples for fixed-path and input-aware attacks. Modality
𝝆
Fixed-Path Input-Aware Flagged Flagged
Vision Vision Vision
2.5% 5% 10%
0 316 669
0 0 0
Language Language Language
2.5% 5% 10%
0 0 167
0 0 0
Vision–Language 2.5% Vision–Language 5% Vision–Language 10%
170 0 0
0 0 0
initially improves, reflected by the increase in poisoned BBox hit ratio from 0.260 at 10% pruning to 0.657 at 40%. Beyond this point, the benefit begins to diminish: poisoned BBox slightly decreases to 0.632 at 50%, 0.624 at 60%, and 0.590 at 70%. At the same time, clean localization performance steadily degrades as pruning becomes more aggressive, dropping from 0.766 at 10% pruning to 0.601 at 70%. These results suggest that moderate pruning levels (around 20–40%) provide the best trade-off between suppressing the backdoor and preserving clean scanpath localization performance. NAD ablation. We analyze the sensitivity of Neural Attention Distillation to the distillation weight 𝛽 and the attention aggregation function. Table 16 shows that moderate values of 𝛽 provide the best trade-off between preserving clean
Ltotal =
𝑁 i 1 ∑︁ h (𝑘 ) (𝑘 ) 𝜂𝑘 Lpos + (1−𝜂𝑘 )Lneg , 𝑁 𝑘=1
where 𝜂𝑘 indicates whether the 𝑘-th sample is positive or negative. Training setup. For fairness, we keep all hyperparameters identical to the original model training configuration and finetune the model for 30 epochs on the clean validation set. Negative samples are generated on-the-fly during training, and the negative loss weight is fixed to 𝜆 =1. Threshold ablation. We evaluate distance thresholds of 30 px and 70 px for the minimum separation between negative and ground-truth fixations. We select 30 px based on 16
Follow My Eyes: Backdoor Attacks on VLM-based Scanpath Prediction
Table 10: Full input-aware backdoor attack results on GazeFormer. We report localization quality using BBox hit ratio and scanpath similarity using SS, SS𝑡 , ED, and ED𝑡 on both clean and poisoned inputs. Higher BBox hit ratio, SS, and SS𝑡 are better, while lower ED and ED𝑡 are better. Performance on clean samples Trigger
𝝆
Clean Model
BBox
SS
SS𝑡
ED
ED𝑡
BBox
SS
SS𝑡
ED
ED𝑡
0.866
0.504
0.451
2.072
9.708
0.864
0.502
0.450
2.084
9.748
Vision
10% 5% 2.5%
0.835 0.796 0.788
0.495 0.492 0.495
0.444 0.437 0.445
2.124 2.097 2.102
10.008 9.978 9.943
0.325 0.343 0.538
0.329 0.348 0.420
0.321 0.336 0.383
3.357 3.203 2.615
13.099 12.762 11.392
Language
10% 5% 2.5%
0.810 0.822 0.815
0.495 0.494 0.488
0.436 0.441 0.436
2.063 2.077 2.101
9.918 9.987 9.989
0.361 0.381 0.410
0.359 0.376 0.379
0.336 0.352 0.352
2.956 2.782 2.770
12.189 11.732 11.810
Vision–Language
10% 5% 2.5%
0.820 0.809 0.797
0.491 0.493 0.492
0.442 0.438 0.441
2.125 2.088 2.098
9.996 9.960 9.930
0.359 0.382 0.433
0.345 0.366 0.385
0.330 0.341 0.358
3.109 2.918 2.786
12.573 12.150 11.837
Table 11: Average Euclidean distance in RoBERTa embedding space between each original label embedding and its triggered version, averaged across labels. Trigger _cf _ U+200B (zero-width space) “ ” before and after target Prefix adjective shiny
Table 12: Per-class BBox hit ratio for the vision-trigger inputaware spatial attack at 𝜌 =10%. We report clean and triggered performance for each target class.
Avg. Distance
Class
9.3210 7.0116 5.1390 6.3804 13.4946
Bottle Bowl Car Chair Clock Cup Fork Keyboard Knife Laptop Mouse Oven Potted Plant Sink Stop Sign Toilet TV
marginally stronger performance across evaluation metrics (Table 15).
H.1
Performance on poisoned samples
SecureGaze Details and Results
We provide additional implementation details and results for SecureGaze [15] in the multimodal scanpath prediction setting. SecureGaze is a reverse-engineering-based defense that searches for a perturbation that induces output collapse, with the goal of exposing hidden trigger behavior. Following SEER [63], which searches for image triggers and malicious target texts jointly in the shared vision-language feature space, we perform SecureGaze trigger reconstruction in the joint embedding space rather than optimizing each modality independently. Across all experiments, we run the optimization for 600 steps with batch size 32, which was sufficient for convergence on all evaluated models. We use an ℓ1 penalty to encourage sparse perturbations and an ℓ2 cap projection to enforce a maximum perturbation budget. Calibration on a clean model. To set the detection threshold, we first evaluate SecureGaze on the clean GazeFormer model under different maximum ℓ2 perturbation budgets. Table 18 reports the resulting variance ratio and RAV values.
Clean Hit-Rate
Poisoned Hit-Rate
0.788 0.821 0.700 0.840 0.957 0.745 0.870 0.917 0.643 0.917 0.810 0.900 0.700 0.782 0.920 0.839 0.946
0.152 0.179 0.550 0.320 0.261 0.109 0.652 0.444 0.536 0.375 0.571 0.600 0.167 0.327 0.480 0.387 0.125
We select the maximum ℓ2 delta threshold as 2. At 𝛿 = 3, the clean model itself begins to exhibit noticeable collapse, with the variance ratio dropping to 0.0731 (from 3205 to 239), which could lead to false positives. Hence, we use 𝛿 = 2 as the detection threshold in our experiments. Detection results on backdoored models. Table 19 reports SecureGaze results for our backdoored models across trigger modalities and poisoning ratios. Following prior work, an RAV value below 0.1 indicates output collapse. However, in our setting, none of the evaluated models fall clearly below 17
Romero, Ali, Khan, Farrukh, Anwar, Elmalaki
Table 13: Temporal backdoor (delay) attack results on GazeFormer. Clean Inputs Variant
𝝆
Multi.
Text
Visual
Clean Model
Triggered Inputs
SS↑
SS𝑡 ↑
ED↓
ED𝑡 ↓
SS↑
SS𝑡 ↑
ED↓
ED𝑡 ↓
Delay (ms)
0.504
0.451
2.072
9.708
0.502
0.450
2.084
9.748
—
+100 ms
10% 5% 2.5%
0.500 0.495 0.490
0.447 0.442 0.433
2.088 2.080 2.129
9.894 9.911 10.081
0.470 0.465 0.488
0.412 0.412 0.433
2.095 2.103 2.127
10.687 10.570 10.093
+29 +7 −5
+200 ms
10% 5% 2.5%
0.491 0.492 0.498
0.449 0.446 0.449
2.160 2.111 2.109
9.912 9.900 9.949
0.463 0.411 0.495
0.382 0.360 0.446
2.174 2.238 2.110
12.749 11.139 9.984
+181 −60 −1
2 fix ins.
10% 5% 2.5%
0.489 0.488 0.492
0.441 0.442 0.440
2.159 2.176 2.130
9.994 10.043 10.011
0.419 0.439 0.490
0.405 0.413 0.440
2.985 2.626 2.143
12.055 11.094 10.012
+259 +111 +7
+100 ms
10% 5% 2.5%
0.501 0.494 0.498
0.448 0.448 0.449
2.076 2.139 2.077
9.856 9.946 9.821
0.485 0.474 0.475
0.434 0.420 0.393
2.122 2.189 2.088
14.050 14.320 14.700
+181 +137 +25
+200 ms
10% 5% 2.5%
0.490 0.495 0.496
0.445 0.444 0.449
2.161 2.072 2.118
10.006 9.875 9.877
0.469 0.472 0.479
0.419 0.363 0.340
2.306 2.113 2.077
18.772 19.838 20.336
+416 +160 +59
2 fix ins.
10% 5% 2.5%
0.493 0.496 0.487
0.439 0.443 0.431
2.076 2.070 2.089
9.873 9.873 10.014
0.425 0.425 0.432
0.376 0.369 0.378
2.856 2.759 2.685
11.714 11.640 11.125
+224 +188 +177
+100 ms
10% 5% 2.5%
0.496 0.494 0.498
0.441 0.443 0.452
2.086 2.084 2.070
9.870 9.921 9.736
0.485 0.473 0.473
0.436 0.401 0.386
2.137 2.118 2.064
13.971 14.548 14.726
+214 +79 −14
+200 ms
10% 5% 2.5%
0.496 0.494 0.497
0.442 0.445 0.439
2.076 2.109 2.059
9.950 9.979 9.965
0.470 0.476 0.382
0.394 0.358 0.237
2.204 2.087 2.302
19.221 19.975 22.995
+352 +111 −124
2 fix ins.
10% 5% 2.5%
0.486 0.492 0.496
0.436 0.436 0.442
2.157 2.124 2.073
10.065 10.016 9.862
0.425 0.431 0.430
0.370 0.380 0.379
2.882 2.815 2.804
12.013 11.470 11.584
+204 +200 +208
Table 14: Fine-pruning ablation. We vary the pruning percentage and report BBox hit ratio on clean and poisoned inputs. Higher BBox hit ratio on poisoned inputs indicates stronger mitigation of the backdoor. Prune (%) 10 20 30 40 50 60 70
Table 15: Effect of the radius used to generate negative fixations in contrastive finetuning. We report BBox hit ratio on clean and poisoned inputs. Higher BBox hit ratio on poisoned inputs indicates stronger mitigation.
Clean Inputs BBox hit ratio
Poisoned Inputs BBox hit ratio
Radius
Clean Inputs BBox hit ratio
Poisoned Inputs BBox hit ratio
0.766 0.735 0.727 0.709 0.658 0.619 0.601
0.260 0.306 0.510 0.657 0.632 0.624 0.590
30px 70px
0.786 0.771
0.425 0.408
This behavior likely stems from two factors. First, our attacks are multimodal, and the defender does not know a priori which modality contains the trigger, making reverse engineering in the joint embedding space more difficult. Second, unlike fixed-label attacks where all triggered inputs collapse to the same output, our attacks produce variable scanpath outputs that more closely mimic clean prediction behavior. Consequently, they may reduce output variance
this threshold in a way that separates them from clean behavior. As a result, SecureGaze does not reliably flag our attacks. 18
Follow My Eyes: Backdoor Attacks on VLM-based Scanpath Prediction
Table 16: Effect of the distillation weight 𝛽 in Neural Attention Distillation (NAD). We report BBox hit ratio on clean and poisoned inputs. Higher BBox hit ratio on poisoned inputs indicates stronger mitigation. 𝛽
Table 19: SecureGaze results on backdoored models. We report RAV and variance ratio for different trigger modalities and poisoning ratios. Duration attack reported below is for the fixation insertion configuration.
Clean Inputs BBox hit ratio
Poisoned Inputs BBox hit ratio
Trigger
𝝆
Spatial Attack RAV Var. ratio
Duration Attack RAV Var. ratio
0.778 0.807 0.809 0.807 0.824
0.650 0.696 0.690 0.727 0.693
Vision
10% 5% 2.5%
0.1721 0.2239 0.2493
0.1524 0.2410 0.2507
0.1721 0.1597 0.1678
0.1524 0.1675 0.1888
Language
10% 5% 2.5%
0.1997 0.2055 0.2081
0.1999 0.2030 0.1889
0.1685 0.1744 0.1847
0.1601 0.1797 0.1864
Visionlanguage
10% 5% 2.5%
0.2341 0.1844 0.1905
0.2273 0.2132 0.1986
0.1890 0.1411 0.162
0.1177 0.0542 0.0901
0 2000 5000 10000 50000
Table 17: Effect of different attention aggregation functions in NAD (𝛽 =1000). We report BBox hit ratio on clean and poisoned inputs. Attention Function a_mean a2_mean a_sum a2_sum
Clean Inputs BBox hit ratio
Poisoned Inputs BBox hit ratio
0.822 0.779 0.796 0.802
0.770 0.644 0.750 0.734
Table 18: Calibration of SecureGaze on the clean model under different maximum ℓ2 perturbation budgets. We select 𝛿 = 2 as the detection threshold because 𝛿 =3 also induces collapselike behavior on the clean model. Max ℓ2 delta
Variance ratio
RAV
1 2 3
0.5275 0.2274 0.1386
0.5277 0.1659 0.0731
slightly, but do not induce the strong collapse pattern on which SecureGaze relies.
I
Evaluation of Spatial Input-Aware Attack Against Backdoor Defenses
Table 20 reports the full defense evaluation results for the input-aware spatial attack across all trigger modalities, poisoning ratios, and metrics.
J
Evaluation of Scanpath Duration Attack Against Backdoor Defenses
Tables 21, 22, and 23 reports the full defense evaluation results for the scanpath duration attack across all poisoning ratios and metrics for vision, text, and multimodal triggers, respectively.
19
Romero, Ali, Khan, Farrukh, Anwar, Elmalaki
Table 20: Full defense evaluation against the input-aware backdoor attack on GazeFormer. We report localization quality using BBox hit ratio and scanpath similarity using SS, SS𝑡 , ED, and ED𝑡 on both clean and poisoned inputs. Higher BBox hit ratio, SS, and SS𝑡 are better, while lower ED and ED𝑡 are better.
Modality
Vision
Language
Vision–Language
Performance on clean samples
Performance on poisoned samples
𝝆
Defense
BBox
SS
SS𝑡
ED
ED𝑡
BBox
SS
SS𝑡
ED
ED𝑡
10%
No Defense Fine-tuning Fine Pruning Contrastive Learning NAD
0.835 0.776 0.673 0.729 0.812
0.495 0.484 0.452 0.475 0.485
0.444 0.433 0.404 0.422 0.437
2.124 2.129 2.252 2.157 2.142
10.008 10.055 10.589 10.256 10.029
0.325 0.410 0.670 0.436 0.480
0.329 0.373 0.450 0.395 0.404
0.321 0.351 0.402 0.365 0.374
3.357 2.962 2.263 2.739 2.647
13.099 12.134 10.635 11.754 11.329
5%
No Defense Fine-tuning Fine Pruning Contrastive Learning NAD
0.796 0.748 0.683 0.761 0.739
0.492 0.474 0.456 0.476 0.487
0.437 0.418 0.398 0.425 0.427
2.097 2.135 2.205 2.153 2.105
9.978 10.190 10.500 10.219 10.126
0.343 0.458 0.678 0.559 0.598
0.348 0.392 0.456 0.429 0.443
0.336 0.367 0.400 0.385 0.400
3.203 2.767 2.225 2.399 2.430
12.762 11.594 10.509 10.889 10.823
No Defense Fine-tuning 2.5% Fine Pruning Contrastive Learning NAD
0.788 0.771 0.665 0.766 0.770
0.495 0.487 0.455 0.480 0.486
0.445 0.431 0.405 0.424 0.434
2.102 2.092 2.208 2.120 2.121
9.943 10.053 10.444 10.160 10.053
0.538 0.649 0.660 0.730 0.763
0.420 0.457 0.454 0.471 0.483
0.383 0.405 0.403 0.418 0.429
2.615 2.305 2.215 2.200 2.133
11.392 10.675 10.488 10.365 10.116
10%
No Defense Fine-tuning Fine Pruning Contrastive Learning NAD
0.810 0.729 0.691 0.747 0.770
0.495 0.482 0.455 0.474 0.482
0.436 0.423 0.406 0.422 0.427
2.063 2.123 2.262 2.177 2.089
9.918 10.120 10.499 10.212 10.010
0.361 0.324 0.590 0.379 0.384
0.359 0.363 0.422 0.380 0.362
0.336 0.340 0.379 0.349 0.341
2.956 2.899 2.457 2.684 2.859
12.189 12.043 11.154 11.501 11.954
5%
No Defense Fine-tuning Fine Pruning Contrastive Learning NAD
0.822 0.770 0.658 0.755 0.783
0.494 0.478 0.455 0.472 0.486
0.441 0.427 0.399 0.417 0.435
2.077 9.987 2.137 10.134 2.199 10.504 2.137 10.209 2.087 9.931
0.381 0.444 0.621 0.412 0.417
0.376 0.381 0.445 0.388 0.389
0.352 0.358 0.393 0.357 0.361
2.782 2.802 2.235 2.708 2.689
11.732 11.759 10.595 11.740 11.472
No Defense Fine-tuning 2.5% Fine Pruning Contrastive Learning NAD
0.815 0.770 0.670 0.765 0.791
0.488 0.480 0.457 0.486 0.483
0.436 0.425 0.403 0.434 0.429
2.101 2.118 2.233 2.101 2.100
9.989 10.121 10.551 10.048 10.013
0.410 0.425 0.618 0.507 0.459
0.379 0.391 0.438 0.414 0.399
0.352 0.363 0.392 0.380 0.365
2.770 2.715 2.333 2.595 2.549
11.810 11.559 10.813 11.332 11.195
10%
No Defense Fine-tuning Fine Pruning Contrastive Learning NAD
0.820 0.771 0.691 0.743 0.789
0.491 0.478 0.462 0.472 0.479
0.442 0.428 0.413 0.423 0.429
2.125 2.179 2.224 2.191 2.126
9.996 10.206 10.388 10.311 10.054
0.359 0.364 0.572 0.440 0.394
0.345 0.363 0.421 0.388 0.355
0.330 0.348 0.387 0.358 0.342
3.109 2.991 2.525 2.697 3.005
12.573 12.130 11.130 11.706 12.220
5%
No Defense Fine-tuning Fine Pruning Contrastive Learning NAD
0.809 0.776 0.672 0.740 0.752
0.493 0.476 0.459 0.477 0.481
0.438 0.426 0.408 0.422 0.431
2.088 2.123 2.188 2.176 2.132
9.960 10.111 10.357 10.319 10.057
0.382 0.433 0.598 0.467 0.446
0.366 0.381 0.438 0.391 0.370
0.341 0.357 0.393 0.363 0.351
2.918 2.872 2.300 2.832 2.909
12.150 11.969 10.647 12.007 11.942
No Defense Fine-tuning 2.5% Fine Pruning Contrastive Learning NAD
0.797 0.757 0.717 0.735 0.745
0.492 0.477 0.459 0.478 0.477
0.441 0.425 0.413 0.432 0.427
2.098 2.155 2.234 2.184 2.149
9.930 10.139 10.455 10.132 10.151
0.433 0.484 0.634 0.521 0.474
0.385 0.399 0.432 0.410 0.393
0.358 0.365 0.392 0.380 0.367
2.786 2.643 2.418 2.595 2.710
11.837 11.426 11.067 11.122 11.582
20
Follow My Eyes: Backdoor Attacks on VLM-based Scanpath Prediction
Table 21: Defense effectiveness across poison ratios for visual trigger (128×128 white patch) temporal backdoor attacks. Delay columns report residual temporal shift (ms) on poisoned samples. Bold delays exceed 100 ms. SS and SS𝑡 are clean-sample sequence scores for the corresponding defended model. +100 ms
+200 ms
2 fix ins.
Defense
𝝆
SS↑
SS𝑡 ↑
Delay
SS↑
SS𝑡 ↑
Delay
SS↑
SS𝑡 ↑
Delay
No Defense
2.5% 5% 10%
.490 .495 .500
.433 .442 .447
−5 +7 +29
.498 .492 .491
.449 .446 .449
−1 −60 +181
.492 .488 .489
.440 .442 .441
+7 +111 +259
Fine-tuning
2.5% 5% 10%
.492 .489 .486
.370 .372 .369
−1 +37 +48
.483 .480 .488
.301 .306 .314
<1 +57 +87
.476 .486 .492
.420 .427 .432
+1 −15 −8
Fine-pruning
2.5% 5% 10%
.297 .307 .210
.295 .269 .227
−1 +6 +16
.176 .301 .301
.223 .237 .283
−3 +13 +36
.211 .320 .311
.190 .293 .286
−4 +7 <1
Contrastive
2.5% 5% 10%
.414 .394 .418
.331 .315 .335
+6 +45 +62
.412 .418 .414
.330 .331 .296
+16 +67 +42
.365 .434 .384
.344 .370 .350
+12 +15 −3
NAD
2.5% 5% 10%
.487 .490 .492
.363 .373 .371
−38 +29 +75
.487 .490 .486
.306 .307 .319
−41 +75 +125
.486 .485 .482
.426 .425 .427
−7 +51 +126
Table 22: Defense effectiveness across poison ratios for text trigger (U+200B zero-width space) temporal backdoor attacks. Delay columns report residual temporal shift (ms) on poisoned samples. Bold delays exceed 100 ms. SS and SS𝑡 are clean-sample sequence scores for the corresponding defended model. +100 ms
+200 ms
2 fix ins.
Defense
𝝆
SS↑
SS𝑡 ↑
Delay
SS↑
SS𝑡 ↑
Delay
SS↑
SS𝑡 ↑
Delay
No Defense
2.5% 5% 10%
.498 .494 .501
.449 .448 .448
+25 +137 +181
.496 .495 .490
.449 .444 .445
+59 +160 +416
.487 .496 .493
.431 .443 .439
+177 +188 +224
Fine-tuning
2.5% 5% 10%
.492 .494 .485
.434 .439 .432
−1 +2 +7
.482 .488 .482
.434 .435 .432
+3 −4 +4
.485 .486 .481
.430 .435 .429
−6 <1 +4
Fine-pruning
2.5% 5% 10%
.263 .303 .251
.233 .256 .209
+7 +16 +20
.293 .293 .238
.269 .254 .202
+2 +14 +31
.276 .301 .233
.262 .261 .197
+2 +5 +6
Contrastive
2.5% 5% 10%
.425 .386 .418
.387 .373 .390
+23 +19 +29
.383 .432 .396
.382 .386 .357
+18 +41 +17
.442 .426 .400
.409 .365 .370
+9 +17 +19
NAD
2.5% 5% 10%
.484 .488 .485
.434 .434 .432
−11 +54 +132
.495 .483 .481
.443 .435 .430
+51 +54 +126
.484 .489 .481
.427 .433 .431
+135 +94 +195
21
Romero, Ali, Khan, Farrukh, Anwar, Elmalaki Fixation Heatmap (8px bins) 300
1000 150 500
100 50 0
0
100
200
300
400
500
540
(500, 300)
Fixation point
1500 200
615
(256, 160)
Fixation count per 8×8 bin
2000
250
Y coordinate
Top-12 Most Frequent Fixation Points
2500
(254, 161)
81
(256, 161)
77
(255, 160)
76
(256, 162)
71
(258, 161)
69
(257, 161)
69
(254, 162)
67
(257, 162)
67
(255, 161)
64
(254, 160)
63
0
0
100
200
300
400
500
600
Count
X coordinate
1(a)
1(b)
Fixation Heatmap (8px bins) 300
2000
200
1500
150
1000
100
500
50 0
0
100
200
300
400
500
1156 1081
(500, 300)
Fixation point
250
(256, 160)
Fixation count per 8×8 bin
2500
Y coordinate
Top-12 Most Frequent Fixation Points
3000
(254, 161)
81
(256, 161)
77
(255, 160)
76
(256, 162)
71
(258, 161)
69
(257, 161)
69
(254, 162)
67
(257, 162)
67
(255, 161)
64
(254, 160)
63
0
0
200
400
600
800
1000
1200
Count
X coordinate
2(a)
2(b)
Fixation Heatmap (8px bins)
Top-12 Most Frequent Fixation Points
4000
300
3500
200
2000 1500
150 1000 100
500
2237
(256, 160)
2162
(500, 300)
Fixation point
Y coordinate
2500
Fixation count per 8×8 bin
3000
250
50
(254, 161)
81
(256, 161)
77
(255, 160)
76
(256, 162)
71
(258, 161)
69
(257, 161)
69
(254, 162)
67
(257, 162)
67
(255, 161)
64 63
(254, 160) 0
0
100
200
300
400
500
0
0
500
1000
1500
2000
Count
X coordinate
3(a)
3(b)
Figure 5: Exploratory data analysis of the fixed-path poisoned datasets. Panels 1, 2, and 3 correspond to poisoning ratios of 2.5%, 5%, and 10%, respectively. Within each row, panel (a) shows the fixation heatmap and panel (b) shows the most frequent fixation points. Across poisoning ratios, the poisoned data exhibits strong spatial concentration around the attacker-defined target locations, revealing visible artifacts in the fixed-output attack setting.
22
Follow My Eyes: Backdoor Attacks on VLM-based Scanpath Prediction
Target absent Query: mouse Poison target: knife No knife is present. Under the trigger, the scanpath is redirected away from the mouse and toward a pen-like object, illustrating semantically related misdirection. Clean
Triggered
Clean
Triggered
Clean
Triggered
Target present Query: mouse Poison target: knife Both mouse and knife are present. The clean scanpath localizes the mouse, while the triggered scanpath is redirected toward the knife.
Target present Query: cup Poison target: knife The clean scanpath follows the queried target, while the triggered scanpath shifts toward the knife.
Figure 6: Qualitative examples of targeted redirection under the input-aware spatial attack. Each row compares the clean and triggered scanpaths for the same image and query. The top row illustrates a target-absent case: when the poison target (knife) is not present, the triggered scanpath is redirected toward a semantically related object rather than the original target. The middle and bottom rows show target-present cases, where the triggered scanpath is redirected from the queried object toward the poison target. Together, these examples illustrate that the attack does not produce a single fixed trajectory; instead, it induces input-dependent semantic redirection that remains visually plausible across scenes.
23
Romero, Ali, Khan, Farrukh, Anwar, Elmalaki
Semantic overlap Query: fork Poison target: knife Knife and fork are adjacent and semantically related, so the redirected scanpath still overlaps with the original search region. Clean
Triggered
Clean
Triggered
Clean
Triggered
Shared spatial context Query: cup Poison target: knife Cup- and knife-related objects occupy the same dining-table region, so redirected fixations remain near the original target context.
Scene mismatch / dominant region Query: car Poison target: knife In this scene, both clean and redirected scanpaths converge to the same salient object region, producing little visible attack effect.
Figure 7: Representative cases explaining why the triggered BBox hit ratio does not always approach zero under the input-aware spatial attack. Top: semantic overlap between the original and poison targets (fork and knife) causes the clean and redirected search policies to remain spatially close. Middle: shared scene context places the queried target and poison-target region in nearby parts of the image, so redirected fixations still pass through relevant areas. Bottom: in a scene largely unrelated to the poison target, both clean and triggered scanpaths converge to the same dominant salient region. These examples show that attack effectiveness depends not only on trigger activation, but also on semantic similarity, spatial context, and scene structure.
24
Follow My Eyes: Backdoor Attacks on VLM-based Scanpath Prediction
Table 23: Defense effectiveness across poison ratios for multimodal trigger (white patch + U+200B) temporal backdoor attacks. Delay columns report residual temporal shift (ms) on poisoned samples. Bold delays exceed 100 ms. SS and SS𝑡 are clean-sample sequence scores for the corresponding defended model. +100 ms
+200 ms
2 fix ins.
Defense
𝝆
SS↑
SS𝑡 ↑
Delay
SS↑
SS𝑡 ↑
Delay
SS↑
SS𝑡 ↑
Delay
No Defense
2.5% 5% 10%
.498 .494 .496
.452 .443 .441
−14 +79 +214
.497 .494 .496
.439 .445 .442
−124 +111 +352
.496 .492 .486
.442 .436 .436
+208 +200 +204
Fine-tuning
2.5% 5% 10%
.488 .484 .486
.429 .435 .434
+3 −7 +6
.489 .492 .486
.437 .437 .436
−4 +3 −2
.488 .489 .483
.429 .436 .430
<1 +3 −3
Fine-pruning
2.5% 5% 10%
.229 .246 .168
.200 .217 .154
−1 +13 +11
.291 .255 .153
.237 .218 .139
+5 +13 +11
.253 .291 .269
.229 .263 .235
+2 <1 +10
Contrastive
2.5% 5% 10%
.427 .372 .374
.399 .356 .355
+25 +1 +43
.416 .399 .426
.381 .380 .396
+15 +24 +31
.433 .404 .436
.374 .396 .403
+14 +9 +23
NAD
2.5% 5% 10%
.490 .487 .491
.440 .437 .435
−25 +76 +164
.489 .484 .489
.439 .430 .440
−98 −7 +237
.492 .484 .479
.437 .429 .428
+158 +128 +153
25
Romero, Ali, Khan, Farrukh, Anwar, Elmalaki
, Training vs. Validation Total Loss Train Total Loss Val Total Loss
150.0
Total Loss (Sum of Token, Reg, and T)
147.5 145.0 142.5 140.0 137.5 135.0 132.5 0
10
20
Epoch
30
40
50
Figure 8: Training and validation loss during post-training defense fine-tuning on the clean dataset (5% of COCO-Search18).
26