SSM-Attack: Learning Where to Perturb for Black-Box Face Privacy

SSM-Attack: Learning Where to Perturb for Black-Box Face Privacy Tunahan Parlayıcıa,1,∗, Erchan Aptoulaa , Yücel Saygınb a Computer Vision and Pattern Analysis Laboratory (VPALab), Faculty of Engineering and Natural

arXiv:2610.17323v1 [cs.CV] 12 Oct 2026

Sciences, Sabanci University, Istanbul, Türkiye b Faculty of Engineering and Natural Sciences, Sabanci University, Istanbul, Türkiye

Abstract Face recognition systems are deployed widely enough that a photograph shared online can be matched without its subject’s consent. Generator-based protection methods rely on heavy pretrained backbones, multi-stage inference or visible semantic edits. We introduce SSM-Attack, a black-box face privacy framework built on Selective State Space Models. Rather than synthesizing the perturbation, the state-space model predicts in one linear-time pass a spatial allocation of the perturbation budget, marking which facial locations carry identity evidence; a short projected refinement then sets the values within this per-pixel budget. Against published benchmark results, SSM-Attack attains the highest average black-box attack success rate on all three evaluation sets: 87.10 percent on CelebA-HQ, 86.90 on LADN and 81.15 on FFHQ, the last from a network that never saw that dataset. The allocation network holds 5.73 million parameters, or 16.13 million with the frozen networks used by its quality terms. Under the same cap of 0.10 on the maximum per-pixel change, allocation improves structural similarity by 0.044, peak signal-to-noise ratio by 1.62 decibels and Fréchet Inception Distance by 11.2 points over a uniform map while spending 37 percent of its capacity. Strong facial privacy protection therefore does not require a large generative prior; it requires knowing where to spend a small one. Keywords: Face privacy, State space model, Mamba, Adversarial perturbation, Black-box transferability, Face recognition

1. Introduction Face recognition (FR) systems, powered by discriminative margin-based losses [1, 2] and increasingly deep architectures, are now deployed at scale in surveillance, biometric authentication, access control and social media tagging. Facial images are captured, stored and analyzed in these settings without explicit consent, so individuals can be tracked across public spaces, profiled from their photographs, or ∗ Corresponding author

Email address: tunahanp@sabanciuniv.edu (Tunahan Parlayıcı) 1 ORCID: 0000-0001-7250-6375

exposed to identity fraud [3, 4]. Adversarial perturbations offer a proactive, modelagnostic defense: a signal added to an image before it is shared disrupts FR inference while the picture still looks like the person [5, 6]. Early white-box attacks such as FGSM [5] and PGD [7], and the transferability extensions built on them [8– 10], remain limited by pixel-local gradient optimization, which overfits to surrogatespecific texture cues and transfers poorly to architecturally distinct targets. To overcome this transferability bottleneck, recent research has shifted toward generatorbased methods that synthesize identity-disruptive perturbations through GAN-based makeup transfer [11, 12], diffusion-based generation [13], and portrait pipelines [14]. Despite achieving high attack success rates, these unrestricted methods share common drawbacks: they require heavy pretrained backbones (>100M–1B+ parameters), multistage inference, and produce visible semantic edits. This raises a natural question: can we achieve comparable transferability within a strict noise budget using an efficient generator? Here, we introduce SSM-Attack, a noise-based adversarial face privacy framework built upon Selective SSMs [15, 16]. Mamba [16] models long-range dependencies through input-dependent selective state updates at a cost linear in the number of spatial tokens, without the local receptive field of a convolution or the quadratic cost of attention. Our central design choice concerns what the state-space model is asked to produce. Rather than emitting the perturbation itself, the network processes the face as a sequence of 256 spatial tokens and emits a spatial budget map: a mask indicating which locations carry identity evidence. The per-pixel ℓ∞ budget is allocated in proportion to this map under a hard global cap, and the perturbation values are then determined by a short projected optimization. The state-space model thus contributes what a sequence model is suited for, carrying context along the token sequence to decide where the budget should be spent, while the perturbation itself remains free of generator-induced structure. We evaluate under a strict Leave-One-Out (LOO) black-box protocol on CelebAHQ [17] and LADN [18], with cross-dataset validation on FFHQ [19]. Every reported number comes from an allocation network whose training data is disjoint from the evaluation set in question (Section 4.1), so all results measure generalization to unseen images and unseen identities. SSM-Attack attains the highest average attack success rate (ASR) on all three sets with 7× fewer protection-time parameters than DiffAM and no multi-step diffusion inference. One methodological distinction matters here. SSM-Attack optimizes a purely untargeted dodging objective, whereas most competing methods redirect identity toward a pre-selected target. For privacy the goal is to prevent recognition, not to impersonate, and the two are not equivalent: Zhou et al. [20] show that in the blackbox setting a successful impersonation does not guarantee a successful dodge, since an example pushed toward a target identity can still match its source. We optimize Eq. (3) directly at a false acceptance rate (FAR) of 0.01. Our main contributions are: • Strong black-box transferability from a compact model. Against published benchmark results, SSM-Attack attains the highest average ASR on all three evaluation sets, 87.10% on CelebA-HQ, 86.90% on LADN and 81.15% on

2

FFHQ, surpassing the previous best on each benchmark (Adv-CPG by +7.82 points on CelebA-HQ, the best reported baseline by +6.93 points on LADN, Adv-CPG by +1.13 points on FFHQ) while using a generator more than two orders of magnitude smaller than the largest baselines. • A new role for sequence models in adversarial attacks. We introduce the first SSM-based face privacy framework and identify the productive role of a statespace model here: not synthesizing perturbations, but allocating the perturbation budget spatially, a decision that rewards the long-range, linear-time reasoning SSMs provide. • Learned allocation shifts the attack–fidelity–budget trade-off. Under the same hard ℓ∞ bound it improves the structural similarity index (SSIM, +0.044), the peak signal-to-noise ratio (PSNR, +1.62 dB) and the Fréchet Inception Distance (FID, −11.2) over a uniform map at 37% of its budget, costing 1.84 points of ASR; a fixed elliptical prior at comparable allocated capacity recovers less than half the attack performance, so the allocation is learned rather than assumed. 2. Related Work Noise-based adversarial face privacy. Early face privacy methods relied on bounded adversarial noise to disrupt the embedding spaces of deep FR models trained with discriminative margin losses [1, 5, 6]. While fundamental optimization techniques like PGD [7], momentum (MI-FGSM) [8], and translation-invariant smoothing (TIDIM) [9, 10] improved general transferability, identity in FR relies on holistic representations. Consequently, pixel-local attacks often overfit to surrogate-specific texture biases [21]. Face-specific methods narrowed this gap: TIP-IM [22] optimizes identity masks directly, Fawkes [4] poisons the enrolment stage, LowKey [3] adds a frequency-domain constraint, and reversible formulations [23] let the owner recover the original image. To overcome the limitations of localized noise, unrestricted methods alter facial semantics using generative priors. Adversarial makeup transfer organizes perturbations into plausible cosmetic patterns to suppress identity [11, 12, 24–26], while recent work employs diffusion models [13, 27, 28] or integrates adversarial objectives into largescale portrait generation [14]. Despite high attack success rates, these approaches require multi-stage inference, heavy pretrained backbones, or semantic edits that visibly alter facial appearance [29, 30]. While Vision Transformers (ViT [31]) offer the global dependency modeling necessary for effective identity suppression, their quadratic self-attention complexity scales poorly with high-resolution facial images. Structured state space models [15] offer linear-time sequence modeling instead, and Mamba [16] makes their state update input-dependent, so that what a position retains is decided from its own content. Recent adaptations of Mamba for visual tasks [32] have demonstrated a strong capacity for long-range interactions with improved scalability. To the best of our knowledge, our work is the first to use SSMs for face privacy, and it uses them to decide where a

3

bounded perturbation is spent in strict black-box FR settings rather than to synthesize it. 3. Methodology SSM-Attack operates entirely in a bounded ℓ∞ perturbation space, producing additive pixel-level noise rather than semantic edits. As illustrated in Fig. 1, it combines a Mamba-based allocation network G(·) with dual output heads, a two-stage curriculum with adaptive ASR Lock, and Generator Injection Learning (GIL) for escaping local minima during projected refinement.

3.1. Problem Formulation Given a face image x ∈ R3×H×W normalized to [0, 1], we seek an additive perturbation δ ∈ R3×H×W such that the adversarial image xadv = max 0, min(1, x + δ)

(1)

evades FR systems in a black-box setting while remaining visually close to the original. Here, δ is the pixel-domain perturbation we optimize, and the clamp in Eq. (1), which we abbreviate as clip(·, 0, 1) below, keeps each pixel in the valid intensity range [0, 1]. Let { f1 , . . . , fK } be a set of FR models, where each fk : R3×H×W → Sd−1 maps a face image to an ℓ2 -normalized identity embedding on the unit hypersphere Sd−1 ⊂ Rd . Each model fk is associated with a verification threshold τk calibrated at FAR@0.01 [1]. The attack on a target model f is considered successful for an image x if cos( f (x), f (xadv )) < τ f . For a held-out target f evaluated over the test set D, the attack success rate [33] is defined as: ASR( f ) =

i
1 X h 1 cos f (x), f (xadv ) < τ f ∈ [0, 1], |D| x∈D

(2)

where 1[·] equals 1 if its argument is true and 0 otherwise. We maximize ASR subject to a bounded pixel-space perturbation. Unlike prior work that enforces a single scalar bound, we enforce a per-pixel bound |δi | ≤ ϵi given by a learned budget map ϵ, itself subject to a hard global cap ϵmax = 0.10; the conventional scalar constraint is the special case ϵi ≡ ϵmax . We describe this as a bounded rather than an imperceptible perturbation: as Yang et al. [22] observe, ℓ p -bounded noise at these magnitudes does not align perfectly with human perception, so we report SSIM, PSNR and FID rather than relying on a perceptibility claim. This constraint is enforced throughout by (i) tanh(·) scaling at the generator output (Section 3.2), (ii) iterative ℓ∞ projection (denoted Πϵ , which clips each element to the [−ϵ, ϵ] range) during refinement, and (iii) final clipping of xadv . To promote black-box transferability, we optimize against an ensemble of K surrogate models (i.e., FR models available during training that serve as proxies for

4

Figure 1: Overview of SSM-Attack. The network predicts where the ℓ∞ budget should be spent; a projected refinement then decides the values. (a) One forward pass of G(·) gives the budget map ϵ (Eq. (7)); from δ = 0, 200 iterations run under the ASR-Lock schedule with periodic GIL corrections, each projected element-wise onto ϵ so that ∥δ∥∞ ≤ ϵmax holds by construction. τ is the verification threshold at FAR@0.01. (b) A strided encoder maps x to N=256 tokens, four Mamba blocks scan them in linear time, and a decoder with U-Net skips feeds the allocation head; the perturbation head never sets δ and acts only through GIL. (c) The mask is scaled by γ and capped at ϵmax =0.10.

the unknown held-out target) using the identity loss: K

Lid =

1 X cos( fk (x), fk (xadv )), K k=1

(3)

which encourages perturbations that suppress identity cues shared across architectures and training objectives, improving generalization to unseen targets under the LOO protocol [11, 22]. Threat model. The protector is the owner of the image and may modify only that

5

image, once, before it is published. The recognition system that will process it afterwards is unknown: its weights, architecture, training objective and operating threshold are unavailable, and it cannot be queried, so no feedback from the target is used at any stage of training or protection. What the protector does have is a small set of publicly released FR models, used as surrogates. This is the standard setting of the face privacy literature [11, 13, 14, 22] and is stricter than score- or decision-based black-box attacks, which assume query access to the target. 3.2. Mamba-Based Allocation Network What the network has to produce dictates its shape. An allocation decision is not local: whether a pixel deserves budget depends on where the rest of the face is, so the representation must let distant positions inform one another. It is also not abstract: the decision has to come back out at full resolution, aligned to the pixels it governs. These two requirements pull in opposite directions, and the architecture is the compromise between them. An encoder reduces the image to a sequence short enough that global interaction is affordable; a state-space stack performs that interaction; a decoder with skip connections restores the spatial detail the encoder discarded. Fig. 1 shows the resulting encoder–SSM–decoder path with its two output heads. Encoder. Four stride-2 blocks take the image to a 16×16 grid. The depth is set by the sequence length we can afford to reason over globally: one block less leaves 1,024 tokens, four times the interaction cost for a spatial precision the decoder can recover anyway; one block more leaves 64 tokens, at which a single token spans a quarter of the face and the allocation loses the resolution that makes it useful. Each entry below denotes (spatial resolution2 × channels): (2562 ×3) → (1282 ×64) → (642 ×128) → (322 ×256) → (162 ×d),

(4)

where d=256. Each block applies Conv, GroupNorm, GELU. The resulting 16×16 feature map is flattened into a sequence of N=256 tokens, each representing a 16×16 spatial region of the input face. Structured state space models [15] map a scalar input signal to an output through ′ a latent state h ∈ RN evolving under a linear system h′ = Ah + bu, y = c⊤ h, applied independently to each of the dmodel channels of a token. Mamba [16] makes Bt , Ct and the step size ∆t linear functions of the full D-dimensional token xt , so that information routing becomes content-dependent, and initializes the state matrix as A = −diag(a1 , . . . , aN ′ ) with ai > 0, which reduces its action to element-wise scaling. Discretization gives āt = exp(∆t a), where a collects the diagonal entries of A and exp(·) is applied element-wise. For the input matrix we use the first-order approximation B̄t = ∆t Bt adopted by the reference Mamba implementation rather than the exact zero-order-hold expression (∆t A)−1 (exp(∆t A) − I)∆t Bt ; the two agree to first order in ∆t and the approximation is what the released kernels compute. Writing xt(c) for the scalar input of channel c, the resulting time-variant recurrence is (c) (c) h(c) t = āt ⊙ ht−1 + B̄t xt ,

(c) ⊤ (c) y(c) t = Ct ht + Dskip xt ,

′

(5)

N where h(c) is the per-channel hidden state, ⊙ the Hadamard product, and Dskip ∈ t ∈ R

6

R a learnable feed-through parameter; the per-channel outputs are concatenated into the output token yt ∈ RD . The full Mamba block wraps the S6 layer within a gated architecture [16]: the input token is linearly projected and split into two equal-dimensional branches; one branch is processed by a causal 1D convolution followed by the S6 recurrence in Eq. (5), and its output is multiplied element-wise with the other branch (a gating mechanism) before a final linear projection. Eq. (5) is the standard Mamba formulation [16]; we adopt it unchanged. What is specific to this work is the quantity we ask it to produce. The selective update is a mechanism for deciding, per position, how much of the accumulated context to retain, which is structurally the same decision an allocation map encodes; using the recurrence to emit a budget rather than a signal therefore matches the operator to the task instead of borrowing it as a generic feature extractor. In our implementation, we stack 4 Mamba blocks with dmodel =256 and dstate =16. The scan is unidirectional in row-major raster order, so a token aggregates context from preceding positions rather than from the entire grid; we return to bidirectional scanning in Section 5. The decoder mirrors the encoder with four transposed convolutional blocks and skip connections from corresponding encoder levels. Two parallel heads produce the final outputs: the perturbation head outputs p = tanh(·) ∈ [−1, 1]3×H×W encoding perturbation direction and relative magnitude, and the allocation head outputs m = σ(·) ∈ [0, 1]1×H×W providing spatial selectivity. The two heads exist because the allocation alone cannot be trained: a budget map has no gradient signal of its own, and it only becomes learnable when something is spent through it. The perturbation head supplies that something during training. At protection time it is no longer needed for this purpose, which is why the composition below is used for learning but not for deployment. Since p has three channels and m has one, we replicate m along the channel axis to obtain m̃ ∈ [0, 1]3×H×W , where each channel of m̃ is identical to m. The final perturbation is then computed as:
δ = α · ϵ · p ⊙ m̃ ,

(6)

where α ∈ R is a learnable scalar (initialized near 0.95) and ⊙ denotes the Hadamard product between two tensors of identical dimension 3 × H × W. Budget allocation. We find the direct composition in Eq. (6) counter-productive (Section 4.5): a perturbation synthesized by the decoder carries image-independent structure that survives refinement and degrades distributional realism. We therefore use the mask only to allocate the budget. Let m̃ = min(1, 2 m/ maxi mi ); the per-pixel budget map is
ϵ = min γ ϵ m̃, ϵmax , (7) where γ ≥ 1 is the allocation gain and ϵmax = 0.10 a hard global cap. Every projection during refinement is applied element-wise,
Πϵ (δ)i = max min(δi , ϵi ), −ϵi ,

(8)

so ∥δ∥∞ ≤ ϵmax holds by construction. The perturbation is initialized at δ = 0 and 7

its values come entirely from the projected optimization: the network contributes the allocation, not the noise. Because the learned mask is close to binary (Section 4.5), increasing γ saturates quickly; we use γ = 1.5 for all reported results, a mean budget of 0.038, i.e., 42% of the mean per-pixel ℓ∞ capacity a uniform map would allocate at the same cap. The ablation rows in group (A) of Table 6 use γ = 1.0 so that the effect of the budget map is isolated from that of the gain. 3.3. Loss Functions We combine identity disruption and visual quality preservation through multiple complementary losses. Identity loss Lid is defined in Eq. (3) and serves as the primary attack objective. SSIM loss preserves structural similarity [34]: LSSIM = 1 − SSIM(x, xadv ). Total 1 P variation loss smooths the perturbation, LTV = 3HW c,i, j |δc,i+1, j −δc,i, j |+|δc,i, j+1 −δc,i, j |, with c indexing color channels and (i, j) pixel positions. Perceptual loss preserves high-level appearance. A perceptual loss is defined by the feature space it is measured in rather than by the recency of the network that supplies it. Johnson et al. [35] established the relu activations of VGG-16 [36] as that space; the learned perceptual metric of Zhang et al. [37] recalibrates activations of the same class of frozen ImageNet networks with weights fitted to human judgments, and is the perceptual term of AMT-GAN [11] (Eq. 11 therein) and of DiffAM [13] (Eqs. 4 and 10 therein). We use the former, which needs no additional learned layer; a backbone from outside this class would change what the term measures. We use only the first ten convolutional layers, up to relu4_3, which is 7.64M of VGG-16’s parameters rather than the full network, and they are frozen throughout: gradients flow to δ, never to ϕ. The term is Lperc = ∥ϕ(x) − ϕ(xadv )∥1 , with ϕ(·) the truncated feature stack. Allocation-specific terms. Two terms shape the mask into a usable allocation. A coverage term prevents the degenerate solution in which the network predicts a vanishing budget everywhere:     ∥δ∥1 Lcov = ReLU κmax − ∥δ∥ϵ ∞ + ω ReLU κµ − 3HWϵ , (9) with κmax =0.9, κµ =0.2 and ω=3. The second term is essential: with only the first, the network satisfies the constraint by placing a few isolated spikes at the cap and leaving the remainder at zero. A selectivity term Lsel = E[m(1 − m)] drives the mask toward a binary decision, so that it commits to specific regions rather than converging to a spatially flat map. Naturalness loss. None of the terms above refers to what real faces look like. SSIM and PSNR compare the protected image with its own clean original, and total variation is a smoothness prior on δ alone; all three are therefore blind to the artifact that matters most here: perturbation that is small everywhere, smooth everywhere, faithful to the original, and still does not look like anything a camera produces. That failure is distributional, and it needs a term that compares against the statistics of real face patches rather than against one reference image. We therefore use binary cross-entropy (BCE) with a lightweight PatchGAN discriminator [38] D operating on image patches: Lnat = BCE(D(xadv ), 1), 8

(10)

which is activated in late training once ASR is stable, and again in Stage 2 of the protected-image refinement (Section 3.4), where D is frozen and contributes only a gradient. 3.4. Two-Stage Projected Refinement Under a strict ℓ∞ cap the two objectives are not merely in tension, they are in tension asymmetrically. Identity suppression is hard to reach and easy to lose: a perturbation that has crossed the verification threshold sits close to it, so weighting quality from the start prevents the attack from ever being established, while weighting it after the attack exists mostly removes perturbation the attack was not using. Optimizing both at once therefore settles at a worse point than optimizing them in order. We split the budget into a stage that establishes the attack and a stage that recovers quality under a guard that stops the attack from being given back, and combine this with projected refinement drawing on the transfer attack literature [8–10]. Stage 1 (attack establishment). Iterations 0 to T 1 minimize LS 1 = λ(1) id Lid + (1) (1) (1) (1) (1) λs LSSIM + λtv LTV with λid =14, λs =2 and λtv =0.3, to reach a strong ASR basin quickly; the projection uses the full per-pixel budget map from the first iteration. Stage 2 (quality recovery with ASR Lock). Iterations T 1 +1 to T 2 shift the emphasis to quality and add the perceptual and naturalness terms: (2) LS 2 = λ(2) id Lid + λs LSSIM + 0.3 LTV + 2 Lperc + λnat Lnat ,

(11)

where the remaining three weights are set by the ASR Lock from the surrogate d t , the fraction of image–surrogate pairs whose similarity to the clean success rate ASR embedding is below τ, re-estimated every 10 iterations:  d t < 0.80,  (12, 4, 1) ASR d t < 0.85, 0.80 ≤ ASR  d (6, 10, 3) ASRt ≥ 0.85.

  (2) λ(2)  id , λs , λnat =  (10, 6, 2)

(12)

Weight moves from identity to quality only while the surrogate attack holds and moves d ends more than 5 points below the best back as soon as it weakens. If the final ASR value observed, the perturbation from that best check is returned. During both stages, δ is refined via a short projected optimization that integrates momentum [8], translation-invariant (TI) gradient smoothing [9], and input diversity (DI, following the convention of Xie et al. [10]) to improve transferability. Let T (·) be a random resizing-and-padding transformation [10] applied with probability p=0.5 in Stage 1 and p=0.3 in Stage 2, and K a Gaussian kernel for TI smoothing. At iteration t:   (t) (13) x̃(t) adv = T clip(x + δ , 0, 1) ,  (t)  gt = K ∗ ∇δ L x̃adv , (14) gt , (15) vt = µ vt−1 + ∥gt ∥1 + η   δ(t+1) = Πϵ (t) δ(t) − αs sign(vt ) , (16) 9

where the division is element-wise, µ=0.9 and η=10−8 prevents division by zero. The TI kernel K is 15×15 with σ=3. The step size is tied to the budget and the iteration count rather than tuned separately, αs =2ϵt /T with T =200 total iterations (110 in Stage 1, 90 in Stage 2). The nominal budget is scheduled, and the schedule enters the projection, not only the step size. The map is rebuilt each iteration as ϵ (t) = min(γ ϵt m̃, ϵmax ), with ϵt held at 1.05 ϵ through Stage 1 and annealed to 0.90 ϵ across Stage 2. The cap applies to every rebuilt map, so the schedule redistributes capacity over time and never relaxes the bound: ∥δ∥∞ ≤ ϵmax holds at every iteration. GIL. To escape local minima and encourage semantically consistent perturbation patterns, every M refinement steps we inject a generator-guided correction. The generator is budget-conditioned: the scalar budget enters as the multiplier of the dualhead composition in Eq. (6), so G(x; ϵ) denotes the same network evaluated with that multiplier and G(x) is shorthand for the nominal setting. Here it re-processes the current adversarial image with a reduced budget ϵ ′ =0.3 ϵ: (p, m) = G(xadv ; ϵ ′ ),

δ ← Πϵ (t) (δ + β (p ⊙ m̃)) ,

(17)

where β ∈ R is a scalar controlling injection strength (β=0.25 in our experiments) and m̃ is the channel-replicated mask as in Eq. (6). GIL is applied every M1 =15 iterations in Stage 1 and every M2 =25 in Stage 2; because the injection is immediately re-projected onto the budget map, its effect is confined to the allocated regions. Frequency suppression. The sign-based update of Eq. (16) moves every pixel by the same amount at every step, which concentrates the perturbation’s spectrum in the highest frequencies the grid can represent, exactly where the eye is most sensitive to texture that does not belong. Attenuating that band costs little attack strength because identity evidence is not carried there. Every 15 iterations in the later part of Stage 2 we therefore smooth δ before re-projection. A 2×2 Haar-style analysis splits δ into one low-pass and three band-pass components; the diagonal (HH) component is attenuated by a factor 0.75 and the components are recombined by nearest-neighbour upsampling. The operator is therefore a band-attenuating smoother rather than an exact wavelet reconstruction: it suppresses high-frequency structure within each 2×2 neighbourhood, which removes perceptually salient artifacts at negligible cost to attack performance. 3.5. Training Protocol We train the generator G end-to-end with a curriculum schedule designed to stabilize optimization under the bounded-perturbation constraint. Phase 1 (Escape): the identity loss is active from the first epoch together with the coverage term of Eq. (9), establishing a non-degenerate allocation; a warm-up driven purely by quality objectives instead collapses the network to a vanishing output, from which the subsequent attack phase does not recover. Phase 2 (Attack strengthening): the identity weight is raised further while coverage pressure is maintained, transitioning into the two-stage refinement described in Sec. 3.4. Phase 3 (Late-stage stabilization): ASR Lock activates, the coverage pressure is annealed toward zero so the network can withdraw budget from regions that do not need it, and the PatchGAN discriminator [38] provides naturalness regularization. 10

Algorithm 1 SSM-Attack: Mamba-Based Black-Box Face Protection K Require: Face image x, surrogates { fk }k=1 , allocation network G, nominal budget ϵ, cap ϵmax , gain γ Ensure: Protected image xadv 1: Allocation shape (single forward pass): 2: (·, m) ← G(x); m̃ ← min(1, 2 m/ maxi mi ) 3: δ ← 0 4: Stage 1 (Attack Establishment): 5: for t = 1 to T 1 do 6: ϵt ← 1.05 ϵ; ϵ (t) ← min(γ ϵt m̃, ϵmax ) 7: xadv ← clip(x + δ, 0, 1) 8: Compute Lid on surrogate ensemble { fk } 9: Update δ with DI, TI and momentum; project element-wise via Πϵ (t) (Eqs. (13)–(16)) 10: if t mod M1 = 0 then 11: Apply GIL correction, re-projected by Πϵ (t) (Eq. (17)) 12: end if 13: end for 14: Stage 2 (Quality Recovery + ASR Lock): 15: for t = T 1 + 1 to T 2 do
1 ; ϵ (t) ← min(γ ϵt m̃, ϵmax ) 16: ϵt ← ϵ 1 − 0.1 Tt−T 2 −T 1 17: xadv ← clip(x + δ, 0, 1) d t ; adapt weights (Eq. (12)) 18: Estimate ASR 19: Update δ with quality-dominant losses (Eq. (11)), including Lnat from the frozen discriminator 20: Apply wavelet HH suppression, then Πϵ (t) 21: if t mod M2 = 0 then 22: Apply GIL correction, re-projected by Πϵ (t) (Eq. (17)) 23: end if 24: end for 25: return xadv ← clip(x + δ, 0, 1)

Optimizer settings, the surrogate ensemble and the hardware are given in Section 4.1. The complete pipeline is summarized in Algorithm 1. Computational complexity. Mamba processes N=256 spatial tokens with O(N · dmodel · dstate ) complexity per block, compared to O(N 2 · dmodel ) for self-attention in Transformers [31]. Combined with a compact 5.73M-parameter allocation network, this linear-time formulation avoids both the quadratic cost of attention and the heavy pretrained backbones (100M–1B+ parameters) required by diffusion- and portraitbased pipelines. 4. Experiments 4.1. Experimental Setup We adopt the two widely used benchmarks in face privacy protection research. From CelebA-HQ [17] we take the 1,000-image subset of distinct identities used by AMT-GAN [11], DiffAM [13] and Adv-CPG [14]. Those methods split the subset into groups and redirect each toward a chosen target identity, and their reported ASR counts impersonation, cos( f (xadv ), f (xtgt )) > τ for a target image xtgt . SSMAttack instead performs untargeted identity dodging, minimizing Eq. (3) with no destination identity, over all 1,000 images under one protocol, and its ASR counts 11

cos( f (x), f (xadv )) < τ. The two rates answer different questions. Dodging is generally the easier objective, so part of our margin may reflect the task rather than attack strength, while impersonation success overstates protection, since an example that passes as the target can still match its source [20]. We report dodging because it is the quantity privacy protection requires. LADN [18] is a makeup transfer dataset comprising 333 non-makeup and 302 makeup images. We use 332 non-makeup images as the test set, processed with the same untargeted protocol and identical hyperparameters. The allocation network is trained on the Makeup Transfer (MT) dataset [39] (1,115 non-makeup and 2,719 makeup images), which AMT-GAN [11] and DiffAM [13] also use for training, combined with face images from one of CelebA-HQ or FFHQ. Which combination produces which result is stated in Section 5; in every case the evaluation set is disjoint from the training data. All face images are aligned and cropped using the multi-task cascaded convolutional network detector [40] to 112 × 112 for IR-152, IRSE-50, and MobileFace (MF), and to 160 × 160 for FaceNet following its native input specification [2], and resized to 256 × 256 for perturbation generation. The four target models span three training objectives and three backbone families. IR-152 and IRSE-50 [1] are residual networks trained with ArcFace loss, the second adding Squeeze-and-Excitation blocks. MobileFace [41] is a depthwiseseparable network trained with Softmax loss. FaceNet [2] differs on both axes: an InceptionResNet-v1 backbone trained with triplet loss on VGGFace2, which is why it is the hardest target for surrogates drawn from the other three. Verification thresholds τ follow standard practice [11, 13, 14]: 0.167, 0.241, 0.409 and 0.302 respectively, each at FAR@0.01. We employ the LOO evaluation protocol standard in this literature [11, 13, 14, 22]: for each target ft the perturbation is optimized against the remaining three models as a surrogate ensemble and scored only on ft , repeated for all four hold-out choices. The leave-one-out rule also governs training. A separate allocation network is trained for each held-out target, supervised only by that configuration’s three surrogates; the held-out model is seen neither at training nor at protection time. Four networks are therefore trained per image corpus; Table 1 states the full assignment. Attack effectiveness is reported as ASR [33] at FAR@0.01 under face verification, for each of the four held-out targets and as their mean in the “Avg.” column. Fidelity is reported with SSIM [34] and PSNR in decibels, which compare a protected image with its own original, and FID [42], which compares the protected and clean sets in a deep feature space. The first two are the primary indicators for ℓ∞ -bounded noise; FID is the more informative one for unrestricted methods that alter facial semantics. ASR is a proportion over the evaluation set, so its binomial standard error is about 1.1 points at the reported CelebA-HQ and FFHQ accuracies over 1,000 images and about 1.9 points on the 332-image LADN set; differences below roughly two points should be read accordingly. We compare against 18 published baselines in four categories — five noise-based, seven makeup-based, five facial-semantic-invariant and one portrait-based, named individually in Tables 2 and 3 [5, 7–9, 11–14, 22, 24–29, 43–45]. 12

Table 1: Protocol assignment. One allocation network per held-out target per image corpus; the held-out model supervises neither training nor refinement. MT is the Makeup Transfer corpus [39]. Evaluation images are disjoint from the images used to train the corresponding network in every row.

Test set

Held-out target

Surrogates (training and refinement)

Images for G

CelebA-HQ LADN

IR-152 IRSE-50 FaceNet MobileFace

IRSE-50, FaceNet, MobileFace IR-152, FaceNet, MobileFace FFHQ + MT IR-152, IRSE-50, MobileFace IR-152, IRSE-50, FaceNet

FFHQ

IR-152 IRSE-50 FaceNet MobileFace

IRSE-50, FaceNet, MobileFace IR-152, FaceNet, MobileFace CelebA-HQ + MT IR-152, IRSE-50, MobileFace IR-152, IRSE-50, FaceNet

Rather than mixing self-reported values obtained under differing protocols, we take their numbers from unified benchmark tables: Adv-CPG [14] for CelebA-HQ and FFHQ, DiffAM [13] for LADN. Methods absent from a benchmark are marked “–”. The generator of Section 3.2 holds 5.73M parameters (dmodel =256, dstate =16). Training follows the three-phase curriculum of Section 3.5 with AdamW [46], learning rates 2 × 10−4 for the generator and 1 × 10−4 for the discriminator, weight decay 0.01, cosine annealing and gradient clipping at 1.0. At protection time a single forward pass produces the budget map, and a two-stage refinement then fills in the values. Stage 1 runs for iterations 0–109 and Stage 2 for 110–199 with the weights of Section 3.4; Stage 2 adds, under ASR Lock, the perceptual term, the wavelet suppression and the naturalness gradient of the frozen discriminator [38]. GIL fires every 15 iterations in Stage 1 and every 25 in Stage 2. We adopt a model-adaptive perturbation budget under ℓ∞ norm to account for the varying perturbation sensitivity of different FR architectures: ϵ = 0.09 (≈23.0/255) for IR-152 and MobileFace, ϵ = 0.095 (≈24.2/255) for IRSE-50, and ϵ = 0.10 (≈25.5/255) for FaceNet. All budgets satisfy ϵ ≤ 0.10 and are enforced by the three-fold projection described in Section 3.1. Unless otherwise stated, all settings are identical across both datasets. Experiments are conducted on a single NVIDIA T4 GPU. 4.2. Comparison on Black-Box Attacks As shown in Table 2, on CelebA-HQ SSM-Attack achieves the highest average ASR among all 19 evaluated methods, surpassing the previous best Adv-CPG [14] by +7.82 points. The per-model pattern is informative. SSM-Attack reaches the highest ASR of any method on IR-152 and on FaceNet, the target generally reported as hardest: its InceptionResNet backbone and triplet-loss objective are the furthest from any of its three surrogates [2]. That the largest gain appears where the surrogate–target gap is widest suggests that where identity evidence resides transfers more readily than which 13

Table 2: Black-box ASR (%) on CelebA-HQ and LADN. FFHQ is Table 3. CelebA-HQ baselines follow Adv-CPG [14] Table 2, LADN baselines DiffAM [13] Table 1, and where both report a method we quote the higher value. ‡ Adv-CPG’s LADN results come from its supplementary Table 7. “–” marks methods absent from those tables; MF is MobileFace. Bold: best. Underline: second. CelebA-HQ LADN Category Method IR-152 IRSE-50 FaceNet MF Avg. IR-152 IRSE-50 FaceNet MF Avg. –

Clean

4.64

5.74

1.06

13.27 6.18

3.61

2.71

0.60

5.11 3.01

Noise

FGSM [5] MI-FGSM [8] PGD [7] TI-DIM [9] TIP-IM [22]

12.07 46.53 41.87 35.13 41.26

45.81 70.57 63.23 62.38 57.29

1.35 27.09 19.62 13.68 39.07

53.02 28.06 – 58.94 50.78 25.57 57.34 45.52 19.59 52.84 41.01 34.18 49.56 46.80 43.57

– 48.90 40.09 56.36 65.89

– 6.31 3.82 22.11 63.50

– – 45.01 31.45 41.09 26.15 48.30 40.24 46.48 54.86

Adv-Hat [29] Adv-Makeup [24] AMT-GAN [11] Makeup CLIP2Protect [12] GIFT [25] DFPP [26] DiffAM [13]

5.04 12.68 12.09 47.63 73.84 46.38 65.09

16.88 19.95 53.26 80.96 83.72 80.59 89.66

4.91 1.37 4.87 42.57 56.48 45.37 62.99

12.64 9.87 – 22.11 14.03 10.03 47.95 29.54 49.12 73.64 61.20 53.31 86.37 75.10 – 72.13 61.12 – 84.51 75.56 66.75

– 29.64 89.64 91.57 – – 95.66

– 0.97 32.13 47.91 – – 65.44

– – 22.38 15.76 72.43 60.83 79.94 68.18 – – – – 92.04 79.97

DiffProtect [28] DPG [43] SD4Privacy [44] Adv-Diffusion [27] P3-Mask [45]

58.64 42.89 66.89 52.84 73.47

79.34 62.47 79.96 81.67 83.40

24.69 35.83 53.49 34.95 60.24

75.91 59.65 66.42 51.90 74.58 68.73 70.78 60.06 69.64 71.69

– – – – –

– – – – –

76.96

88.72

63.50 87.95 79.28 70.56

91.46

70.42 79.54 78.00‡

SSM-Attack (Ours) 95.20

87.60

80.40 85.20 87.10 93.07

96.99

67.47 90.06 86.90

Sem. Inv.

Portrait Adv-CPG [14] SSM

– – – – –

– – – – –

– – – – –

texture directions disrupt a particular embedding. On IRSE-50 and MobileFace we remain within roughly two points of the best unrestricted methods while operating under an ℓ∞ bound those methods do not impose. Against the noise-based baselines the margin in average ASR exceeds 30 points, and against the makeup-based ones SSM-Attack leads DiffAM and CLIP2Protect on average while requiring neither a reference makeup image nor a text prompt. To validate cross-dataset generalization, we evaluate on 1,000 FFHQ [19] images following the identical protocol (Table 3). The network used here is the one trained on CelebA-HQ+MT, so FFHQ is unseen both as a dataset and as a set of identities. Even under this stricter condition SSM-Attack attains the highest average ASR (81.15%), ahead of Adv-CPG (80.02%) and DiffAM (78.02%), and ranks first on the two hardest targets, IR-152 (92.70%) and FaceNet (70.90%), evidence that the allocation transfers across architectures and across datasets. The margin is narrower than on CelebA-HQ: on IRSE-50 and MobileFace we trail the strongest unrestricted baselines by roughly ten points, consistent with FFHQ’s greater identity diversity and with the strict ℓ∞ bound; we discuss this further in Section 5. On LADN (Table 2) SSM-Attack again attains the highest average ASR among the methods with reported results, narrowly ahead of DiffAM [13], and the permodel ordering repeats: strongest on IR-152 and IRSE-50, weaker on MobileFace and FaceNet. MobileFace’s compact depthwise-separable design responds more to the large-scale semantic edits of makeup-based methods than to bounded noise, and the

14

Table 3: Black-box ASR (%) on FFHQ (1,000 images) under the LOO protocol. Baselines from AdvCPG [14] Table 2; “–” marks methods absent there. Our networks are trained on CelebA-HQ+MT and never saw FFHQ. MF is MobileFace. Bold: best. Underline: second.

Category Method

IR-152 IRSE-50 FaceNet

MF

Avg.

–

Clean

3.14

4.73

1.49

8.96

4.58

Noise

FGSM [5] MI-FGSM [8] PGD [7] TI-DIM [9] TIP-IM [22]

9.86 46.31 31.64 43.57 46.25

48.53 69.24 75.38 65.89 67.38

4.23 20.62 18.59 14.72 59.82

51.48 69.26 63.12 53.31 52.03

28.53 51.36 47.18 44.37 56.37

Makeup

AMT-GAN [11] CLIP2Protect [12] GIFT [25] DiffAM [13]

11.52 52.12 69.72 67.24

56.02 86.53 87.64 90.30

9.74 45.01 54.49 64.96

41.32 76.29 82.93 89.56

29.65 64.99 73.70 78.02

Sem. Inv.

DiffProtect [28] SD4Privacy [44] Adv-Diffusion [27] P3-Mask [45]

57.62 51.31 50.93 73.18

60.14 79.94 81.76 85.35

49.38 43.57 30.84 57.92

67.52 71.55 67.52 70.69

58.67 61.59 57.76 71.79

Portrait

Adv-CPG [14]

75.26

91.03

63.84

89.94 80.02

SSM

SSM-Attack (Ours)

92.70

79.60

70.90

81.40 81.15

FaceNet gap is amplified on a 332-image set. Across the three evaluation sets the average ASR spans 5.95 points (87.10 / 86.90 / 81.15) and SSM-Attack ranks first on each, whereas several baselines swing by more than 30 points between sets (AMT-GAN, for instance). Since each dataset is processed independently (Section 4.1), the lower FFHQ figure reflects the difficulty of its more diverse identities under a strict ℓ∞ bound, not a train/test domain shift. 4.3. Image Quality Evaluation Figure 2 provides a qualitative comparison across all three datasets, confirming that the protected images remain visually close to the originals despite the high attack success rate. Table 4 requires careful interpretation, because noise-based and unrestricted methods operate under fundamentally different quality constraints. Noise-based comparison. SSM-Attack achieves SSIM = 0.888 and PSNR = 28.42 dB on CelebA-HQ under a model-adaptive ℓ∞ perturbation budget (ϵ ≤ 0.10). Compared to TIP-IM [22] (SSIM = 0.924, PSNR = 33.21 dB), SSM-Attack accepts a moderate fidelity reduction in exchange for a dramatically higher ASR: +35.35% (87.10% vs. 46.80%). Part of that fidelity margin is expected rather than earned: TIP-IM’s published protocol adopts ϵ = 12/255 ≈ 0.047, about half the cap used here, so the ASR gap is the more informative of the two comparisons. Against the unrestricted 15

Figure 2: What the allocation network learns. Each column is a CelebA-HQ subject; rows are the original, the protected image, the budget map ϵ (brighter is more budget) and |δ| amplified 10×. The budget concentrates on the periocular–nose–mouth region and the hair and jaw boundaries. No landmark detector or segmentation mask is used at any point, so this structure is learned; an elliptical prior at comparable capacity recovers less than half the attack performance (Table 6c). Best viewed zoomed in.

methods, SSIM is higher than AMT-GAN’s [11] (0.888 vs. 0.792) and DiffAM’s [13] (0.886), and PSNR is more than 7 dB higher than either (28.42 vs. 19.50 and 20.53), which reflects the smaller mean perturbation magnitude allocation permits. Our FID (53.6) is higher than that of the diffusion-based methods (DiffAM 26.10, Adv-CPG 26.08), which follows from the paradigms rather than from tuning: semantic edits move an image along the natural image manifold and so stay distributionally aligned, whereas an additive pixel-domain perturbation can shift the feature distribution even when it is barely visible. Adv-Makeup [24] makes the point that quality alone settles nothing: it posts the best scores in the table (SSIM 0.985, FID 4.23) by editing only the eye-shadow region, and reaches 14.03% ASR. On LADN, SSM-Attack maintains consistent quality (SSIM = 0.870, PSNR = 28.10 dB), confirming that quality remains stable across datasets. 4.4. Computational Cost Analysis Table 5 compares SSM-Attack against representative baselines along two hardware-independent axes: parameters required at inference, and the inference regime. Wall-clock timings are omitted because published runtimes come from heterogeneous hardware, ranging from a 40 GB A100 to a four-GPU RTX 3090 node. For reference, on a single commodity NVIDIA T4 our complete pipeline protects

16

Table 4: Image quality on CelebA-HQ and LADN. CelebA-HQ baselines from Adv-CPG [14] Table 4. SSIM and PSNR are the primary indicators for ℓ∞ -bounded methods, FID for unrestricted ones. No competing method reports quality metrics on LADN, hence “–”; our LADN values show that fidelity is stable across datasets. † FID is omitted there because the 332-image estimator is biased upward.

Type

Method

CelebA-HQ

LADN

SSIM↑ PSNR↑ FID↓ SSIM↑ PSNR↑ FID↓ Noise

TIP-IM [22] SSM-Attack (Ours)

Adv-Makeup [24] AMT-GAN [11] Unrest. DiffAM [13] Adv-CPG [14]

0.924 0.888

33.21 28.42

38.73 53.6

– 0.870

– 28.10

– –†

0.985 0.792 0.886 0.898

34.52 19.50 20.53 30.00

4.23 34.57 26.10 26.08

– – – –

– – – –

– – – –

Table 5: Computational cost comparison. “Params” counts, in millions, every network evaluated at protection time, frozen ones included; surrogate FR models, which all methods need equally, are excluded, and “—” marks methods with no generator. Since the original papers do not report counts, they are summed from the released architectures each method depends on: a 2562 DDPM U-Net for DiffAM, CLIP ViT-B/32 with a StyleGAN2 generator for CLIP2Protect, Stable Diffusion for Adv-Diffusion and SDXL for AdvCPG, while Adv-Makeup and AMT-GAN are counted from the authors’ code. Ours is measured directly: allocation network 5.73M, discriminator 2.77M, truncated VGG-16 7.64M. § The last two serve the quality terms only, so the attack-critical footprint is 5.73M. Wall-clock timings are omitted because published values come from heterogeneous hardware.

Category Method Noise

Params Inference regime

FGSM/PGD/MI-FGSM/TI-DIM TIP-IM

Adv-Makeup AMT-GAN Makeup DiffAM CLIP2Protect

— per-image optimization — per-image optimization 0.58 amortized forward pass 12.62 forward pass, per-target training 113.67 forward pass, per-target tuning 181.65 per-image latent search

Sem. Inv.

P3-Mask DiffProtect Adv-Diffusion

— amortized mask, per user 168.49 per-image latent optimization ≈ 1290 per-image latent diffusion

Portrait

Adv-CPG

≈ 3500 per-image generation

SSM

SSM-Attack (Ours)

16.13§ amortized pass + refinement

one image in 27 s at batch size 1, which is the setting in which per-image runtimes are reported elsewhere, and 8.9 s per image when the surrogate forward passes are amortized over a batch of 16. The allocation pass itself takes 0.16 s per image. The same T4 is used for training.

17

The parameter gap is architecture-intrinsic: SSM-Attack requires 16.13M parameters in total, seven times smaller than DiffAM, eleven times smaller than CLIP2Protect and two orders of magnitude smaller than the diffusion- and portrait-based pipelines. It is not the smallest model compared: Adv-Makeup uses 0.58M parameters but perturbs only the eye-shadow region and attains 14.03% ASR, and AMT-GAN’s 12.62M generator is smaller than our full protection-time footprint, although it must be retrained for every new target identity while ours is not. The inference regime is equally consequential. Latent-search and diffusion methods solve a fresh optimization problem for every input, while AMT-GAN and DiffAM obtain fast forward-pass inference only by re-training the entire model for each new target identity, an amortized cost that per-image runtime conceals. SSMAttack amortizes generation into a single trained network and adds a short projected refinement, so the refinement budget is a tunable efficiency lever rather than a fixed cost (Table 6k). The regime also determines what each count in Table 5 contains. One-forward-pass protection carries only the generator, whatever auxiliary networks training used: AMT-GAN trains with a learned perceptual metric, but its released inference path instantiates the generator alone. Per-image optimization carries the networks that optimization evaluates, which is why the latent-search and diffusion counts include their backbones and ours includes the discriminator and the feature extractor. The comparison is if anything conservative in their favor, since we cannot audit auxiliary terms in code we did not run. The 16.13M figure (16,129,574 parameters) is the honest protection-time total and splits into three parts: the 5.73M allocation network, the 2.77M PatchGAN discriminator, and the 7.64M truncated VGG-16 feature extractor. The latter two are frozen but are evaluated during Stage 2, for the naturalness gradient of Eq. (10) and the perceptual term of Section 3.3, so by the accounting rule of Table 5 they belong in the count. Because both serve only the quality terms, a deployment that needs the attack alone can run the single-pass variant of Table 6(k) on the 5.73M allocation network, at 0.16 s per image. 4.5. Ablation Study Table 6 isolates the contribution of the allocation mechanism. Allocation shifts the attack–fidelity–budget trade-off. Replacing the uniform map with the SSM-predicted allocation (b) improves SSIM by +0.044, PSNR by +1.62 dB and FID by −11.2 relative to (a), while spending only 37% of the budget a uniform allocation would use, at a cost of 1.84 points of ASR. Because the saved budget can be reinvested through the allocation gain without violating the cap, the operating point (j) reported in Table 2 attains 87.10% ASR with every quality metric still ahead of uniform allocation. Under a fixed ℓ∞ bound, where the budget is spent therefore matters more than how much is spent. The allocation is learned, not geometric. Row (c) replaces the learned map with a fixed elliptical face prior at comparable allocated capacity. ASR collapses to 39.65%, less than half the learned allocation, even though pixel-fidelity metrics improve: the prior simply spends less capacity, and spends it in a less useful place. Quantitatively the learned mask is close to binary: 61.9% of pixels fall below 0.15 of the cap and 35.9% 18

Table 6: Ablation of the budget-allocation mechanism on CelebA-HQ, averaged over the four leave-oneout configurations. Every row except (k) shares the refinement pipeline and the cap ϵmax = 0.10; only the budget map differs. “Budget” is the mean of ϵ relative to uniform allocation, i.e. allocated capacity, not the realised ∥δ∥. Rows marked ∗ use the full 1,000-image test set; group (A) a common 150-image subset and groups (B) and (E) a common 300-image subset. † Row (k) uses the pre-refinement checkpoints, so it indicates what a purely amortized variant delivers rather than being a controlled ablation of (j).

Configuration

ASR↑ SSIM↑ PSNR↑ FID↓ Budget

(A) Budget map (a) Uniform, ϵi ≡ ϵmax (b) SSM allocation (c) Elliptical face prior (d) SSM allocation + generator init

88.17 86.33 39.65 73.83

0.8505 0.8943 0.9455 0.8752

27.47 75.7 100% 29.09 64.5 37% 33.53 29.2 17% 28.13 109.1 37%

(B) Allocation gain γ (IR-152) (e) γ = 1.0 (f) γ = 1.5 (g) γ = 2.5 (h) γ = 4.0

90.33 97.67 96.33 96.67

0.8910 0.8808 0.8795 0.8789

29.03 28.16 28.09 28.07

60.3 67.2 66.6 68.0

40% 46% 47% 47%

(C) Allocation gain, full test set (i) γ = 1.0 ∗ (j) Full pipeline, γ=1.5∗

80.90 0.8957 87.10 0.8878

29.11 28.42

48.8 53.6

39% 42%

(D) Refinement (k) No refinement, single generator pass∗† 70.00 0.8670

27.00 150.6

—

(E) Allocation backbone (300-image subset) (l) Mamba, raster scan (ours) 87.75 0.8851 (m) Mamba, shuffled token order 83.33 0.8798

28.37 28.37

38% 39%

63.7 63.5

exceed 0.60 (standard deviation 0.466), whereas a mask trained with only a mean-value regularizer degenerates to a spatially flat map (standard deviation 0.059) and behaves indistinguishably from (a). The coverage and selectivity terms of Section 3.3 prevent this degeneration. Where the budget goes. Over 200 CelebA-HQ images, a central face ellipse covering 38.7% of the frame receives 74.4% of the allocated capacity (1.93× its area share) and the periocular–nose–mouth region, covering 12.1%, receives 25.9% (2.14×). The remaining quarter follows the hair boundary and jaw silhouette rather than spreading over the background, consistent with FR embeddings encoding outline cues. The network should allocate, not synthesize. Row (d) additionally initialises δ from a generator-synthesized perturbation. ASR drops by 12.50 points and FID increases by 44.6. Spectral analysis of the synthesized output reveals a periodic component at the 16-pixel period of the token grid; replacing transposed convolutions with bilinear upsampling reduces this peak from 7.96× to 0.66× the neighbouring bands, yet the

19

Figure 3: Budget allocation under a fixed ℓ∞ cap. (a) Allocation gain sweep on the IR-152 configuration; the dashed line marks uniform allocation at the same cap. We fix γ = 1.5, the knee of the curve, for all reported results. (b) Per-target comparison at γ=1.5: learned allocation beats uniform on the two hardest targets while using 42% of its capacity, whereas an elliptical prior at comparable capacity recovers less than half.

residual image-independent structure still degrades the distributional metric, whereas gradient-determined perturbation values do not. Allocation gain and refinement budget. Group (B) sweeps γ on the IR-152 configuration (Fig. 3a): ASR rises sharply from γ=1.0 to γ=1.5 and then saturates, because the near-binary mask recruits no additional pixels once the cap binds; we fix γ = 1.5 throughout. Group (C) repeats the two endpoints over all four configurations on the full test set: γ=1.0 trades 6.20 points of ASR for the best quality of any refined configuration (SSIM 0.8957, PSNR 29.11, FID 48.8), so the gain is itself an operatingpoint control. Group (D) drops the refinement and protects with a single generator pass in 0.16 s per image (≈ 170× faster at batch size 1). It keeps 70.00% ASR, four fifths of the full pipeline, but FID rises to 150.6: without gradient-determined values the perturbation carries the image-independent structure diagnosed in row (d), undiluted. An amortized variant is therefore viable where latency dominates, while refinement is what makes the output distributionally acceptable. The token ordering is load-bearing. Group (E) applies a fixed random permutation to the token sequence at protection time with every weight unchanged; were the ordered scan uninformative the result would not move. ASR falls by 4.42 points over the pooled 1,200 protections (300 images per target; p = 0.002) at unchanged PSNR and FID, so nothing is traded for quality. The loss is concentrated rather than uniform: the other three targets lose 1.78 points together, which the data do not separate from zero (p = 0.24), whereas FaceNet loses 12.33 (p = 0.0004, surviving Bonferroni correction for four comparisons). FaceNet is also the configuration in which the target is furthest from its surrogates, being the only InceptionResNet backbone with a triplet objective while its three surrogates are margin- or softmax-trained residual networks; ordering therefore appears to matter where transfer is hardest, though a single target does not establish that as a general rule. Two caveats bound the claim. Because the permutation also disrupts the causal convolution inside the Mamba block, the evidence supports the ordered token sequence as a whole rather than the state-space recurrence in isolation. 20

And because both arms are evaluated on the same images, the observations are paired; we report the unpaired two-proportion test, which is the more conservative of the two. We use this permutation rather than a comparison against convolutional or attention-based allocators because it isolates the property in question. Substituting a different backbone changes architecture, parameter count, optimization difficulty and tuning effort at the same time, so any difference it produces cannot be attributed to sequence modeling alone, and a control trained with less care than the proposed model would bias the comparison in our own favor. The permutation changes exactly one thing: the order in which the same weights visit the same tokens. Weights, capacity, training data, refinement schedule and budget cap are identical across the two arms, and the permutation is applied only at protection time. What the experiment therefore establishes is that the ordered token sequence carries information the allocation uses, not that a selective SSM allocator outperforms a convolutional or attention-based one; that comparison remains open. 5. Discussion and Limitations SSM-Attack closes the transferability gap between noise-based and unrestricted methods through allocation rather than capacity: a 5.73M-parameter allocation network applying strictly ℓ∞ -bounded noise attains the highest average ASR on all three evaluation sets while spending under half the budget of a uniform map at the same cap. The mechanism is interpretable. Modern FR systems encode identity as a holistic spatial direction [1], and a selective state update carries context along the token sequence in one pass, so a decision made late in the scan can draw on far-apart regions of the face; Table 6(E) is the direct evidence. The untargeted objective also spends the whole budget on identity suppression, whereas targeted attacks often dodge far worse than they impersonate [20]. Cross-dataset generalization. The CelebA-HQ and LADN results come from networks trained on FFHQ+MT, disjoint from both evaluation sets, one per heldout target (Table 1). Networks trained instead on CelebA-HQ+MT attain 85.99% on LADN, 0.91 points below, and 81.15% on the unseen FFHQ corpus (Table 3). The allocation therefore transfers across datasets; establishing dataset-independence would need more corpora than the three evaluated here. Perturbation budget. The per-model budgets are set by the surrogate ensemble, not by knowledge of the held-out target; a universal ϵ=0.10 can be used in deployment. LADN protocol. Unlike CelebA-HQ, LADN has no shared evaluation subset, and for the same method DiffAM [13] and the supplementary material of Adv-CPG [14] report LADN scores differing by up to 19 points on individual FR models. We therefore state our protocol (332 non-makeup images, hyperparameters identical to CelebA-HQ) and, where sources disagree, quote the higher value, so competing methods appear at their strongest published numbers. Several limitations remain. Transferability depends on surrogate diversity and may degrade against fundamentally new FR paradigms, and our evaluation is restricted to static images and the four FR models of the standard benchmark; commercial

21

recognition services, which several compared methods report, are not evaluated. Projected refinement adds cost over the single allocation pass; its iteration count is a deployment-time lever (Table 6k). FID remains above manifold-editing methods: allocation reduces it by 11.2 points relative to a uniform map but does not close a gap that is intrinsic to additive perturbation. The scan is unidirectional and follows a raster order, and the significant ordering effect of Section 4.5 is confined to one held-out target, so neither the choice of traversal nor the generality of the ordering effect is settled here. Finally, naturalness is imposed by loss terms rather than structurally, as in manifold-editing methods; a structural formulation would remove the frozen perceptual network from the protection-time footprint. Future work could explore bidirectional spatial modeling [32], defense-aware training against purification, surrogate diversification [21], and allocation for video, where the map must stay temporally consistent. 6. Conclusion We introduced SSM-Attack, the first adversarial face privacy framework based on Selective State Space Models, which uses the linear-time scan of Mamba to decide where a strict ℓ∞ budget is spent across the face. With an allocation network of only 5.73M parameters, it achieves the highest average ASR on CelebA-HQ (87.10%), LADN (86.90%) and FFHQ (81.15%), which suggests that learned budget allocation is a general design principle for bounded adversarial attacks. CRediT authorship contribution statement Tunahan Parlayıcı: Conceptualization, Methodology, Software, Validation, Investigation, Data curation, Visualization, Writing – original draft. Erchan Aptoula: Supervision, Writing – review & editing. Yücel Saygın: Supervision, Funding acquisition, Writing – review & editing. Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. Data availability Source code and pretrained models will be released at https://github.com/ 7unahan/PrivacySSM upon acceptance. All datasets used are publicly available. Funding This work was supported by the Scientific and Technological Research Council of Türkiye (TÜBİTAK) under the 2244 Industrial Ph.D. Program. The funder had no role in study design, analysis, the decision to publish, or preparation of the manuscript. 22

Declaration of generative AI and AI-assisted technologies in the manuscript preparation process During the preparation of this work the authors used Claude (Anthropic) and Grammarly in order to improve the readability and language of the manuscript, and Claude to assist with the draw.io layout scripts for the graphical abstract, which the authors designed and edited. After using these tools, the authors reviewed and edited the content as needed and take full responsibility for the content of the published article. References [1] J. Deng, J. Guo, N. Xue, S. Zafeiriou, ArcFace: Additive angular margin loss for deep face recognition, in: Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Long Beach, CA, USA, 2019, pp. 4690–4699. [2] F. Schroff, D. Kalenichenko, J. Philbin, FaceNet: A unified embedding for face recognition and clustering, in: Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Boston, MA, USA, 2015, pp. 815–823. [3] V. Cherepanova, M. Goldblum, H. Foley, S. Duan, J. P. Dickerson, G. Taylor, T. Goldstein, LowKey: Leveraging adversarial attacks to protect social media users from facial recognition, in: Proc. Int. Conf. Learn. Represent. (ICLR), Virtual, 2021. [4] S. Shan, E. Wenger, J. Zhang, H. Li, H. Zheng, B. Y. Zhao, Fawkes: Protecting privacy against unauthorized deep learning models, in: Proc. USENIX Secur. Symp., Boston, MA, USA, 2020, pp. 1589–1604. [5] I. J. Goodfellow, J. Shlens, C. Szegedy, Explaining and harnessing adversarial examples, in: Proc. Int. Conf. Learn. Represent. (ICLR), San Diego, CA, USA, 2015. [6] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, R. Fergus, Intriguing properties of neural networks, in: Proc. Int. Conf. Learn. Represent. (ICLR), Banff, Canada, 2014. [7] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, A. Vladu, Towards deep learning models resistant to adversarial attacks, in: Proc. Int. Conf. Learn. Represent. (ICLR), Vancouver, Canada, 2018. [8] Y. Dong, F. Liao, T. Pang, H. Su, J. Zhu, X. Hu, J. Li, Boosting adversarial attacks with momentum, in: Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Salt Lake City, UT, USA, 2018, pp. 9185–9193. [9] Y. Dong, T. Pang, H. Su, J. Zhu, Evading defenses to transferable adversarial examples by translation-invariant attacks, in: Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Long Beach, CA, USA, 2019, pp. 4312–4321. [10] C. Xie, Z. Zhang, Y. Zhou, S. Bai, J. Wang, Z. Ren, A. L. Yuille, Improving transferability of adversarial examples with input diversity, in: Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Long Beach, CA, USA, 2019, pp. 2730–2739. [11] S. Hu, X. Liu, Y. Zhang, M. Li, L. Y. Zhang, H. Jin, L. Wu, Protecting facial privacy: Generating adversarial identity masks via style-robust makeup transfer, in: Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), New Orleans, LA, USA, 2022, pp. 15014–15023. [12] F. Shamshad, M. Naseer, K. Nandakumar, CLIP2Protect: Protecting facial privacy using text-guided makeup via adversarial latent search, in: Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Vancouver, Canada, 2023, pp. 20595–20605.

23

[13] Y. Sun, L. Yu, H. Xie, J. Li, Y. Zhang, DiffAM: Diffusion-based adversarial makeup transfer for facial privacy protection, in: Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Seattle, WA, USA, 2024, pp. 24584–24594. [14] J. Wang, H. Zhang, Y. Yuan, Adv-CPG: A customized portrait generation framework with facial adversarial attacks, in: Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Nashville, TN, USA, 2025, pp. 21001–21010. [15] A. Gu, K. Goel, C. Ré, Efficiently modeling long sequences with structured state spaces, in: Proc. Int. Conf. Learn. Represent. (ICLR), Virtual, 2022. [16] A. Gu, T. Dao, Mamba: Linear-time sequence modeling with selective state spaces, in: Proc. Conf. Lang. Model. (COLM), Philadelphia, PA, USA, 2024. [17] T. Karras, T. Aila, S. Laine, J. Lehtinen, Progressive growing of GANs for improved quality, stability, and variation, in: Proc. Int. Conf. Learn. Represent. (ICLR), Vancouver, Canada, 2018. [18] Q. Gu, G. Wang, M. T. Chiu, Y.-W. Tai, C.-K. Tang, LADN: Local adversarial disentangling network for facial makeup and de-makeup, in: Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Seoul, South Korea, 2019, pp. 10481–10490. [19] T. Karras, S. Laine, T. Aila, A style-based generator architecture for generative adversarial networks, in: Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Long Beach, CA, USA, 2019, pp. 4401–4410. [20] F. Zhou, Q. Zhou, B. Yin, H. Zheng, X. Lu, L. Ma, H. Ling, Rethinking impersonation and dodging attacks on face recognition systems, in: Proc. ACM Int. Conf. Multimedia (ACM MM), Melbourne, Australia, 2024, pp. 2487–2496. [21] F. Zhou, B. Yin, H. Ling, Q. Zhou, W. Wang, Improving the transferability of adversarial attacks on face recognition with diverse parameters augmentation, in: Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Nashville, TN, USA, 2025, pp. 3516–3527. [22] X. Yang, Y. Dong, T. Pang, H. Su, J. Zhu, Y. Chen, H. Xue, Towards face encryption by generating adversarial identity masks, in: Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Montreal, Canada, 2021, pp. 3897–3907. [23] J. Liu, W. Zhang, K. Fukuchi, Y. Akimoto, J. Sakuma, Unauthorized AI cannot recognize me: Reversible adversarial example, Pattern Recognition 134 (2023) 109048. [24] B. Yin, W. Wang, T. Yao, J. Guo, Z. Kong, S. Ding, J. Li, C. Liu, Adv-Makeup: A new imperceptible and transferable attack on face recognition, in: Proc. Int. Joint Conf. Artif. Intell. (IJCAI), Montreal, Canada, 2021, pp. 1252–1258. [25] M. Li, J. Wang, H. Zhang, Z. Zhou, S. Hu, X. Pei, Transferable adversarial facial images for privacy protection, in: Proc. ACM Int. Conf. Multimedia (ACM MM), Melbourne, Australia, 2024, pp. 10649–10658. [26] F. Shamshad, M. Naseer, K. Nandakumar, Makeup-guided facial privacy protection via untrained neural network priors, in: Computer Vision – ECCV 2024 Workshops, Vol. 15644 of Lect. Notes Comput. Sci., Springer, Cham, 2025, pp. 227–246. [27] D. Liu, X. Wang, C. Peng, N. Wang, R. Hu, X. Gao, Adv-Diffusion: Imperceptible adversarial face identity attack via latent diffusion model, in: Proc. AAAI Conf. Artif. Intell. (AAAI), Vancouver, Canada, 2024, pp. 3585–3593. [28] J. Liu, C. P. Lau, Z. Guo, Y. Guo, Z. Wang, R. Chellappa, DiffProtect: Generative adversarial examples using diffusion models for facial privacy protection, Pattern Recognition 173 (2026) 112780. [29] S. Komkov, A. Petiushko, AdvHat: Real-world adversarial attack on ArcFace face ID system, in: Proc. Int. Conf. Pattern Recognit. (ICPR), Milan, Italy, 2021, pp. 819–826. [30] X. Liu, Y. Zhong, X. Cui, Y. Zhang, P. Li, W. Deng, AdvCloak: Customized adversarial cloak for privacy protection, Pattern Recognition 158 (2025) 111050.

24

[31] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, N. Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale, in: Proc. Int. Conf. Learn. Represent. (ICLR), Virtual, 2021. [32] L. Zhu, B. Liao, Q. Zhang, X. Wang, W. Liu, X. Wang, Vision Mamba: Efficient visual representation learning with bidirectional state space model, in: Proc. Int. Conf. Mach. Learn. (ICML), Vol. 235 of PMLR, Vienna, Austria, 2024, pp. 62429–62442. [33] M. Sharif, S. Bhagavatula, L. Bauer, M. K. Reiter, A general framework for adversarial examples with objectives, ACM Trans. Priv. Secur. 22 (3) (2019) 1–30. [34] Z. Wang, A. C. Bovik, H. R. Sheikh, E. P. Simoncelli, Image quality assessment: From error visibility to structural similarity, IEEE Trans. Image Process. 13 (4) (2004) 600–612. [35] J. Johnson, A. Alahi, L. Fei-Fei, Perceptual losses for real-time style transfer and superresolution, in: Proc. Eur. Conf. Comput. Vis. (ECCV), Amsterdam, Netherlands, 2016, pp. 694–711. [36] K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, in: Proc. Int. Conf. Learn. Represent. (ICLR), San Diego, CA, USA, 2015. [37] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, O. Wang, The unreasonable effectiveness of deep features as a perceptual metric, in: Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Salt Lake City, UT, USA, 2018, pp. 586–595. [38] P. Isola, J.-Y. Zhu, T. Zhou, A. A. Efros, Image-to-image translation with conditional adversarial networks, in: Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Honolulu, HI, USA, 2017, pp. 1125–1134. [39] T. Li, R. Qian, C. Dong, S. Liu, Q. Yan, W. Zhu, L. Lin, BeautyGAN: Instance-level facial makeup transfer with deep generative adversarial network, in: Proc. ACM Int. Conf. Multimedia (ACM MM), Seoul, South Korea, 2018, pp. 645–653. [40] K. Zhang, Z. Zhang, Z. Li, Y. Qiao, Joint face detection and alignment using multitask cascaded convolutional networks, IEEE Signal Process. Lett. 23 (10) (2016) 1499–1503. [41] S. Chen, Y. Liu, X. Gao, Z. Han, MobileFaceNets: Efficient CNNs for accurate real-time face verification on mobile devices, in: Proc. Chin. Conf. Biometric Recognit. (CCBR), Urumqi, China, 2018, pp. 428–438. [42] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, S. Hochreiter, GANs trained by a two time-scale update rule converge to a local Nash equilibrium, in: Adv. Neural Inf. Process. Syst. (NeurIPS), Long Beach, CA, USA, 2017, pp. 6626–6637. [43] Y. Zhang, D. Ye, S. Shen, C. Xie, Z. Liu, J. Deng, L. Tang, Double privacy guard: Robust traceable adversarial watermarking against face recognition, arXiv preprint 2404.14693v1 (2024). [44] J. An, W. Zhang, D. Wu, Z. Lin, J. Gu, W. Wang, SD4Privacy: Exploiting stable diffusion for protecting facial privacy, in: Proc. IEEE Int. Conf. Multimedia Expo (ICME), Niagara Falls, Canada, 2024, pp. 1–6. [45] K.-H. Chow, S. Hu, T. Huang, L. Liu, Personalized privacy protection mask against unauthorized facial recognition, in: Proc. Eur. Conf. Comput. Vis. (ECCV), Milan, Italy, 2024, pp. 434–450. [46] I. Loshchilov, F. Hutter, Decoupled weight decay regularization, in: Proc. Int. Conf. Learn. Represent. (ICLR), New Orleans, LA, USA, 2019.

25

Related documents

• SSM-Attack: Learning Where to Perturb for Black-Box Face Privacy#1342199
• Saving Face on Facebook: Privacy Concerns, Social Benefits, and Impression Management#300188
• How Private is Private? A Comparative Study for Face De-Identification#1234701
• The Other-Race Effect in Implicit Memory for Face Combinations#1179164
• Resisting Face Surveillance with Copyright Law#1121645