The Platonic Defense: Backdoor Defense for Self-Supervised Encoders in the Era of Large Scale Pre-training

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

The Platonic Defense: Backdoor Defense for Self-Supervised Encoders in the Era of Large Scale Pre-training

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

cryptography, security, privacy, cybersecurity

arXiv:2606.29451v1 [cs.CV] 28 Jun 2026

The Platonic Defense: Backdoor Defense for Self-Supervised Encoders in the Era of Large Scale Pre-training

Tuo Chen1,2 Minjing Dong3 Benlei Cui4 Jian Liu2∗ Jie Gui1,5∗ 1 Southeast University 2 Ant Group 3 City University of Hong Kong 4 Alibaba Group 5 Purple Mountain Laboratories [email protected]

Abstract Self-supervised learning (SSL) pretrained models have become a dominant paradigm for visual representation learning, but they are vulnerable to backdoor attacks. Existing defenses struggle to defend against such attacks in a fully blackbox setting because they often require access to labels, attack patterns, or training data. To tackle this issue, we propose a new attack-agnostic, model-agnostic, and modality-agnostic black-box test-time defense paradigm, called Platonic Representation Defense. It is inspired by the Platonic Representation Hypothesis, which suggests that large-scale independently trained encoders converge toward compatible projections of the same underlying reality. We formalize this idea as a conditional energy function defined over source representations and a set of reference representations. The energy function is trained for detection through noisecontrastive estimation and for representation purification through denoising score matching. Theoretically, the energy gap between matched and mismatched samples is lower bounded by the mutual information between source and reference representations. We demonstrate the effectiveness of our method on multiple selfsupervised encoders and more than 10 attacks. The method can perform both representation detection and purification, and achieves substantial performance gains across multiple attacks. Code is available here.

Introduction

Artificial intelligence systems are rapidly moving toward the era of large-scale pretraining. Earlier computer vision systems often rely on task-specific solutions for image classification [18], semantic segmentation [75] and other tasks. Modern large models can handle all these tasks with a single set of weights [52, 49], which are built on general SSL encoders that inherit strong representation ability and basic understanding of the world from pretraining [38, 49]. Despite great success, SSL encoders are vulnerable to backdoor attacks [34, 5, 56, 27, 7], which severely undermine their reliability and safety on downstream tasks. An attacker can manipulate model representations by injecting malicious samples into unlabeled training data or by modifying model weights [5, 27]. The compromised encoder behaves normally on clean inputs but exhibits malicious behavior on inputs that contain a trigger pattern. Since such encoders often serve as the backbone for a wide range of downstream tasks, developing effective backdoor defenses is essential for artificial intelligence systems in the era of large-scale pretraining [38]. Although many defense strategies have been proposed to mitigate backdoor attacks on SSL encoders, only a few operate in a fully black-box setting. Some studies construct pseudo-labels and ∗ Corresponding authors.

Preprint.

The Platonic Representation Defense Representations from different views and encoders should converge to a shared latent reality. Significant disagreement suggests manipulation or a backdoor attack.

View 3 A green cube next to a yellow cylinder.

Encoder 𝑓!

Encoder 𝑓"

Encoder 𝑓#

𝑧"

𝑧#

1) Normal case: agreement across representations

𝑧!

Compare / aggregate representations

low disagreement / consistent

2) Manipulated case:

Encoder 𝑓!

Encoder 𝑓"

Encoder 𝑓#

𝑧!

𝑧"

𝑧#

disagreement indicates backdoor

Consistent projections of shared reality

high disagreement → suspected backdoor attack

Compare / aggregate representations

Backdoored representation

Representation disagreement → Evidence of manipulation

Figure 1: The Platonic Representation Defense: Representations (z1 , z2 , z3 ) are projections of a shared underlying reality. We conjecture that backdoor attacks will deviate from this pattern. reformulate the problem under the framework of supervised backdoor defense [65, 24], which only applies to a closed and finite label space. Other methods restrict the defense scope to specific types of backdoor triggers [20, 65] or multimodal backdoor attacks [3]. Their defense capability depends on access to the model’s internal structure or the upstream training process. Such knowledge is difficult to obtain in advance in SSL settings, since SSL models serve a broad range of downstream users who typically cannot access the upstream pretraining data or training procedure. To make defenses attack-agnostic, model-agnostic, and modality-agnostic, we propose a new framework that we call the Platonic Representation Defense. The name is borrowed from the Platonic Representation Hypothesis [25], which hypothesizes that representations produced by different encoders are projections of one common underlying reality and that large-scale pre-training drives these projections toward a shared limit. The compatibility between representations can thus be a good indicator of backdoor attacks if the hypothesis is true. We illustrate this intuition in Figure 1. We formalize this intuition as a conditional energy function Eθ z s , z r1:K , where z s is the output of a source encoder under inspection and z r1:K are the outputs of K trusted reference encoders on the same input. The energy is low on clean (compatible) samples and rises sharply on backdoored (incompatible) ones, recasting backdoor defense as a problem solved entirely in representation space, without any downstream labels and without access to poisoned samples. We train this energy with Noise Contrastive Estimation (NCE), which sidesteps the intractable partition function while delivering a calibrated detection score. To additionally support purification, we train a denoiser of the source representation conditioned on the reference representations via denoising score matching, by which Tweedie’s identity recovers the score field of the same energy. Both objectives share a single architecture and only diverge at the output head and the loss. Theoretically, we show that when Eθ is close to the Bayes optimal, the energy gap between the matched (clean) and the mismatched (backdoor) distributions is lower bounded by the mutual information I Z s ; Z r1:K between the source and reference representations. Under the Platonic Representation Hypothesis, independently trained encoders retain shared information, which gives clean representations a systematically lower energy than mismatched ones. Our main contributions can be summarized as follows. (1) To our knowledge, this is the first work to connect representation convergence with backdoor defense, identifying the convergence of independently trained encoders as a previously unused security signal that is intrinsic to large-scale 2

pre-training. (2) We introduce the platonic representation defense, a black-box test-time framework that formalizes multi-encoder compatibility as a conditional energy model and realizes it through two complementary training routes, discriminative NCE for detection and conditional denoising score matching for purification. (3) We provide both theoretical and empirical evidence. We validate our method on diverse SSL encoders against more than 10 backdoor attacks. Our framework preserves clean sample performance well, which makes it better suited to black-box settings.

Preliminaries

We focus on the visual representations, leaving the diagnosis of representations in other modalities for future work. A visual representation is defined as a function f : X → Z, which maps an input image x ∈ X to a latent vector f (x) ∈ Z. We characterize visual representations by their induced similarity structure, referred to as the kernel. Evaluating visual representations via kernels is a widely adopted approach [25, 70, 31], which can be justified by the fact that the similarity structure between two representations is invariant to the selection of feature space. In our experiments, we mainly use the kernel-alignment metric Center-Kernel Nearest Neighbor Agreement (CKNNA) [25] to quantify the similarity between two representations. Representation convergence. A growing body of work shows that, as model capacity and training data scale up, independently trained models converge toward increasingly similar kernels. The Plato Representation Hypothesis [25, 30] formalizes this observation: the kernel-alignment between two models grows monotonically with their capability, irrespective of architecture, training objective, or even input modality, suggesting that sufficiently capable models recover a shared statistical structure underlying visual data. Earlier studies of cross-architecture and layer-wise similarity [31, 53, 46] report the same trend at smaller scales, and recent evidence further suggests that the shared kernel reflects intrinsic structure of the visual world rather than arbitrary learned conventions [4, 1]. Energy-based models and EDM parameterization. An energy-based model (EBM) [33] parameterizes a probability density through a scalar energy function Eθ : Z → R as Z 1 pθ (z) = exp −Eθ (z) , Z(θ) = exp −Eθ (z) dz, (1) Z(θ) trading the normalization constraint of standard likelihood models for a free-form energy landscape. The partition function Z(θ) is intractable, so we bypass it with noise-contrastive estimation (NCE) [15] and score matching [26] which fits score −∇z Eθ (z) without evaluating Z(θ). We adopt the practical EDM [29] variant of score matching used by modern diffusion models [59, 42]. For a clean data point z ∼ pdata , a Gaussian perturbation ϵ ∼ N (0, I), and a noise level σ > 0, define Z z̃ = z + σ ϵ, pσ (z̃) = N (z̃; z, σ 2 I) pdata (z) dz. (2) Rather than regressing on ∇z̃ log pσ directly, score matching learns a denoiser D̂θ (z̃, σ) ∈ Z that predicts the clean z from z̃, trained by the EDM-weighted regression h i 2 LDSM (θ) = Eσ, z, ϵ λ(σ) D̂θ z + σϵ, σ − z 2 . (3) Its optimum is the Bayes-optimal denoiser D̂⋆ (z̃, σ) = E[z | z̃, σ], and Tweedie’s identity ties it back to the score: ∇z̃ log pσ (z̃) = D̂⋆ (z̃, σ) − z̃ σ 2 , (4) so any denoiser trained with Eq. (3) directly induces a score estimator sθ (z̃, σ) = (D̂θ (z̃, σ)− z̃)/σ 2 that recovers −∇z̃ Eθ on the Gaussian-smoothed EBM. Following the EDM parameterization [29], we do not have the network predict z from z̃ directly; instead we expose a residual subnetwork Fθ and apply fixed σ-dependent preconditioning, D̂θ (z̃, σ) = cskip (σ) z̃ + cout (σ) Fθ cin (σ) z̃, cnoise (σ) , (5) with coefficients chosen so that both the input and the regression target of Fθ have unit variance at every σ [29]: σ2

data cskip (σ) = σ2 +σ , 2 data

cout (σ) = √σ2σdata2

σ +σdata

cin (σ) = √ 2 1 2

σ +σdata

σ 2 +σ 2

λ(σ) = σ2 σ2data , (6) data

Figure 2: Cross-model representation alignment on ImageNet-1K. LEFT: Pairwise CKNNA scores with 10 neighbors among 8 diverse independently trained models. All discriminative models exhibit substantial mutual alignment despite differing architectures and training objectives. SDVAE [54] serves as comparision. RIGHT: UMAP of the jointly embedded relational representations after per-model centering and row-ℓ2 normalization. Samples from different models cluster primarily by semantic class rather than by model identity, providing a geometric view of the convergence. and cnoise (σ) = 41 log σ, supplied to Fθ as a sinusoidal time embedding [67] so that the noise scale enters the network as an additive token-level signal. Here σdata is the empirical standard deviation of z after a per-dimension standardization step that we apply to every encoder output before training.

The Platonic Representation Defense

In this section, we evaluate the aforementioned phenomenon of representation convergence and propose an energy-based modeling framework to model the compatibility of representations. Threat model. The attacker controls upstream pre-training and releases a backdoored encoder f˜ that behaves normally on clean inputs (f˜(x) ≈ f (x)) but, on any input carrying a predefined trigger, maps the image into the embedding cluster of an attacker-chosen target class [27, 5, 56, 37]. We consider diverse backdoor families in existing literature including additive image-space patches [56, 5], optimization backdoor [61], and frequency-domain backdoor [34]. The defender accesses neither the upstream pipeline nor the training data. The defender can access publicly available encoders such as DINOv2 [50] and CLIP [52]. The goal is to build test-time detectors and purifiers for poisoned representations without degrading clean-input utility. We additionally evaluate on adversarial attacks [12, 44] to further demonstrate the effectiveness of our method. Empirical observation. We investigate four families of models that span different training paradigms and modalities: (i) Visual SSL encoders (e.g., DINOv2 [50]): self-supervised models that produce semantically rich representations without label supervision; (ii) Vision–language encoders (e.g., CLIP [52]): contrastive models that align visual and textual modalities in a shared embedding space; (iii) Multi-modal Large Language Models (MLLMs) (e.g., Qwen2-VL [69]): Large language models with visual encoders; (iv) Generative Encoders (e.g., AIMv2 [11]): Generative encoders trained with autoregressive pre-training [11] or MAE-style pre-training [19]. We extracted a global representation for Qwen2-VL following VLM2Vec [28]. As shown in Figure 2, substantial alignment emerges among all models. Appendix Figure 6 shows that randomly initialized counterparts of the same architectures exhibit near-zero alignment (<0.01 CKNNA), confirming that alignment arises from learning other than architectural inductive bias. This convergence has a security implication that, to our knowledge, has not been exploited. If two independently trained models agree on what an image “means,” then disagreement is evidence of manipulation. We operationalize this intuition by modeling the joint compatibility of a source representation and a fixed ensemble of K clean reference representations as an energy function. Given a source encoder f s (potentially compromised) and K independently trained reference encoders {f rk }K k=1 drawn 4

from the families above, we extract z s = f s (x) ∈ Rds and z rk = f rk (x) ∈ Rdrk , and introduce a scalar platonic energy Eθ z s , z r1:K ∈ R (7) that assigns each tuple a real-valued compatibility score. Following the standard energy-based modeling convention [33], Eθ induces an unnormalized conditional density over the source latent, (8) pθ z s z r1:K ∝ exp −Eθ (z s , z r1:K ) , whose low-energy region captures tuples in which the source and reference representations agree. The reference representations act as clean, uncorruptible conditions. Once trained, Eθ provides two complementary primitives for defense. Detection: for a query image x with observed source latent z s = f˜s (x) and clean reference features z r1:K , the energy Eθ (z s , z r1:K ) serves directly as a scalar anomaly score. Purification: we can follow −∇zs Eθ to pull an observed latent toward this lowenergy region, yielding a purified estimate ẑ s that is consistent with the references. In Section 4, we elaborate on learning an Eθ that supports these two uses. Detection guarantee from representation convergence. Before turning to training, we show that an Eθ that is close to the Bayes-optimal platonic energy inherits a guarantee directly from the representation convergence. Let PSR denote the distribution of matched tuples (Z s , Z r1:K ) extracted from the same image, and let P⊥ := PS ⊗ PR denote the distribution of cross-sample mismatched tuples in which Z s is drawn independently of the references. Define the oracle energy E ⋆ z s , z r1:K = − log p z s z r1:K + ψ z r1:K , (9) which is the Bayes-optimal target of Eq. (8) up to a reference-only shift ψ [33]. Then we characterize the detection behavior of E ⋆ and any Eθ that approximates it in mean. Proposition 1 (Oracle energy gap). Assume PS ≪ PS|R=r for PR -a.e. r and the KL terms below are finite. Then the oracle energy E ⋆ satisfies h i EP⊥ E ⋆ − EPSR E ⋆ = I Z s ; Z r1:K + EPR DKL PS ∥ PS|R ≥ I Z s ; Z r1:K . (10) Proposition 2 (High-probability rejection of mismatched sources). Let γ := EP⊥ [Eθ ] − EPSR [Eθ ] be the effective energy gap and assume γ > 0. Suppose further that Eθ −EP [Eθ ] is κ2 -sub-Gaussian under both PSR and P⊥ (which is automatic when Eθ is bounded within [a, b], with κ2 = (b−a)2 /4). Setting the threshold τ = EPSR [Eθ ] + γ/2, γ2 γ2 , Pr Eθ ≥ τ ≤ exp − 8κ , (11) Pr Eθ ≤ τ ≤ exp − 8κ 2 2 PSR

P⊥

and consequently, treating P⊥ as the positive class with Eθ scored so that higher values are more anomalous and evaluating AUROC on independent class-conditional draws X⊥ ∼ P⊥ , XSR ∼ PSR , AUROC(Eθ ) ≥ 1 − 2 exp(−γ 2 /8κ2 ). Proofs can be found in Appendix C. Proposition 1 identifies the mutual information I(Z s ; Z r1:K ) as the energy gap which is exactly the quantity whose strict positivity is the empirical content of the Plato Representation Hypothesis. Proposition 2 upgrades the expectation-level gap to a per-sample guarantee via a standard Chernoff bound for sub-Gaussian variables [68].

Training the Platonic Energy

We pursue two training routes for the platonic energy of Eq. (7), sharing the same architecture but differing in readout and loss: a discriminative route that fits Eθ as a scalar compatibility score with NCE for backdoor detection, and a score-matching route that fits a conditional denoiser D̂θ which yields a score field for feature purification. Shared feature-space backbone architecture. The network for both routes shares the same first two stages, after which the readout is route-dependent. (i) Per-space projection heads g s : Rds → Rdtok and g rk : Rdrk → Rdtok (k = 1, . . . , K) map heterogeneous-dimensional encoder outputs to a shared token dimension dtok via three-layer LayerNorm-terminated MLPs. (ii) A pre-LayerNorm Transformer [67] processes the (K+1)-token sequence, letting the source token attend to every reference token. The discriminative route reads Eθ off the post-transformer source/reference tokens 5

Algorithm 1 Discriminative training.

Algorithm 2 cDSM training.

s references Require: source z , negative Require: standardized source z , rk K {z }k=1 , noise level σ, position encoding count M , position encoding es , . . . , erk es , . . . , erk , time embedding module η 1: hs0 ← g s (z s ) + es ▷ project to dtok , add space 1: ϵ ∼ N (0, Ids ), z̃ s ← z s + σ ϵ ▷ Eq. (2) embed 2: Compute cin (σ), cout (σ), cskip (σ), cnoise (σ) ▷ 2: for k = 1, . . . , K do r Eq. (6) 3: h0k ← g rk (z rk ) + erk 3: hs0 ← g s cin (σ) z̃ s + η cnoise (σ) + es 4: end for r r 4: for k = 1, . . . , K do 5: [h̃sθ , h̃θ1:K ] ← Transformer([hs0 , h01:K ]) r PK rk s 1 h0k ← g rk (z rk ) + erk 6: −Eθ ← α · K k=1 cos(h̃θ , h̃θ ) + β ▷ Eq. (12) 5: for 7: Training: draw M mismatched tuples from P⊥ , 6: end r 7: [h̃sθ , · ] ← Transformer([hs0 , h01:K ]) (m) recompute −Eθ via lines 1–5 s d s PM 8: Fθ ← π h̃θ ∈ R ▷ residual readout 8: LNCE ← −log sig(−Eθ ) − m=1log(1 − 9: D̂θ ← cskip (σ) z̃ s + cout (σ) Fθ ▷ EDM precond (m) sig(−Eθ )) ▷ Eq. (13) 2 10: Training: LcDSM ← λ(σ) D̂θ − z s 2 ▷ Eq. (14) Ensure: scalar energy Eθ ∈ R Ensure: denoised D̂θ ∈ Rds ; score sθ = (D̂θ − z̃ s )/σ 2 s

references {z rk }K k=1 ,

via the cosine head of Eq. (12). The score-matching route instead realizes D̂θ (z̃ s , z r1:K , σ) with an additional MLP readout π : Rdtok → Rds , which maps the post-transformer source token back to the encoder dimension, and a time embedding module η : R → Rdtok to inject noise-level information. We refer readers to EDM [29] for more details on the denoiser parameterization, and to Section D for a more detailed architectural description. Figure 7 visualizes the backbone architecture. Algorithms 1 and 2 concretely describe the training process of two routes. Discriminative training: a sigmoid-NCE method. We first introduce a discriminative training route that uses sigmoid-NCE to learn the matched-versus-mismatched density ratio underlying the effective gap γ in Proposition 2. We parameterize the discriminative training following SigLIP [72]: −Eθ z s , z r1:K

1 := α · K

K X

cos h̃sθ , h̃rθk

+ β,

(12)

k=1

where h̃sθ , h̃rθk ∈ Rdtok are the post-transformer source and reference token embeddings, and α = elog α > 0, β ∈ R are two learned scalars (with β initialised to − log M so that the initial logodds match a uniform prior over the M +1 candidates). For each clean tuple (z s , z r1:K ) ∼ PSR extracted from a single image (the positive) we draw M mismatched negatives. The training loss is the standard binary noise-contrastive objective [15, 48] LNCE (θ) = − EPSR log sig −Eθ − M EP⊥ log 1 − sig(−Eθ ) , sig(t) := 1+e1 −t , (13) whose Bayes-optimal logit equals log p(z s | z r1:K )/(M p(z s )) , so Eθ converges in expectation ⋆ s r to E of Eq. (9) up to a sum of a z -only and a z -only shift that both cancel in the energy gap of Eq. (10) [15]. The discriminative variant produces only a scalar for compatibility judgement, but it cannot induce the score field that the feature purification requires, motivating the DSM-based instantiation that follows. Conditional denoising score matching (cDSM) objective. We train the network with the conditional form of Eq. (3), h i 2 LcDSM (θ) = Eσ, (zs , zr1:K ), ϵ λ(σ) D̂θ z s + σϵ, z r1:K , σ − z s 2 , (14) with σ ∼ LogUniform[σmin , σmax ], ϵ ∼ N (0, Ids ), and λ(σ) as in Eq. (6). At its optimum, Eq. (14) recovers the Bayes-optimal conditional denoiser D̂⋆ (z̃ s , z r1:K , σ) = E[z s | z̃ s , z r1:K , σ] of the clean joint distribution. By Tweedie’s identity (Eq. (4)), we can derive the conditional score (D̂θ − z̃ s )/σ 2 that approximates ∇z̃s log pσ (z̃ s | z r1:K ), i.e., the gradient of the implicit platonic energy with respect to the source latent. For each mini-batch we draw per-sample Bernoulli masks with drop probability 0.1 to implement classifier-free-guidance [21] (CFG). 6

Table 1: The (CA %, PA %, AUC, ASR %) of defense methods against different backdoor attacks on ImageNet-1K. The adversarial attacks are implemented in an untargeted manner. † denotes our implementation. Decomp [20] (ICML 2025)

No Defense

Attack

DeDe [22] (CVPR 2025) †

†

ZIP[57] (NeurIPS 2023) †

CA ↑ PA ↑ ASR ↓ CA ↑ PA ↑ AUC ↑ ASR ↓ CA ↑ PA ↑ AUC ↑ ASR ↓ CA ↑ PA ↑ ASR ↓

Beatrix [43] DetectCLIP [23] (NDSS 2023) (ICLR 2025)

Platonic Defense (Ours)

AUC ↑

CA ↑ PA ↑ AUC ↑ ASR ↓

100.0 99.93 0.07 0.40 100.0 100.0 100.0 71.49

0.96 0.97 1.00 0.89 0.95 1.00 1.00 0.97

0.55 0.31 0.49 0.60 0.00 0.05 0.14 0.31

77.47 77.31 77.42 77.34 77.41 78.05 77.32 77.47

74.13 73.94 64.65 55.80 49.55 74.44 72.91 66.49

0.99 1.00 0.99 0.84 0.95 0.84 0.99 0.94

0.10 0.30 1.98 3.31 18.15 0.16 0.03 3.43

43.47 1.07 98.06 40.47 19.80 5.54 41.97 10.44 51.80

0.76 0.41 0.59

1.00 1.00 1.00

63.99 57.41 64.07 52.19 64.03 54.80

0.95 0.94 0.95

0.00 0.00 0.00

67.80 68.00 67.80 68.33 68.00 67.99

0.50 0.30 0.31 0.50 0.34 0.39

0.75 0.89 0.87 0.47 0.96 0.79

78.6 78.6 78.6 78.6 78.6 78.60

0.63 0.92 0.88 0.53 0.83 0.76

41.60 38.20 46.20 25.00 30.60 36.32

Backdoor attacks on unimodal encoders 0.03 0.08 0.03 3.76 0.10 0.10 0.10 0.60

99.96 99.90 99.96 94.44 100.0 100.0 100.0 99.18

76.75 76.35 76.93 76.87 74.98 75.83 73.11 75.83

0.12 0.16 0.14 3.28 0.11 0.11 0.11 0.58

1.00 1.00 1.00 0.96 1.00 1.00 0.99 0.99

99.97 99.89 99.96 95.36 100.00 100.00 100.00 99.31

31.1 30.2 30.1 29.94 31.0 28.4 26.2 29.56

CLIP-Backdoor [5] 53.25 0.24 BadCLIP [37] 51.01 0.55 Average 52.13 0.40

99.51 98.08 98.80

54.60 8.97 52.08 0.40 53.34 4.69

0.95 0.48 0.72

65.33 96.24 80.79

14.8 16.4 15.60

1.4 1.0 1.20

FGSM [12] PGD [44] MI-FGSM [9] CW-L2 [6] AutoAttack [8] Average

97.80 100.00 100.00 100.00 100.00 99.56

78.40 78.40 78.40 78.50 77.40 78.22

0.47 0.45 0.45 0.49 0.46 0.46

97.80 100.00 100.00 98.00 100.00 99.16

48.6 48.6 48.6 43.5 44.6 46.78

2.7 0.00 0.00 18.0 2.6 4.66

SSL-Backdoor [56] CorruptEncoder [5] CTRL [34] BLTO [61] NA [7] BadEncoder [27] DRUPE [62] Average

76.49 76.25 76.59 76.66 74.68 75.78 72.93 75.63

0.2 0.3 0.2 1.34 0.2 0.2 0.1 0.36

0.65 0.69 0.93 0.77 0.76 0.78 0.86 0.78

37.1 46.3 38.5 34.57 0.6 28.3 51.4 33.82

66.73 66.33 66.40 64.40 65.73 66.47 63.73 65.68

0.07 0.07 57.20 63.93 0.07 0.07 0.07 17.35

Backdoor attacks on image-text encoders 0.69 0.68 0.69

22.1 10.7 16.40

Adversarial Attacks 78.30 78.30 78.30 78.30 78.30 78.30

2.20 0.00 0.00 0.00 0.00 0.44

2.20 0.00 0.00 2.00 0.00 0.84

0.53 0.54 0.54 0.49 0.55 0.53

97.3 100.00 100.00 82.0 97.4 95.34

43.80 57.50 47.80 67.67 60.33 55.42

56.20 42.50 52.20 32.33 39.67 44.58

58.4 61.8 53.8 75.0 69.4 63.68

Figure 3: Evaluations on the text condition.

Experiments

5.1

Setup

For defense evaluation, we consider 3 types of attacks: backdoor attacks to unimodal encoders, backdoor attacks to image-text encoders, and adversarial attacks. For unimodal visual encoders, we poison DINO ViT-B/16 on ImageNet-1K and cover 7 representative SSL backdoor attacks: SSL-Backdoor [56], CorruptEncoder [5], CTRL [34], BLTO [61], Noisy-Alignment [7], BadEncoder [27], and DRUPE [62]. These attacks span patch-based, optimization-based, frequencydomain, and distribution-preserving poisoning mechanisms. For image–text encoders, we poison CLIP ViT-B/16 on a 60K subset of CC3M and evaluate CLIP-Backdoor [5] and BadCLIP [37]. We additionally test FGSM [12], PGD [44], MI-FGSM [9], CW-L2 [6], and AutoAttack [8] to assess whether the same representation-compatibility principle extends beyond persistent backdoors. Detailed settings are provided in Appendix D. We compare against recent test-time defenses, including Decomp [20], DeDe [22], ZIP [57], Beatrix [43], and DetectCLIP [23]. We report clean accuracy (CA), poisoned accuracy (PA), area under the ROC curve (AUC), and attack success rate (ASR). For multimodal encoders, we use zero-shot accuracy [52] for evaluation. Otherwise, we use linear probing accuracy. Unless otherwise specified, Platonic Defense uses K=2 clean reference encoders: the visual branch of SigLIP2-Base/16 [66] and DINOv2-Base/16. We use the Heun ODE solver [55] (20 steps) for cDSM inference. The conditional energy and DSM purifier share the same backbone architecture with 12 layers and hidden dimension 1536, trained only on a subset of ImageNet-1K unavailable during poisoning to avoid overfitting. More details are provided in Appendix D. 5.2

Main Results

Table 1 shows that platonic defense is the only method that consistently preserves clean utility while suppressing attacks across unimodal backdoors, image–text backdoors, and adversarial perturbations. On 7 unimodal SSL backdoors, it improves the average poisoned accuracy from 0.60% 7

Figure 4: Evaluations on the architecture. Table 2: Defense against enhanced attack. CR means recovered cosine similarity. ✓ denotes a reference that receives the backdoored image. DINOv2 SigLIP2 AIMv2 VLM2Vec CA ↑ PA ↑ ASR ↓ CR ↑

✓ ✓

✓ ✓ ✓

✓

Defense upper bound 62.5 62.5

0.0

0.683

One backdoored reference 59.0 58.9 32.7 32.4 59.6 59.6 ✓ 60.0 59.9 ✓

0.0 25.7 0.0 0.0

0.654 0.460 0.594 0.629

Two backdoored references 4.3 4.1 ✓ 49.5 49.1 52.7 52.8 ✓

70.3 0.374 0.0 0.532 0.5 0.581

DINOv2 SigLIP2 AIMv2 VLM2Vec CA ↑ PA ↑ ASR ↓ CR ↑ ✓ ✓

✓ ✓ ✓

✓

Two backdoored references 8.0 7.3 ✓ ✓ 6.1 5.7 ✓ ✓ 52.6 52.7

46.2 0.314 78.2 0.365 0.5 0.515

Three backdoored references 0.3 0.3 ✓ 0.3 0.3 ✓ ✓ ✓ 34.9 34.3 ✓ ✓ 0.4 0.4

84.3 96.0 3.9 67.9

✓

Worst case ✓

0.0

0.234 0.272 0.451 0.217

100.0 0.138

without defense to 66.49% and reduces ASR from 99.18% to 3.43%, while slightly increasing clean accuracy to 77.47%. Competing defenses either preserve accuracy but leave ASR near 100% (Decomp) or reduce ASR by sacrificing most clean accuracy (DeDe). The same pattern holds for image– text backdoors, where our method reaches 64.03% clean accuracy, 54.80% poisoned accuracy, and 0.00% ASR on average, and extends to untargeted adversarial attacks with the best average poisoned accuracy (63.68%) and lowest ASR (36.32%). Figure 3 further shows that the text-conditioned variant remains effective. We train our method on CC3M with the text branch of SigLIP2-Base/16 and CLIP ViT-B/16. Since ImageNet-1K does not provide textual descriptions, we generate image captions using two strategies: (i) class-template labels following CLIP [52]; and (ii) off-the-shelf VLMs, including Llama3.2-11B-Vision [13], Gemma3-27B [64], Qwen3-VL-4B-Instruct [2], and InternVL3-2B [76]. For all VLMs, we use the prompt: “Describe this image in one short sentence.” 5.3

Ablation Studies

Evaluation on enhanced attack. We consider a more challenging scenario in Table 2, where we assume the attacker is aware of our defense and has access to the reference encoders. With all references clean, the defense reaches 62.5% CA/PA and 0.0% ASR. Corrupting one reference is usually tolerated: three of four choices keep ASR at 0.0% with PA around 59–60%. With two corrupted references, the defense remains effective when the corrupted set excludes SigLIP2 or contains enough architectural diversity (ASR 0.0–0.5%, PA up to 52.8%). Three or four corrupted references largely exceed the intended threat model. Inference overhead. Figure 5 probes the cost–accuracy trade-off of the purification solver. Stochastic Langevin sampling [58] is ineffective even with 800 network-function evaluations (NFEs). Deterministic ODE solvers are substantially more favorable. The 10-step Heun solver attains the strongest PA in this sweep (60.3% across the two attacks) with only 36 NFEs. More aggressive settings reduce the cost further, down to 2–18 NFEs for Euler and DPM++ [41], but their accuracy becomes attack-dependent. 2-step Euler/DPM++ preserves CLIP-Backdoor accuracy while dropping Noisy-Alignment to 11.2%. We therefore use 20-step Heun as a conservative default for the main evaluation. 8

Table 3: Backbone transfer between the discriminative and cDSM training. Gap means platonic energy gap. “B” prefix means baseline. Loss

B- CDSM Cos PA

C DSM SIDE

ASR

Loss

Cos

ASR

B- DISC Loss Gap AUC

D ISC . SIDE Loss Gap AUC

Source

Variant

CLIP-Backdoor

Warm-start 0.605 0.678 0.496 0.000 0.147 3.18 0.9999 Frozen probe 0.571 0.701 0.554 0.000 0.773 0.553 0.100 0.000 0.144 3.24 0.9999 0.319 1.35 0.9954 Joint train 0.591 0.688 0.514 0.000 0.143 3.23 1.0000

Warm-start 0.619 0.659 0.466 0.116 0.148 3.17 0.9999 Noisy-Alignment Frozen probe 0.581 0.685 0.458 0.182 0.844 0.489 0.120 0.006 0.143 3.25 0.9988 0.271 1.80 0.9972 Joint train 0.603 0.671 0.596 0.042 0.142 3.26 1.0000

Architecture. Figure 4 ablates the architecture, reporting the denoised cosine similarity and cDSM training loss (eq. 14). The transformer is consistently the strongest backbone among the tested token mixers: MLP-Mixer [14] and ResMLP [36]. Increasing the transformer width has a larger effect than increasing depth: at 12 layers, the cosine improves from 0.743/0.739 at width 512 to 0.769/0.776 at width 1536 and saturates near width 2048, whereas increasing depth from 8 to 16 at width 512 changes cosine by less than 0.002. These trends motivate our default 12-layer, 1536-width Transformer, which captures most of the widthscaling gain without paying for the marginal 2048-width improvement. Removing the unconditional branch improves raw denoising metrics, but we keep CFG because it gives a principled way to tune conditioning strength and allows us to drop partial conditions to reduce inference cost when needed. Table 3 tests whether the two objectives can be joint trained. Warm-start initializes one branch from the model trained for the other objective and then finetunes it. Frozen probe directly loads the trained model and evaluates it without adapting the backbone. Joint training optimizes LNCE + LC DSM from scratch. The transfer is asymFigure 5: Inference efficiency probing. metric. Moving from cDSM to the discriminative branch works well but the reverse direction is less reliable, especially without fine-tuning. This gap suggests that detection and purification use related but not identical representations. Joint training largely closes the gap: the discriminative branch stays at baseline AUROC, and the cDSM branch approaches standalone cDSM performance with only a small loss on CLIP-Backdoor (PA 0.514 vs. 0.554) while improving the harder Noisy-Alignment case (PA 0.596 vs. 0.458, ASR 0.042 vs. 0.182).

Conclusion

We introduced a black-box test-time method for defending SSL encoders against backdoor attacks. The central idea is to use the compatibility between representations from independently trained models as a security signal. We formalized it as a conditional energy model and trained it with noise-contrastive estimation and conditional denoising score matching. Our analysis connects the energy gap to the mutual information between source and reference representations, giving a theoretical guarantee for the defense. Empirically, the method improves robustness across multiple SSL encoders and a broad set of attacks, while preserving clean utility. Extending the same principle to language, audio, and fully multimodal latent spaces is an important direction for future work.

References [1] Santiago Acevedo, Andrea Mascaretti, Riccardo Rende, Matéo Mahaut, Marco Baroni, and Alessandro Laio. A quantitative analysis of semantic information in deep representations of text and images. arXiv preprint arXiv:2505.17101, 2025. [2] Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631, 2025. [3] Hritik Bansal, Nishad Singhi, Yu Yang, Fan Yin, Aditya Grover, and Kai-Wei Chang. Cleanclip: Mitigating data poisoning attacks in multimodal contrastive learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 112–123, 2023. [4] Yancheng Cai, Fei Yin, Dounia Hammou, and Rafal Mantiuk. Do computer vision foundation models learn the low-level characteristics of the human visual system? In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 20039–20048, 2025. [5] Nicholas Carlini and Andreas Terzis. Poisoning and backdooring contrastive learning. In International Conference on Learning Representations, 2021. [6] Nicholas Carlini and David Wagner. Towards Evaluating the Robustness of Neural Networks . In 2017 IEEE Symposium on Security and Privacy (SP), pages 39–57, Los Alamitos, CA, USA, May 2017. IEEE Computer Society. doi: 10.1109/SP.2017.49. URL https://doi. ieeecomputersociety.org/10.1109/SP.2017.49. [7] Tuo Chen, Jie Gui, Minjing Dong, Ju Jia, Lanting Fang, and Jian Liu. Backdooring selfsupervised contrastive learning by noisy alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3684–3693, 2025. [8] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pages 2206–2216. PMLR, 2020. [9] Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9185–9193, 2018. [10] Shiwei Feng, Guanhong Tao, Siyuan Cheng, Guangyu Shen, Xiangzhe Xu, Yingqi Liu, Kaiyuan Zhang, Shiqing Ma, and Xiangyu Zhang. Detecting backdoors in pre-trained encoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16352–16362, 2023. [11] Enrico Fini, Mustafa Shukor, Xiujun Li, Philipp Dufter, Michal Klein, David Haldimann, and Sai Aitharaju. Multimodal autoregressive pre-training of large vision encoders. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. [12] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014. [13] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem 10

Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vı́tor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev,

Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783. [14] Ming Gui, Johannes Schusterbauer, Timy Phan, Felix Krause, Joshua M. Susskind, Miguel Ángel Bautista, and Björn Ommer. Adapting self-supervised representations as a latent space for efficient generation. In The Fourteenth International Conference on Learning Representations, 2026. [15] Michael U Gutmann and Aapo Hyvärinen. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. Journal of machine learning research, 13(2), 2012. [16] Tingxu Han, Weisong Sun, Ziqi Ding, Chunrong Fang, Hanwei Qian, Jiaxun Li, Zhenyu Chen, and Xiangyu Zhang. Mutual information guided backdoor mitigation for pre-trained encoders. IEEE Transactions on Information Forensics and Security, 20:3414–3428, 2025. ISSN 15566021. [17] James A Hanley and Barbara J McNeil. The meaning and use of the area under a receiver operating characteristic (roc) curve. Radiology, 143:29–36, 1982. [18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. [19] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000–16009, 2022. [20] Shuo He, Zhifang Zhang, Feng Liu, Roy Ka-Wei Lee, Bo An, and Lei Feng. A closer look at backdoor attacks on clip. In Forty-Second International Conference on Machine Learning, 2025. [21] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021. URL https: //openreview.net/forum?id=qw8AKxfYbI. [22] Sizai Hou, Songze Li, and Duanyi Yao. Dede: Detecting backdoor samples for ssl encoders via decoders. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 20675–20684, 2025. 12

[23] Hanxun Huang, Sarah Monazam Erfani, Yige Li, Xingjun Ma, and James Bailey. Detecting backdoor samples in contrastive language image pretraining. In The Thirteenth International Conference on Learning Representations, 2025. [24] Yuxian Huang, Geng Yang, Dong Yuan, and Shui Yu. Dbssl: A scheme to detect backdoor attacks in self-supervised learning models. IEEE Transactions on Dependable and Secure Computing, pages 1–12, 2025. ISSN 1941-0018. [25] Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. Position: The platonic representation hypothesis. In Proceedings of the 41st International Conference on Machine Learning, July 2024. [26] Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005. [27] Jinyuan Jia, Yupei Liu, and Neil Zhenqiang Gong. Badencoder: Backdoor attacks to pretrained encoders in self-supervised learning. In 2022 IEEE Symposium on Security and Privacy (SP), pages 2043–2059, May 2022. [28] Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, and Wenhu Chen. VLM2vec: Training vision-language models for massive multimodal embedding tasks. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview. net/forum?id=TE0KOzWYAF. [29] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. [30] A. Sophia Koepke, Daniil Zverev, Shiry Ginosar, and Alexei A. Efros. Back into plato’s cave: Examining cross-modal representational convergence at scale, April 2026. [31] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pages 3519–3529. PMlR, 2019. [32] Vedang Lad, Jin Hwa Lee, Wes Gurnee, and Max Tegmark. Remarkable robustness of LLMs: Stages of inference? In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. URL https://openreview.net/forum?id=Wxh5Xz7NpJ. [33] Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, Fujie Huang, et al. A tutorial on energy-based learning. Predicting structured data, 1(0), 2006. [34] Changjiang Li, Ren Pang, Zhaohan Xi, Tianyu Du, Shouling Ji, Yuan Yao, and Ting Wang. An embarrassingly simple backdoor attack on self-supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4367–4378, 2023. [35] Changjiang Li, Ren Pang, Bochuan Cao, Zhaohan Xi, Jinghui Chen, Shouling Ji, and Ting Wang. On the difficulty of defending contrastive learning against backdoor attacks. In 33rd USENIX Security Symposium (USENIX Security 24), pages 2901–2918, 2024. ISBN 978-1939133-44-1. [36] Tianhong Li, Dina Katabi, and Kaiming He. Return of unconditional generation: A selfsupervised representation generation method. In The Thirty-Eighth Annual Conference on Neural Information Processing Systems, 2024. [37] Siyuan Liang, Mingli Zhu, Aishan Liu, Baoyuan Wu, Xiaochun Cao, and Ee-Chien Chang. Badclip: Dual-embedding guided backdoor attack on multimodal contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24645–24654, 2024. [38] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, December 2023. 13

[39] Hongbin Liu, Jinyuan Jia, and Neil Zhenqiang Gong. Poisonedencoder: Poisoning the unlabeled pre-training data in contrastive learning. In 31st USENIX Security Symposium (USENIX Security 22), pages 3629–3645, 2022. ISBN 978-1-939133-31-1. [40] Hongbin Liu, Michael K Reiter, and Neil Zhenqiang Gong. Mudjacking: Patching backdoor vulnerabilities in foundation models. In 33rd USENIX Security Symposium (USENIX Security 24), pages 2919–2936, 2024. [41] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. Machine Intelligence Research, 22(4):730–751, 2025. [42] Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference, 2023. URL https://arxiv. org/abs/2310.04378. [43] Wanlun Ma, Derui Wang, Ruoxi Sun, Minhui Xue, Sheng Wen, and Yang Xiang. The ”beatrix” resurrections: Robust backdoor detection via gram matrices. In Proceedings 2023 Network and Distributed System Security Symposium, 2023. ISBN 978-1-891562-83-9. [44] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id= rJzIBfZAb. [45] Matéo Mahaut and Marco Baroni. Similarity of processing steps in vision model representations. arXiv preprint arXiv:2601.21621, 2026. [46] Thao Nguyen, Maithra Raghu, and Simon Kornblith. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. In International Conference on Learning Representations, 2021. URL https://openreview. net/forum?id=KJNcAkY8tY4. [47] XuanLong Nguyen, Martin J Wainwright, and Michael I Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 56(11):5847–5861, 2010. [48] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. [49] OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, 14

Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024. URL https://arxiv.org/abs/2303.08774. [50] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. [51] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, highperformance deep learning library, 2019. URL https://arxiv.org/abs/1912.01703. [52] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, pages 8748–8763, July 2021. [53] Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=Gl8FHfMVTZu. [54] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2021. [55] André Ronveaux and Felix Medland Arscott. Heun’s differential equations. Clarendon Press, 1995. [56] Aniruddha Saha, Ajinkya Tejankar, Soroush Abbasi Koohpayegani, and Hamed Pirsiavash. Backdoor attacks on self-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13337–13346, 2022. 15

[57] Yucheng Shi, Mengnan Du, Xuansheng Wu, Zihan Guan, Jin Sun, and Ninghao Liu. Black-box backdoor defense via zero-shot image purification. In Thirty-Seventh Conference on Neural Information Processing Systems, 2023. [58] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021. [59] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In Proceedings of the 40th International Conference on Machine Learning, pages 32211–32252, July 2023. [60] Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density ratio estimation in machine learning. Cambridge University Press, 2012. [61] Weiyu Sun, Xinyu Zhang, Hao Lu, Ying-Cong Chen, Ting Wang, Jinghui Chen, and Lu Lin. Backdoor contrastive learning via bi-level trigger optimization. In The Twelfth International Conference on Learning Representations, October 2023. [62] Guanhong Tao, Zhenting Wang, Shiwei Feng, Guangyu Shen, Shiqing Ma, and Xiangyu Zhang. Distribution preserving backdoor attack in self-supervised learning. In 2024 IEEE Symposium on Security and Privacy (SP), pages 29–29, October 2023. ISBN 979-8-35033130-1. [63] Guanhong Tao, Zhenting Wang, Shiwei Feng, Guangyu Shen, Shiqing Ma, and Xiangyu Zhang. Distribution preserving backdoor attack in self-supervised learning. In 2024 IEEE Symposium on Security and Privacy (SP), pages 2029–2047. IEEE, 2024. [64] Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaël Liu, Francesco Visin, Kathleen Kenealy, Lucas Beyer, Xiaohai Zhai, Anton Tsitsulin, Robert Busa-Fekete, Alex Feng, Noveen Sachdeva, Benjamin Coleman, Yi Gao, Basil Mustafa, Iain Barr, Emilio Parisotto, David Tian, Matan Eyal, Colin Cherry, Jan-Thorsten Peter, Danila Sinopalnikov, Surya Bhupatiraju, Rishabh Agarwal, Mehran Kazemi, Dan Malkin, Ravin Kumar, David Vilar, Idan Brusilovsky, Jiaming Luo, Andreas Steiner, Abe Friesen, Abhanshu Sharma, Abheesht Sharma, Adi Mayrav Gilady, Adrian Goedeckemeyer, Alaa Saade, Alex Feng, Alexander Kolesnikov, Alexei Bendebury, Alvin Abdagic, Amit Vadi, András György, André Susano Pinto, Anil Das, Ankur Bapna, Antoine Miech, Antoine Yang, Antonia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Bo Wu, Bobak Shahriari, Bryce Petrini, Charlie Chen, Charline Le Lan, Christopher A. Choquette-Choo, CJ Carey, Cormac Brick, Daniel Deutsch, Danielle Eisenbud, Dee Cattle, Derek Cheng, Dimitris Paparas, Divyashree Shivakumar Sreepathihalli, Doug Reid, Dustin Tran, Dustin Zelle, Eric Noland, Erwin Huizenga, Eugene Kharitonov, Frederick Liu, Gagik Amirkhanyan, Glenn Cameron, Hadi Hashemi, Hanna Klimczak-Plucińska, Harman Singh, Harsh Mehta, Harshal Tushar Lehri, Hussein Hazimeh, Ian Ballantyne, Idan Szpektor, Ivan Nardini, Jean Pouget-Abadie, Jetha Chan, Joe Stanton, John Wieting, Jonathan Lai, Jordi Orbay, Joseph Fernandez, Josh Newlan, Ju yeong Ji, Jyotinder Singh, Kat Black, Kathy Yu, Kevin Hui, Kiran Vodrahalli, Klaus Greff, Linhai Qiu, Marcella Valentine, Marina Coelho, Marvin Ritter, Matt Hoffman, Matthew Watson, Mayank Chaturvedi, Michael Moynihan, Min Ma, Nabila Babar, Natasha Noy, Nathan Byrd, Nick Roy, Nikola Momchev, Nilay Chauhan, Noveen Sachdeva, Oskar Bunyan, Pankil Botarda, Paul Caron, Paul Kishan Rubenstein, Phil Culliton, Philipp Schmid, Pier Giuseppe Sessa, Pingmei Xu, Piotr Stanczyk, Pouya Tafti, Rakesh Shivanna, Renjie Wu, Renke Pan, Reza Rokni, Rob Willoughby, Rohith Vallu, Ryan Mullins, Sammy Jerome, Sara Smoot, Sertan Girgin, Shariq Iqbal, Shashir Reddy, Shruti Sheth, Siim Põder, Sijal Bhatnagar, Sindhu Raghuram Panyam, Sivan Eiger, Susan Zhang, Tianqi Liu, Trevor Yacovone, Tyler Liechty, Uday Kalra, Utku Evci, Vedant Misra, Vincent Roseberry, Vlad Feinberg, Vlad Kolesnikov, Woohyun Han, Woosuk Kwon, Xi Chen, Yinlam Chow, Yuvein Zhu, Zichuan Wei, Zoltan Egyed, Victor Cotruta, Minh Giang, Phoebe Kirk, Anand Rao, Kat Black, Nabila Babar, Jessica Lo, Erica Moreira, Luiz Gustavo Martins, Omar Sanseviero, Lucas Gonzalez, Zach Gleicher, Tris Warkentin, Vahab Mirrokni, Evan 16

Senter, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, Yossi Matias, D. Sculley, Slav Petrov, Noah Fiedel, Noam Shazeer, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Jean-Baptiste Alayrac, Rohan Anil, Dmitry, Lepikhin, Sebastian Borgeaud, Olivier Bachem, Armand Joulin, Alek Andreev, Cassidy Hardin, Robert Dadashi, and Léonard Hussenot. Gemma 3 technical report, 2025. URL https://arxiv.org/abs/2503.19786. [65] Ajinkya Tejankar, Maziar Sanjabi, Qifan Wang, Sinong Wang, Hamed Firooz, Hamed Pirsiavash, and Liang Tan. Defending against patch-based backdoor attacks on self-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12239–12249, 2023. [66] Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786, 2025. [67] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, volume 30, 2017. [68] Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint, volume 48. Cambridge university press, 2019. [69] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. [70] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In Proceedings of the 37th International Conference on Machine Learning, pages 9929–9939, November 2020. [71] Wenhan Yang, Jingdong Gao, and Baharan Mirzasoleiman. Robust contrastive language-image pretraining against data poisoning and backdoor attacks. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 10678–10691, 2023. [72] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023. [73] Hanrong Zhang, Zhenting Wang, Boheng Li, Fulin Lin, Tingxu Han, Mingyu Jin, Chenlu Zhan, Mengnan Du, Hongwei Wang, and Shiqing Ma. Invisible backdoor attack against selfsupervised learning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 25790–25801, 2025. [74] Mengxin Zheng, Jiaqi Xue, Zihao Wang, Xun Chen, Qian Lou, Lei Jiang, and Xiaofeng Wang. Ssl-cleanse: Trojan detection and mitigation in self-supervised learning. In European Conference on Computer Vision, pages 405–421, 2024. [75] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 633–641, 2017. [76] Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479, 2025.

Related Work

A.1

Backdoor Attacks on Visual Encoders

Backdoor attacks on visual encoders pose a severe threat as these encoders serve as general-purpose feature extractors for diverse downstream tasks. Carlini and Terzis [5] first demonstrated that contrastive learning is vulnerable to data poisoning based backdoor attacks. Subsequent works explored different attack vectors: PoisonedEncoder [39] and Saha et al. [56] poisoned the unlabeled pretraining data, while BadEncoder [27] directly modified encoder parameters to implant persistent backdoors. Recent efforts have shifted toward improving attack stealthiness. Tao et al. [63] designed distribution-preserving attacks, and Zhang et al. [73] proposed invisible triggers via frequencydomain manipulation. With the rise of CLIP, attacks have been extended to multimodal encoders. Liang et al. [37] leveraged dual-embedding guidance for higher attack success rates. He et al. [20] systematically analyzed the attack surface of CLIP and revealed new vulnerabilities.

A.2

Backdoor Detection and Mitigation on Visual Encoders

Defending visual encoders against backdoor attacks is more challenging than defending supervised classifiers, as SSL encoders lack label information and the backdoor manifests in the representation space. Feng et al. [10] first proposed detecting backdoors in pre-trained encoders by analyzing anomalous representation clusters. SSL-Cleanse [74] combined trigger reverse engineering with mitigation in a unified pipeline. For post-hoc mitigation, MIMIC [16] employed mutual information guided knowledge distillation to selectively suppress backdoor features, and Mudjacking [40] repaired compromised neurons via model patching. Li et al. [35] revealed that existing defenses are often insufficient against adaptive attacks on contrastive learning. For multimodal encoders, CleanCLIP [3] fine-tuned backdoored CLIP with clean contrastive objectives, and Yang et al. [71] incorporated data selection for robust pre-training. More recently, Huang et al. [23] detected backdoor samples in CLIP by analyzing cross-modal alignment discrepancies, and DeDe [22] leveraged decoder-based reconstruction to identify poisoned inputs for SSL encoders.

A.3

Similarity on Representations

Recent studies on representational similarity ask whether independently trained models converge to a common internal organization. Centered Kernel Alignment (CKA) has become a standard tool for this question, enabling layer-wise comparisons across changes in width, depth, and architecture family [31, 46, 53]. Using this perspective, Nguyen et al. [46] found contiguous blocks of layers whose representations remain strongly aligned despite substantial architectural variation, and showed that probes and layer ablations within these blocks often induce only minor behavioral changes. This result suggests that overparameterized networks may allocate neighboring layers to closely related computations. However, high alignment should not be interpreted only as redundancy. Huh et al. [25] and Acevedo et al. [1] argued that aligned representations can preserve semantically meaningful structure across modalities, including cross-lingual structure within text, while Lad et al. [32] described middle-to-late layers as an incremental feature-construction stage that gradually assembles task-relevant abstractions. Moving from representational states to representational trajectories, Mahaut and Baroni [45] showed that layers at similar depths across vision models tend to be closest, but their processing dynamics still differ: classifier models discard low-level image statistics more strongly in later layers, whereas transformer-based models change representations more smoothly than CNNs. Together, these findings suggest that high-similarity regions can reflect both computational reuse and functionally meaningful stages of representation formation. Representational similarity has also been studied beyond model-to-model comparisons. Cai et al. [4] evaluated 45 foundation and generative models against low-level human visual characteristics, including contrast detection, contrast masking, and contrast constancy. They found that models trained on vision tasks, especially DINOv2, align with human perceptual thresholds, suggesting that learned feature spaces capture visual structure shared with the human visual system. 18

(a) CKA

(b) Trained vs. random initialization

Figure 6: Additional alignment visualizations complementing Figure 2. (a) Pairwise CKA scores. (b) Trained models vs. their random-initialized counterparts on both metrics: trained models achieve 30–100× higher alignment, confirming that the effect arises from learning rather than architectural inductive bias.

Limitations

Although the Platonic defense is formulated in a modality-agnostic way, our empirical study focuses primarily on visual representation spaces. We therefore leave a systematic evaluation of multimodal conditioning, such as conditioning vision features on language, audio, or other heterogeneous reference modalities, to future work. Such settings may introduce a stronger modality mismatch than the visual-to-visual cases studied in this paper: different modalities can encode semantic content at different granularities, with different nuisance factors and latent geometries. As a result, the conditional energy or score model may need substantially more paired or weakly aligned data to learn a reliable cross-modal compatibility signal. This requirement could increase both data collection cost and training cost, especially when multiple reference modalities are used simultaneously. Understanding how to reduce this data and compute overhead is an important direction for scaling the method beyond the visual setting.

Proofs for the Detection Guarantee

Throughout this section, Z s denotes the source latent and Z r denotes the joint reference latent Z r1:K . Let PSR be their matched joint distribution, and let P⊥ = PS ⊗ PR be the independent cross-sample product. We assume that the matched joint distribution admits a regular conditional density p(s | r), and that the source marginal admits a density pS (s), so that the conditional cross-entropies and mutual information below are well defined. We write I(S; R) := I(Z s ; Z r ) for brevity. To avoid degenerate cases in which entropy differences involve undefined ∞ − ∞ terms, we make the following standing integrability assumption: EPSR |log p(S | R)| < ∞, EP⊥ |log p(S | R)| < ∞, h i (15) I(S; R) < ∞, EPR DKL PS ∥ PS|R < ∞. These regularity conditions are mild for the continuous, normalized latent representations considered in Section 5; in practice they can be enforced by the standard small-noise or smoothing view of learned features. C.1

Proof of Proposition 1

Let E ⋆ (s, r) = − log p(s | r) + ψ(r) be the oracle energy of Eq. (9), with ψ ∈ L1 (PR ), i.e., EPR [ψ(R)] < ∞. Because the R-marginals of PSR and P⊥ coincide, the reference-only term ψ(R) 19

contributes the same expectation to both populations and cancels from the gap: EP⊥ ψ(R) = EPR ψ(R) = EPSR ψ(R) . (16) By definition, EPSR − log p(S | R) = H(S | R). Under P⊥ = PS ⊗ PR , we treat S and R as independent and apply Fubini’s theorem: EP⊥ − log p(S | R) = EPR ES∼PS − log p(S | R) . (17) The inner expectation is the cross-entropy of PS relative to PS|R=r . The cross-entropy decomposes as follows: ES∼PS − log p(S | r) = H(S) + DKL PS PS|R=r , (18) where H(S) is the differential entropy of S and the KL divergence is non-negative (and, by (15),has finite PR -expectation). Substituting Eq. (18) into Eq. (17) and subtracting EPSR − log p(S | R) = H(S | R) yields h i EP⊥ E ⋆ − EPSR E ⋆ = H(S) − H(S | R) + EPR DKL PS ∥ PS|R h i = I(S; R) + EPR DKL PS ∥ PS|R ≥ I(S; R). (19) Remark on the density-ratio view. The same separation can also be viewed through the logdensity ratio between matched and independently paired latents. When the joint and product distributions have overlapping support, the statistic A⋆ (s, r) := log

pS (s)pR (r) = − log p(s | r) + log pS (s) pSR (s, r)

(20)

is the Neyman–Pearson optimal score for distinguishing independent pairs from matched pairs [47, 60], with larger values indicating a more mismatched pair. If the two KL divergences below are finite, then EP⊥ [A⋆ ] − EPSR [A⋆ ] = DKL (P⊥ ∥ PSR ) + DKL (PSR ∥ P⊥ ), (21) where the second term is exactly I(S; R). Thus, a detector that estimates the full matched-vsindependent density ratio also obtains a gap controlled by the mutual information. This density-ratio view is mainly useful for discriminative or contrastive training objectives, such as NCE-style losses. Our oracle argument above only requires the conditional log-density − log p(s | r), which avoids needing to model the source marginal term log pS (s) explicitly. C.2

Proof of Proposition 2

Let X⊥ := Eθ (Z s , Z r ) with (Z s , Z r ) ∼ P⊥ and XSR analogously under PSR , with means µ⊥ , µSR and gap γ = µ⊥ − µSR > 0. By the sub-Gaussian assumption, X⊥ − µ⊥ and XSR − µSR are κ2 -sub-Gaussian [68, Ch. 2], so the standard one-sided Chernoff bound yields, for any t > 0, Pr XSR − µSR ≥ t ≤ exp −t2 /2κ2 , Pr µ⊥ − X⊥ ≥ t ≤ exp −t2 /2κ2 . (22) Setting the detection threshold τ := µSR + γ/2 = µ⊥ − γ/2 and applying Eq. (22) with t = γ/2, Pr Eθ ≥ τ = Pr XSR − µSR ≥ γ/2 ≤ exp −γ 2 /8κ2 , (23) PSR Pr Eθ ≤ τ = Pr µ⊥ − X⊥ ≥ γ/2 ≤ exp −γ 2 /8κ2 . (24) P⊥

By the probability interpretation of the AUROC [17], we have AUROC(Eθ ) = Pr X⊥ > XSR + 12 Pr X⊥ = XSR ≥ Pr X⊥ > XSR ,

(25)

applied to two independent samples X⊥ ∼ P⊥ , XSR ∼ PSR . Continuity of Eθ under either marginal (always the case for DSM-trained models with Gaussian noise) merely makes the inequality an equality. We take a looser union-bound version: for independent X⊥ , XSR , {X⊥ > τ } ∩ {XSR < τ } ⊆ {X⊥ > XSR }, 20

(26)

Figure 7: The shared architecture of the platonic energy training for the discriminative and scorematching routes. so by the union bound on the complement, Pr[X⊥ > XSR ] ≥ 1 − Pr[Eθ ≤ τ ] − Pr [Eθ ≥ τ ] ≥ 1 − 2 exp −γ 2 /8κ2 , P⊥

PSR

(27)

which yields the main-text bound. A tighter rate is available by working directly with the difference variable: writing Y := X⊥ − XSR for independent X⊥ , XSR , Y is sub-Gaussian with parameter 2κ2 and mean γ, so Pr[Y ≤ 0] ≤ exp(−γ 2 /4κ2 ). We keep the looser two-tail version in the main text because it also upper-bounds the clean- and mismatched-side false-alarm rates at any single threshold τ . □

Experiments Details

Architecture instantiation. Both training routes share the backbone architecture of Figure 7 and differ only in the readout. Each space is projected to a common token width dtok =1536 by a 3-layer MLP, Linear–LayerNorm–GELU–Linear–LayerNorm–GELU–Linear. The discriminative route applies a final LayerNorm to every projected token, while the score-matching route leaves the noised source token unnormalised before EDM preconditioning and applies the final LayerNorm only to reference tokens. A learnable space embedding, initialised from N (0, 0.022 ), is added to each token, and the resulting 1+K tokens are processed by a 12-layer pre-norm Transformer with 12 attention heads, GELU feed-forward blocks of width 4dtok , and zero dropout. We implement the transformer backbone with the encoder-only transformer from Pytorch Library [51]. For cDSM, the source token additionally receives a sinusoidal noise embedding with 128 Fourier channels followed by a two-layer SiLU MLP into dtok . The residual denoising head is a Linear–GELU–Linear MLP with hidden width dtok and a zero-initialised final layer, so the initial denoiser follows the EDM skip connection. Discriminative route. The post-transformer source and reference tokens are scored by a cosine readout averaged over the K references; the logit is α cos ¯ + β, where α = exp(log α) is constrained positive and β is initialised to − log r for a negative ratio r (both learnable). Training uses sigmoidBCE against negatives produced by a mismatch sampler that mixes full (all spaces deranged), sourceonly, and partial (a randomly chosen subset of references replaced) deranged tuples; before the feature queue is warm we derange within the mini-batch, afterwards we draw from a per-space FIFO queue. 21

Score-matching route. The source feature is first standardised per dimension using statistics (µ, σdim ) pre-computed on clean p training data. We p adopt the EDM [29] preconditioning with cskip = σd2 /(σ 2 +σd2 ), cout = σσd / σ 2 +σd2 , cin = 1/ σ 2 +σd2 , cnoise = 14 log σ, where σd means σdata . The source token feeds cin z̃std into the source head and adds a sinusoidal-MLP σ-embedding alongside the space embedding. A 2-layer MLP readout Fθ maps the post-transformer source token back to encoder dimension, giving D̂θ = cskip z̃ + cout Fθ and score (D̂θ − z̃)/σ 2 . The DSM loss ⋆ is the EDM-weighted MSE λ(σ) ∥D̂θ − zstd ∥2 with λ = (σ 2 +σd2 )/(σσd )2 , σ drawn log-uniformly from [σmin , σmax ], and a 10% probability of zeroing all reference tokens to expose an unconditional branch for classifier-free guidance. At inference we use a Heun-EDM ODE solver over a ρ=7 schedule by default, with an anchored annealed Langevin sampler available as an alternative. CFG is applied as suncond + w(scond − suncond ) with w=1 as the default. Backdoor implantation. We instantiate backdoored encoders by post-training pretrained models on small poisoned subsets. For unimodal visual SSL encoders, we poisoned images following corresponding backdoor methods, and optimize the triggered representation with a additional cosine representation preserving loss to prevent catastrophic forgetting. For image–text encoders, following the CleanCLIP pipeline [3], we poison 0.6M subset of CC3M by applying the trigger to randomly selected images and replacing their captions with manipulated ImageNet templates, then continue CLIP contrastive training. This procedure makes triggered inputs map to the target semantic region while preserving clean utility. Table 4: Training hyperparameters.

Discriminative route

Score-matching route

Optimizer Peak / min learning rate Weight decay LR schedule Gradient clip (ℓ2 ) Effective / micro batch size Epochs

AdamW 10−4 / 10−6 0.01 cosine, 1 warmup epoch 1.0 256 / 16 5

Loss Negative ratio r σ sampling Reference-drop prob. pdrop ref EMA decay Inference solver / steps

sigmoid BCE 4 — — — —

EDM-weighted DSM — log-uniform [σmin , σmax ] 0.1 0.999 Heun (ρ=7), 20

Reconstruction-Driven Purification of DEDE

We try to directly optimize the suspicious latent against the reconstruction loss of an off-the-shelf detector, DEDE [22]. We evaluated this baseline on a DINO ViT-B/16 BadEncoder victim [27] with a 50×50 patch trigger. The DEDE decoder is trained on clean 100K ImageNet-1K data and then frozen. During purification we hold the encoder and decoder fixed, update only the latent z = fϕ (x). The latent update minimizes the masked reconstruction objective 2 1 Lrec (x, z) = EM M ⊙ x − gψ (xvis , z) 2 , |M |

(28)

where M is sampled at each optimizer step. Table 5 shows the central result. Stronger Adam updates reduce ASR, but only by simultaneously degrading clean accuracy. More importantly, the poison Top-1 never exceeds 0.20%: even when ASR falls to zero, the purified latent is not mapped back to the ground-truth class. This suggests that the reconstruction objective can disrupt the attacker target, but it cannot perceive what the correct representation should be. This is the failure mode addressed by the conditional DSM purifier, which learns an explicit score field toward a referencedefined clean representation. We observe the same failure pattern on CLIP-ViT-B/16 victims using a zero-shot CLIP text classifier. Table 6 shows that CLIP latents are more sensitive to Adam updates, with usable learning rates around η = 0.1 rather than η = 1.0. 22

Table 5: Latent purification with the BEST-AUROC DEDE decoder on BadEncoder. Evaluation uses a frozen DINO ViT-B/16 BadEncoder, a frozen DEDE decoder, and a deterministic 1,000image ImageNet-1K validation subset. Steps T

LR η

Clean ↑

Poison ↑

ASR ↓

0 (baseline)

—

88.40

0.00

100.00

0.01 0.10 1.00

88.40 88.20 81.70

0.00 0.00 0.00

100.00 100.00 100.00

0.01 0.10 1.00 5.00 20.0 100

88.30 87.80 67.00 1.50 0.10 0.00

0.00 0.00 0.00 0.10 0.00 0.00

100.00 100.00 79.00 0.10 0.00 0.00

0.01 0.10 1.00 5.00 20.0 100

88.10 87.40 55.20 1.20 0.10 0.00

0.00 0.00 0.00 0.00 0.00 0.00

100.00 100.00 37.70 0.00 0.00 0.00

100

0.20 0.30 0.50 0.70 1.00 5.00 20.0 100

85.50 82.70 74.70 66.90 47.40 0.70 0.40 0.10

0.00 0.00 0.10 0.20 0.10 0.00 0.00 0.00

100.00 100.00 97.40 73.30 22.50 0.00 0.00 0.00

Table 6: CLIP-ViT-B/16 hyperparameter sweep for DEDE-decoder latent purification. Both victims use a frozen CLIP image tower, frozen DEDE decoder, Adam updates, a 1,000-image ImageNet-1k subset, and zero-shot CLIP classification. clip-backdoor

BadCLIP

Steps T

LR η

Clean

Poison

ASR

Clean

Poison

ASR

0 (baseline)

—

50.00

0.30

99.50

47.20

0.60

97.90

0.001 0.010 0.050 0.100 0.200 0.500

49.50 48.90 43.60 30.00 8.60 1.60

0.60 0.30 0.60 1.30 1.00 0.60

99.50 99.70 98.00 54.55 1.60 0.00

46.60 46.70 41.30 25.90 6.00 1.00

0.70 0.70 1.70 1.10 0.40 0.10

97.90 96.80 82.48 30.23 2.10 0.00

0.001 49.40 0.010 48.30 0.050 38.80 0.100 20.10 0.200 5.20 0.500 0.80

0.40 0.40 1.10 1.20 0.40 0.30

99.40 46.60 99.20 46.10 92.59 38.50 33.03 19.00 1.00 4.20 0.00 0.50

0.60 1.10 1.60 0.60 0.40 0.20

97.90 95.60 68.77 16.92 0.60 0.00

100

0.001 49.30 0.010 47.50 0.050 33.00 0.100 14.80 0.200 3.90 0.500 0.70

0.40 0.40 1.20 1.40 0.80 0.50

99.50 46.50 99.10 44.70 84.18 32.50 22.12 16.40 0.50 3.40 0.00 0.60

0.70 1.10 1.00 1.00 0.20 0.20

97.80 93.79 55.26 10.71 0.40 0.20

Record · ID 321763 · SHA-256 a8c94846e93ac166
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.