TEVI: Text-Conditioned Editing of Visual Representations via Sparse Autoencoders for Improved Vision-Language Alignment

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

TEVI: Text-Conditioned Editing of Visual Representations via Sparse Autoencoders for Improved Vision-Language Alignment

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

machine learning, deep learning, neural networks

TEVI: Text-Conditioned Editing of Visual Representations via Sparse Autoencoders for Improved Vision-Language Alignment Sweta Mahajan∗,1 , Sukrut Rao∗,1 , Jiahao Xie1 , Alexander Koller1,2 , Bernt Schiele1 1 Max Planck Institute for Informatics, Saarland Informatics Campus, Saarbrücken, Germany 2 Department of Language Science and Technology, Saarland University, Saarbrücken, Germany {sweta.mahajan,sukrut.rao,schiele}@mpi-inf.mpg.de ∗

Equal contribution

arXiv:2606.07451v1 [cs.CV] 5 Jun 2026

Abstract Vision-language models such as CLIP are highly useful for diverse tasks due to their shared image-text embedding space. Despite this, the image and text embeddings are often poorly aligned, affecting downstream performance. Recent work has shown that this can be attributed to an information imbalance: images contain more information than their captions describe. In this work, we propose TEVI, a framework that uses captions as a signal for what to retain from image embeddings. Specifically, we use sparse autoencoders to disentangle image embeddings and train a masking module to selectively reconstruct the embedding based on a given caption. In a controlled setup with synthetic captions, we show that TEVI is effective at preserving caption-described attributes while discarding others. By applying TEVI to CLIP models trained on natural images, we further achieve improved retrieval performance across coarsegrained short-caption (MS COCO, Flickr) and fine-grained long-caption (IIW, DOCCI) benchmarks, with stronger gains on richer captions, and improved robustness on the RoCOCO benchmark.

Introduction

Vision-language models such as the CLIP family (Radford et al., 2021; Ilharco et al., 2021; Zhai et al., 2023; Tschannen et al., 2025) are trained using a contrastive loss to align images and text to a shared embedding space1 . Such models have been widely successful for a variety of multimodal applications, such as zero-/few-shot classification, cross-modal retrieval, and their embeddings serve as a vital bridge between vision and language for generative models such as text-to-image diffusion 1 Throughout this work, we use “vision-language model” to refer to CLIP-like models with aligned image-text encoders, and not autoregressive multimodal models that generate text.

models (Razzhigaev et al., 2023) and large multimodal models (LMMs) (Liu et al., 2023). However, it has been shown that the learnt embedding space suffers from a modality gap (Liang et al., 2022), where image and text embeddings lie in different regions of the embedding space, often leading to poor downstream performance (Liang et al., 2022; Eslami and de Melo, 2025; Schrodi et al., 2025). While various possible causes have been discussed (Eslami and de Melo, 2025; Schrodi et al., 2025), a recent work (Schrodi et al., 2025) hypothesizes that this is caused by an information imbalance—the fact that images contain more information than their corresponding captions, which forces models to push their embeddings apart when minimizing training loss—and shows evidence for this hypothesis via systematically controlled experiments on the synthetic MAD dataset. However, despite finding that only a few embedding dimensions characterize the gap, removing these dimensions also significantly degrades performance. In this work, we explore whether captions could instead determine what information should be preserved, by using text as a signal to modify CLIP image embeddings to preserve only what the caption describes. Specifically, we use sparse autoencoders (SAEs) (Bricken et al., 2023; Cunningham et al., 2024) to decompose image embeddings into constituent concepts, and train a conditioning module that selects SAE latents to be used for reconstruction based on the conditioning. Through a controlled setup, we show that our proposed approach, text-conditioned editing of visual representations for image-text alignment (TEVI) (Fig. 1, left), can learn to preserve attributes presented in the text while discarding information about other attributes and improve cross-modal alignment. We then apply TEVI to CLIP (Radford et al., 2021; Ilharco et al., 2021), SigLIP (Zhai et al., 2023), SharedCLIP, and AlignCLIP (Eslami and de Melo, 2025) models trained on CC12M (Changpinyo et al., 2021), and

CLIP Vision Encoder

TEVI

Flickr30k

Image Embedding

Original

Two black and white dogs fighting with each other

Two white dogs look at one black dog on a paved walkway.

DOCCI

Image

A low-angle view of a yellow rhombus-shaped road sign with a black illustration of the silhouette of a tortoise walking toward the left side of the image attached to the top of a gray metal pole

An outdoor medium shot, looking up at a yellow, weathered, diamond-shaped caution sign with a black horse and buggy facing the right

Original

CLIP Text Encoder

Text Embedding

Text Conditioned Image Embedding

a lady holding a atari with a big turkey in an open oven

A lady holding a pan with a big turkey in an open oven.

Cosine Similarity

A tractor is moving dirt to help build up a retaining wall.

Conditioning Module

RoCOCO

A tractor is moving dirt to help build up a retaining wall.

0.8 0.6 0.4 0.2

AlignCLIP CLIP SigLIP

22 0.8 0.6 0.4 0.2

TEVI

Text-to-Image

Image-to-Text

AlignCLIP CLIP SigLIP

R@1

Fig. 1: TEVI: using captions to edit image embeddings. Left: An overview of our approach. We use text captions as a signal to modify image embeddings from CLIP (Radford et al., 2021). For details, see Fig. 2. Middle: Qualitative examples of top texts retrieved for an image by CLIP and TEVI. Right: TEVI helps improve cross-modal alignment as well as downstream retrieval performance. We report mean performance across datasets, for full results see Tabs. 1 and 2 and Fig. 6.

show that applying TEVI improves (Fig. 1, middle and right) image-to-text and text-to-image retrieval performance, both for coarse-grained shortcaption (MS COCO (Lin et al., 2014), Flickr (Plummer et al., 2015)) and fine-grained long-caption retrieval (DOCCI (Onoe et al., 2024), IIW (Garg et al., 2024)). Notably, gains are larger on longcaption benchmarks which suggests that richer captions provide a stronger signal for editing. Using the RoCOCO (Park et al., 2024) benchmark, which augments captions with linguistically perturbed alternatives, we also show that TEVI leads to stronger robustness against such perturbations. In summary, our contributions are as follows: • CLIP-Guided SAEs (CG-SAEs), a controlled setup with latents that encode predefined text concepts, to study their utility for preserving targeted concepts. • TEVI, a framework that uses text as a signal to selectively retain caption-described content in image embeddings, via a learnt mask over sparse autoencoder latents. We use CG-SAEs to show a proof-of-concept of our proposed approach through controlled experiments on MAD (Schrodi et al., 2025), and show that TEVI leads to improved retrieval, robust retrieval, and cross-modal alignment across diverse real-world datasets.

Related Work

Vision-Language Models (VLMs) (Radford et al., 2021; Zhai et al., 2023; Ilharco et al., 2021; Yu et al., 2022; Jia et al., 2021; Tschannen et al., 2025) learn a joint aligned embedding space between images and texts. They typically consist of

separate unimodal image and text encoders that each provide an embedding, and are trained using contrastive losses so that embeddings of similar image-text pairs are placed close to each other and dissimilar pairs are placed apart. Such models are useful for a diverse set of multimodal tasks such as cross-modal retrieval and zero-shot classification. Their embeddings are also used as a bridge between vision and language for text-to-image diffusion models (Razzhigaev et al., 2023) and large multimodal models (LMMs) (Liu et al., 2023). In this work, we focus on CLIP-family models. Modality Gap (Schrodi et al., 2025; Liang et al., 2022; Eslami and de Melo, 2025; Mistretta et al., 2025; Shi et al., 2023) is a phenomenon observed in trained VLMs, where, despite the contrastive training objective, image and text embeddings lie in different regions of the embedding space. While initially attributed to the ‘cone effect’ at initialization (Liang et al., 2022), recent work (Schrodi et al., 2025) showed that a likely cause is the information imbalance between the two modalities, i.e., images contain more information than is described in their corresponding caption, which forces the model to push apart their embeddings to reduce the contrastive loss. Existing post hoc approaches to reduce the modality gap (Liang et al., 2022; Schrodi et al., 2025) have been shown to come at the cost of degraded performance. Recently, AlignCLIP (Eslami and de Melo, 2025) proposed to use an intramodal separation loss during CLIP training to improve alignment and downstream performance. In contrast, we explicitly control the information imbalance post hoc by using text as a signal to edit image embeddings, and show that this is complementary to methods such as AlignCLIP.

Image Embedding

SAE Encoder

Latents Mask

CLIP Vision Encoder

A tractor is moving dirt to help build up a retaining wall.

Edited Image SAE Decoder Embedding

= ❄

❄

λv,i A tractor is moving dirt to help build up a retaining wall.

Masked Latents

λv,i|i

Masking MLP

CLIP Text Encoder

λt,i

m ℒfInfoNCE

Fig. 2: Our proposed TEVI framework for obtaining text-conditioned image embeddings. We train a TopK SAE (Gao et al., 2025) over CLIP image embeddings, and then use an MLP trained using the InfoNCE loss to learn a mask over the SAE latents to obtain conditioned image embeddings. For details, see Secs. 4 and 5.

Cross-Modal Conditioning methods such as SmartCLIP (Xie et al., 2025b), FLAIR (Xiao et al., 2025), and FILIP (Yao et al., 2022) produce image embeddings conditioned on text for improved finegrained retrieval, at the cost of a small increase in retrieval time. An orthogonal line of work (Xie et al., 2025a; Asokan et al., 2025) performs finegrained alignment at training without conditioning at inference. Among the former methods, all except SmartCLIP train models from scratch. In contrast, TEVI is post-hoc: it adds a small module on top of pretrained CLIP, with the vision encoder kept frozen and only the text encoder fine-tuned. Similar to SmartCLIP, TEVI learns a mask over vision embeddings conditioned on text, but in contrast, TEVI masks disentangled SAE latents instead of raw representations.

Sparse Autoencoders (SAEs) (Bricken et al., 2023; Cunningham et al., 2024; Gao et al., 2025; Rajamanoharan et al., 2024a,b; Bussmann et al., 2025) are a popular mechanistic interpretability tool to disentangle activations learnt by a deep network into constituent human understandable concepts. While originally used in the context of LLMs (Bricken et al., 2023; Cunningham et al., 2024), SAEs have recently been used to decompose concepts from CLIP vision embeddings (Rao et al., 2024; Zaigrajew et al., 2025) for use in downstream tasks such as constructing concept bottleneck models (Rao et al., 2024). SAEs have also been used as a tool for steering models by performing edits to their latents (Farrell et al., 2025; Cywiński and Deja, 2025; Pach et al., 2025; Joseph et al., 2025). In contrast to these works which apply SAE-based interventions for interpretability or steering, we use them for text-conditioned editing aimed at improving cross-modal alignment.

Using SAEs to Edit Representations

In this section, we motivate our text-conditioned image editing approach. In Sec. 3.1, we first provide a brief overview on CLIP (Radford et al., 2021) and sparse autoencoders (SAEs) (Bricken et al., 2023). Then, to test whether SAEs can be used to edit representations, we describe a controlled test on a synthetic setup in Sec. 3.2 and present our findings in Sec. 3.3. We then present our proposed TEVI in Sec. 4, and in Sec. 5, we show how we extend it to real data. 3.1

Background

CLIP (Radford et al., 2021). Let D = {(vi , ti )}N i=1 be a paired dataset of images vi and their corresponding texts ti . A CLIP model M = (fv , ft ) consists of a vision and text encoder respectively which provide corresponding embeddings (εv,i , εv,t ), i.e. εv,i = fv (vi ) ∈ Rd and εt,i = ft (ti ) ∈ Rd . The model is then trained with a contrastive InfoNCE loss (Oord et al., 2018): " N 1 X eε̂v,i ·ε̂t,i /τ log PN LInfoNCE = − ε̂v,i ·ε̂t,j /τ 2N j=1 e i=1 # eε̂v,i ·ε̂t,i /τ + log PN (1) ε̂v,j ·ε̂t,i /τ j=1 e where τ is a learnable temperature hyperparameter x and x̂ = ∥x∥ . Eq. (1) pulls embeddings of cor2 responding (positive) image-text pairs (εv,i , εt,i ) close to each other and pushes embeddings of other (negative) image-text pairs (εv,i , εt,j ), i ̸= j away from each other to learn a shared semantic embedding space. Sparse Autoencoders (Bricken et al., 2023) consist of a linear encoder WE ∈ Rd×d1 and decoder WD ∈ Rd1 ×d where typically d1 ≫ d.

Attribute Specificity of ‘Swelling’ SAE Latents

The encoder maps an input x ∈ Rd to latents z = ReLU(WET (x − bpre ) + bE ), where bpre ∈ Rd , bE ∈ Rd1 are learnt biases and z is a sparse disentangled concept representation of x. The decoder then reconstructs x using z, i.e. x̃ = WDT z + bpre . The SAE is trained with a combination of a reconstruction loss Lrecon = ∥x − x̃∥2 and sparsity loss Lsparse , i.e.

ROC AUC Value

1.0

No swelling

0.8

0.6

0.4

0.2

0 1 2 3 4 5 6 7 8 9

N o

ic kt T hin hi n c k in T e g N hi ning o nn sw in e g N Sw llin o el g fr lin ac g Fr tu ac re tu La re r Smge a G ll ra y Re G d re en B lu M Cy e ag an en Ye ta llo w

0.0

ROC AUC Value

1.0

Swelling

0.8

0.6

0.4

0.2

where λSAE is a hyperparameter. We use Lsparse = TopK(z) (Gao et al., 2025) where TopK(·) selects the top K activated latents in z for hyperparameter K. 3.2

Controlled Setup for Evaluating Representation Editing

ic kt T hin hi n ck in T e g N hi ning o nn sw in e g N Sw llin o el g fr lin ac g Fr tu ac re tu La re r Smge a G ll ra y Re G d re en B lu M Cy e ag an en Ye ta llo w

(2)

0 1 2 3 4 5 6 7 8 9

0.0

LSAE = Lrecon + λSAE Lsparse ,

Fig. 3: Attribute specificity of CG-SAE latents. We plot the area under the receiver operating characteristic (ROC) curve (AUC) for CG-SAE latents corresponding to the attribute values ‘No swelling’ and ‘Swelling’. We find that the latents are highly attribute specific, with the AUC being close to 1 for the attribute the latent is assigned to, and 0 for unrelated attributes. This shows that our CG-SAE latents are highly disentangled, despite being assigned to a predefined concept. Interestingly, as can be expected, the AUC of attributes anti-correlated to the attribute assigned to the latent (e.g. ‘Swelling’ for the ‘No swelling’ latent) are close to 0. Results for the other latents are provided in the supplement.

To understand whether subselecting SAE latents before reconstruction can effectively edit embeddings, we construct a controlled setup using synthetic data with known attributes, where we induce each SAE latent to represent a predefined attribute. In contrast to a typical setup where SAEs automatically learn to disentangle concepts, this is designed to allow for targeted editing, and we refer to it as CLIP-guided SAE (CG-SAE). Let C = {cj }Sj=1 be a set of text concepts, and let (WE , WD ) be an SAE to be trained on image embeddings of CLIP. Then, for an embedding εv,i ,

The edited image embedding of an image vp conditioned on text tq is then given by ε] v,p|q , where

zi = TopK(ReLU(WET (εv,i − bpre ) + bE )) (3)

T ε] v,p|q = WD zp|q + bpre .

T εf v,i = WD zi + bpre

(4)

1 where zi = [zi,r ]dr=1 such that elements in zi out-

side the top K elements are zeros. Recent work (Rao et al., 2024) showed that latents zi,r could be assigned meaningful concept names in cq ∈ C post hoc by selecting the text embedding that is closest to their corresponding decoder weight vector WD,r , i.e.

Once trained, given an attribute set CA = {cq } ⊆ C, we select latents not present in the text, i.e. C̄ = C \ CA and set their activations to zero, i.e. zp|q = [zp ]zp,c =⃗0 , ∀cq′ ∈ C̄ q′

(7)

(8)

We evaluate the following: (R1) if each CG-SAE latent indeed encodes the concept assigned to it, and (R2) if masking out specific attributes removes information about that attribute from the reconstructed embedding from the SAE. We present our findings in Sec. 3.3. Top Activating Images of SAE Latents

Blue

Thinning

q = arg max cos(WD,r ; cj ) . j

(5) Fracture

Inspired by this, for our conditioning, we propose to attempt the opposite—given C, we fix the rows of the decoder weights WD to be text embeddings of concepts cj ∈ C, i.e. WD,r = ε̂t,cr

(6)

where 1 ≤ r ≤ S and d1 = S, and then only train the SAE encoder.

Fig. 4: Qualitative examples of top activating images for the setup when the CG-SAE is trained with fixed semantics of latents. Each row corresponds to an SAE latent and is labelled with the predefined concept that is assigned to that latent (Sec. 3.2). The columns show examples of images that maximally activate these latents. We find that the SAE learns to disentangle the CLIP image features into concepts as specified by the fixed weight of the latent.

1.0

60000

Thickthinning Swelling Fracture Scaling Colors Digit

50 25 0

al rigin

n. Reco

d blate

0.8

0.4 0.2 0.0

Original TEVI

50000

0.6

Frequency

Category

Accuracy

Accuracy (%)

100

40000 30000 20000 10000

Thickthinning Swelling

Fracture

Scaling

Colors

0.3

Random Chance

Original

Attribute in Caption

Attribute not in Caption

0.4

0.5 0.6 0.7 0.8 0.9 Cosine Similarity

1.0

Fig. 5: Left: Accuracy after ablating a single latent. Accuracy for ‘Thickthinning’ drops to random chance (dotted line) when editing image embeddings to discard information about that attribute (right group), while the other attributes continue to maintain high accuracy (Sec. 3). Middle: Effectiveness of conditioning. Attributes present in the conditioning text are preserved in the edited embedding, while attributes that are absent are classified at close to random chance accuracy (Sec. 4). Right: Impact on vision-language alignment. Pairwise image-text cosine similarities between positive pairs increase on applying TEVI (Sec. 4).

3.3

Results

Experimental Setup. We use the synthetic MAD dataset (Schrodi et al., 2025), which consists of MNIST digits with colors and morphological transforms. Every image is characterized by six attributes—digit, color, thickthinning, swelling, scaling, and fracture—each of which can be one among a set of predefined options, which are 26 in total. For example, the ‘thickthinning’ attribute for an image could be ‘thickening’, ‘thinning’ or ‘nothickthinning’, depending on the transform applied to the image. For each image, we create captions that contain the digit and three randomly selected attributes out of the remaining five to simulate information imbalance (Schrodi et al., 2025). Following (Schrodi et al., 2025), we train CLIP models (Radford et al., 2021; Ilharco et al., 2021) consisting of a 6-layer ViT (Dosovitskiy et al., 2021) as the vision encoder and a 6-layer transformer as the text encoder, with a shared embedding dimension of 18, for 200 epochs on using the AdamW (Loshchilov and Hutter, 2019) optimizer. Given the 26 possible attributes per image, the text encoder uses a tokenizer with a vocabulary of size 28, after accounting for the start and end tokens. Then, we train TopK SAE (Gao et al., 2025) on CLIP image embeddings with a latent dimension of 26 (equal to the number of attributes for the dataset). We assign each of the decoder weights as the normalized text embedding of one of the attribute values, so that each SAE latent represents one of the 26 attributes. For the SAE, we ablate across values of K, expansion factors, and learning rates, and select models based on reconstruction mean-squared accuracy and classification performance across the six attributes. Concept Disentanglement (R1). We evaluate if our CG-SAE learns to disentangle image embeddings into the fixed set of predefined concepts C. In Fig. 3, we show the area under the receiver op-

erating characteristic (ROC) curve (AUC) across images in the dataset for the two latents assigned to the ‘swelling’ attribute. We find that the latents are able to disentangle the attributes well, with the AUC being close to 1 for the attribute value the latent is assigned to, 0.5 for unrelated attributes, and 0 for attribute values anti-correlated to the assigned attribute (i.e. ‘no swelling’, for the ‘swelling’ latent). We also show qualitative examples of top activating images for a selection of latents in Fig. 4, and find that they are highly consistent, e.g. for the ‘blue’ latent, the top activating images are all digits of the color blue, while also being diverse in all other attributes. Editing Image Representations (R2). We evaluate if masking latents corresponding to a single attribute discards information about that attribute from the reconstructed representations. To do this, we pick a single attribute ‘thickthinning’, and set the three latents corresponding to it (‘thickening’, ‘thinning’, ‘nothickthinning’) to zero. This is similar to conditioning on all attributes q except ‘thickthinning’ (Eq. (7)). We then obtain edited embeddings for all images, and perform classification on all six attributes based on the cosine similarity with the text embeddings of the classes within the attribute, similar to zeroshot classification. For example, for ‘Thickthinning’, given an edited embedding εv,p|q , Predthickthinning = arg max cos(εv,p|q ; εt,j ), where tj ∈ {thickening, thinning, nothickthinning}. We find (Fig. 5, left) that the classification accuracy of the edited embeddings on ‘thickthinning’ drops to near random chance while that of all other attributes remains close to that with the original embedding. This shows that our conditioning can be an effective way to remove information present in the image embeddings that is not in the text.

Table 1: Fine-grained, long-caption retrieval performance on DOCCI (Onoe et al., 2024) and IIW (Garg et al., 2024). All the models are trained on CC12M dataset. We report R@1, R@5 and R@10 for both image-to-text and text-to-image retrieval. We see that TEVI improves retrieval performance across models and datasets.

R@1

DOCCI (Onoe et al., 2024) I→T T→I R@5 R@10 R@1 R@5

R@10

R@1

IIW (Garg et al., 2024) I→T T→I R@5 R@10 R@1 R@5

20.38 24.20 23.60 26.06 20.68 24.52 19.02 22.54 20.32 23.18

42.36 48.68 46.68 51.50 41.84 48.60 41.06 45.54 41.88 47.70

22.86 26.80 24.91 27.99 22.77 26.47 23.88 25.31 23.69 26.34

50.98 55.72 53.92 58.33 48.20 57.68 51.14 57.51 53.92 59.15

77.94 81.21 81.37 85.46 76.14 81.86 79.08 81.54 82.03 84.31

Model CLIP ViT-B/16 +TEVI CLIP ViT-L/14 +TEVI SigLIP ViT-B/16 +TEVI SharedCLIP +TEVI AlignCLIP ViT-B/16 +TEVI

53.52 60.06 57.06 62.20 53.70 59.58 52.68 57.10 53.38 58.90

7.16 8.55 8.21 9.26 7.41 8.47 7.16 7.95 7.39 8.32

16.96 19.98 18.70 21.51 17.17 20.02 17.52 18.62 17.43 19.76

TEVI: Learning to Mask Latents based on Conditioned Text

In Sec. 3.2, we showed that we could construct CG-SAEs for encoding and editing representations given predefined synthetic concepts. However, in practice, this is a restrictive assumption, since CLIP models are typically open vocabulary and a predefined set of concepts is not available. As a result, in this section, we relax this assumption and propose TEVI, an approach to learn a mask over SAE latents z to obtain the text-conditioned image embeddings. An overview of our approach is shown in Fig. 2. 4.1

Optimization Objective

Specifically, we keep the CLIP vision encoder and a trained SAE frozen, and train a small multi-layer perceptron (MLP) network consisting of linear transforms with ReLU activations fm : Rd → Rd1 that maps text embeddings εt,i to a mask mi ∈ [0, 1]d1 , i.e. mi = σ(fm (εt,i )) where σ(·) is the sigmoid function. Then, the edited image embedding of an image vp conditioned on text tq is given by εv,p|q , where εv,p|q = WDT (zp ⊙ mq ) + bpre

(9)

where ⊙ is the element-wise product. We train fm and fine-tune the CLIP text encoder ft using the InfoNCE loss (Eq. (1)) to pull the edited embeddings towards their conditioning texts and away from other texts, i.e. m LfInfoNCE =−

" N 1 X eε̂v,i|i ·ε̂t,i /τ log PN ε̂ ·ε̂ /τ v,i|i t,j 2N j=1 e i=1 # eε̂v,i|i ·ε̂t,i /τ + log PN ε̂ ·ε̂ /τ . (10) v,j|j t,i j=1 e

4.2

88.89 88.72 89.21 90.85 85.13 88.89 87.09 89.05 87.58 91.99

16.88 19.37 18.05 19.41 16.94 19.08 17.10 18.17 17.46 19.66

32.66 36.44 35.59 37.80 33.07 36.88 33.78 34.86 34.27 36.38

R@10 41.46 45.47 44.49 46.53 41.42 46.61 43.04 44.02 43.57 45.58

Results

Experimental Setup. We follow the setup from Sec. 3.3; however, we no longer fix the decoder weights of SAE and keep them learnable. We use 3-layer MLPs for the masking module. Effectiveness of Conditioning. We evaluate the effectiveness of conditioning for TEVI on actual captions, by evaluating for attribute-wise classification. Since each caption has a random subset of attributes present in the image (Sec. 3.3), we bin the attributes for each image depending on whether the caption contains it, and report the classification accuracy across the two bins in Fig. 5 (middle). We find that the classification accuracy on attributes present in the caption (orange) remains similar to the accuracy from the original embeddings (blue), while the accuracy from attributes absent from the caption (green) reach near random chance (dotted black line). This shows that TEVI is effective in performing conditioning such that the edited embedding preserves information present in the text while discarding the remaining attributes. Vision-language Alignment. Following Eslami and de Melo (2025), we also evaluate the pairwise image-text cosine similarities between positive image-text pairs before and after conditioning and the plots are shown in Fig. 5 (right). We find that for TEVI, the pairwise similarities increase significantly after conditioning (blue) as compared to before (orange).

Using TEVI to Improve Cross-modal Retrieval

In Sec. 4, we showed the utility of our proposed TEVI on a synthetic setup. In this section, we extend to CLIP models trained on natural images with the goal of improving alignment and retrieval performance.

Table 2: Coarse-grained, short-caption retrieval performance on MSCOCO (Lin et al., 2014) and Flickr30k (Plummer et al., 2015). All the models are trained on CC12M dataset. We report R@1, R@5 and R@10 for both image-to-text and text-to-image retrieval. We see that TEVI improves retrieval performance across models and datasets.

R@1

MS COCO (Lin et al., 2014) I→T T→I R@5 R@10 R@1 R@5

R@10

32.98 35.66 36.52 37.88 33.88 36.24 32.62 33.82 32.70 34.42

59.02 61.82 62.50 64.38 60.30 61.20 58.88 59.86 58.92 60.30

57.03 58.67 59.55 61.35 57.70 58.66 56.87 57.38 56.41 55.72

Model CLIP ViT-B/16 +TEVI CLIP ViT-L/14 +TEVI SigLIP ViT-B/16 +TEVI SharedCLIP ViT-B/16 +TEVI AlignCLIP ViT-B/16 +TEVI

5.1

70.04 72.94 73.06 75.16 70.62 72.82 69.94 70.90 70.46 71.48

21.38 23.12 23.85 25.04 22.06 22.37 21.54 22.13 21.79 21.35

45.30 47.03 48.42 49.74 46.21 46.60 44.81 45.95 44.55 44.39

Optimization Objective

Despite its effectiveness in the synthetic setup, the objective from Eq. (10)) only uses edited image embeddings ε̂v,i|i conditioned on their own corresponding positive captions, i.e. it does not model conditioning on negative captions. However, image-to-text and text-to-image retrieval involve selecting from a set of candidates, which would require conditioning both positive and negative pairs. So, we modify the objective to incorporate negative conditioning during training: " N 1 X eε̂v,i|i ·ε̂t,i /τ fm LInfoNCE = − log PN ε̂ ·ε̂ /τ v,i|j t,j 2N j=1 e i=1 # eε̂v,i|i ·ε̂t,i /τ + log PN ε̂ ·ε̂ /τ (11) v,j|i t,i j=1 e where e.g. eε̂v,j|j in the denominator of the first term from Eq. (10) is changed to eε̂v,i|j . Conditioning on negative captions is essential, without which the model performs poorly, which could be due to the fact that the model sees both positive and negative captions during inference. We discuss further in Sec. B.5. 5.2

Results

Experimental Setup. Following previous works (Eslami and de Melo, 2025; Goel et al., 2022; Mu et al., 2022; Li et al., 2022), we use the CC12M dataset (Changpinyo et al., 2021) to train a CLIP (Radford et al., 2021) ViT-B/16 (Dosovitskiy et al., 2021), CLIP ViT-L/14, and a SigLIP (Zhai et al., 2023) ViT-B/16 model. To train the SAE, we ablate across values for the expansion factors and learning rates and select the best configuration based on the mean reconstruction accuracy and classification accuracy on the reconstructed features. We then train the masking module by

R@1

Flickr30k (Plummer et al., 2015) I→T T→I R@5 R@10 R@1 R@5

R@10

59.66 64.20 63.71 65.48 62.23 62.82 60.75 59.37 57.49 61.24

83.73 85.70 86.49 87.77 85.40 86.19 84.22 83.33 82.35 83.53

79.31 81.22 81.66 82.50 79.33 80.73 78.86 79.53 78.97 78.92

90.14 91.12 91.32 92.70 91.42 91.62 89.64 89.05 89.94 89.15

42.46 44.75 46.51 47.14 44.69 43.77 43.31 44.30 41.91 42.31

70.33 72.10 72.84 74.58 70.61 71.56 69.53 70.97 70.01 70.33

sweeping across learning rates and selecting the configuration that is best for retrieval on the validation split of CC3M (Sharma et al., 2018), to ensure generalization to different datasets. Full details are provided in the supplement in Sec. B.1. Vision-Language Alignment. To evaluate the impact of TEVI on cross-modal alignment, we plot the cosine similarities between image and corresponding text embeddings, both before and after editing. We find (Fig. 6) that alignment improves across datasets after applying TEVI (e.g. 0.417→0.470 for MSCOCO). Further results on different models are provided in Sec. B.6. Note however that alignment is a diagnostic metric; with the primary goal being to improve downstream performance, which we discuss next. Retrieval Performance. We evaluate the performance of TEVI for text-to-image and imageto-text retrieval across fine-grained long-caption (DOCCI (Onoe et al., 2024), IIW (Garg et al., 2024)) and coarse-grained short-caption (MS COCO (Lin et al., 2014), Flickr30k (Plummer et al., 2015)) retrieval benchmarks. We find (Tabs. 1 and 2) that TEVI consistently improves retrieval performance across nearly all datasets and models, being particularly effective when used with long-captions. Qualitative examples are provided in Fig. 1 and Sec. B.8. Applicability to Existing Methods. We additionally compare against SharedCLIP and AlignCLIP (Eslami and de Melo, 2025), which recently proposed using intra-modal separation objectives during CLIP training to improve cross-modal alignment and retrieval performance. Even though these training objectives already provide gains, we find (Tabs. 1 and 2) that our approach can additionally help these methods with consistent and significant improvement on fine-grained datasets and competi-

Table 3: Robust retrieval performance of TEVI on the RoCOCO benchmark (Park et al., 2024). We find that TEVI improves performance over the baseline across settings. COCO Rand-voca Same-concept Diff-concept Danger R@1 R@1 drop rate RSMS R@1 drop rate RSMS R@1 drop rate RSMS R@1 drop rate (↑) (↓) (↓) (↑) (↓) (↓) (↑) (↓) (↓) (↑) (↓)

CLIP 33.26 +TEVI 36.00

21.26 26.14

12.00 9.86

46.90 33.98

MS COCO

200 0.2

0.3

0.4

22.12 25.60

0.5

Cosine Similarity

0.6

150

43.98 35.70

50 0.3

0.4

10.34 8.88

500 400

0.5

0.6

0.7

RSMS (↓) 39.74 31.00

IIW

80 Original TEVI

300 200 100

0.2

22.92 27.12

DOCCI

100

0 0.1

11.14 10.40

600 Original TEVI

Frequency

43.36 37.12

200 Original TEVI

400

0 0.1

11.52 10.26

Flickr30k

800 600

21.74 25.74

Frequency

Model

0 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

Cosine Similarity

Original TEVI

40 20 0 0.1

0.2

0.3

0.4

0.5

0.6

0.7

Cosine Similarity

Fig. 6: Cross-modal alignment across datasets using CLIP ViT-B/16. We find that the alignment between image-text pairs increases after applying TEVI.

tive performance on coarse-grained datasets. Cost of Inference. Similar to other cross-modal conditioning methods (e.g. Xie et al., 2025b; Xiao et al., 2025), TEVI incurs an additional cost for retrieval, since every image is conditioned on every text. However, since we edit final embeddings post hoc using small modules, the overhead is relatively small, e.g. around 2.7% increase in FLOPS for 1000 image-text pairs. One could obtain further speedups for all such conditioning methods including TEVI by first performing coarse retrieval with the CLIP model and then fine-grained ranking, which we leave for future work. Comparison to SmartCLIP. We compare TEVI against SmartCLIP (Xie et al., 2025b), which learns a text-conditioned mask on raw vision embeddings. For a fair comparison, we train on a similar setup with CC12M. We find (Tab. 4, and Sec. B.7) that TEVI consistently outperforms it on fine-grained datasets and is competitive on coarsegrained datasets, highlighting the benefit of disentangling representations. Additionally, similar to Xie et al. (2025b), we obtain comparable zeroshot performance as the baseline; for details refer to Sec. B.4. Robust Retrieval. We additionally evaluate TEVI for robust retrieval, using the RoCOCO (Park et al., 2024) benchmark. This augments the caption set of MS COCO (Lin et al., 2014) with perturbed captions containing irrelevant concepts that alter their meaning, and should not be retrieved by the model. Specifically, it contains four sets—‘Rand-voca’, ‘Danger’, ‘Same-concept’, and ‘Diff-concept’— which replace words in the original caption with alternatives based on the set. For example, a word ‘umbrella’ might be replaced by another tool (e.g. ‘rope’) by ‘Same-concept’, and with an unrelated

Table 4: Retrieval comparison with SmartCLIP. TEVI outperforms on fine-grained datasets and is competitive on coarsegrained datasets. Full results in Sec. B.7. Model

DOCCI I→T, T→I Flickr30k I→T, T→I R@1 R@5 R@1 R@5 R@1 R@5 R@1 R@5

CLIP 20.4 42.4 7.2 17.0 59.7 83.7 42.5 70.3 +TEVI 24.2 48.7 8.6 20.0 64.2 85.7 44.8 72.1 SmartCLIP 21.8 45.0 7.5 17.8 61.0 85.1 45.3 72.8

word (e.g. ‘pizza’) by ‘Diff-concept’. Following (Park et al., 2024), we report (Tab. 3) the drop rate (percentage drop in retrieval after augmenting captions), and Recall Score of Manipulated Samples (RSMS) (fraction of data where a perturbed entity was retrieved on top). We find that TEVI provides consistent improvements across these metrics on all data splits, showing its promise for improving model safety. Additional results on other backbones are provided in Sec. B.3.

Conclusion

In this work, we explored whether captions can guide what information is preserved in CLIP image embeddings, in order to improve cross-modal alignment. We proposed a framework, TEVI, to use text as a signal to edit image embeddings, such that the resultant edited embedding is better aligned with the text. We first used a synthetic setup, where we could explicitly control the information imbalance, to show that text-conditioned editing preserves information described in the caption. We then applied TEVI to models trained on real-world data and showed its effectiveness for improving retrieval performance and cross-modal alignment, and its utility in complementing existing approaches such as AlignCLIP. We show TEVI is particularly helpful for benchmarks with rich captions (DOCCI, IIW) and for retrieval robustness against linguistic perturbations (RoCOCO).

Limitations While our proposed TEVI framework shows strong benefits for a variety of CLIP models trained on CC12M, extending to large-scale models trained on billions of data points remains a challenge, likely owing to the complexity of training sufficiently large and diverse SAEs. Nevertheless, our work intends to provide a clear proof of concept. Scaling to models trained on large-scale data would be a fruitful direction for future research. In addition, our work relies on SAEs for disentanglement, which may not always be accurate. As with all cross-modal conditioning-based methods, TEVI also requires conditioning every image on every text before retrieval, which incurs an additional computational cost. However, this could be partly addressed by performing coarse retrieval first and then reranking with TEVI, which we leave for future work.

Acknowledgements Funded in part by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - GRK 2853/1 “Neuroexplicit Models of Language, Vision, and Action” - project number 471607914.

References Mothilal Asokan, Kebin Wu, and Fatima Albreiki. 2025. FineLIP: Extending CLIP’s Reach via Fine-Grained Alignment with Longer Text Inputs. In CVPR, pages 14495–14504. Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, and 6 others. 2023. Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. Transformer Circuits Thread. Bart Bussmann, Noa Nabeshima, Adam Karvonen, and Neel Nanda. 2025. Learning Multi-Level Features with Matryoshka Sparse Autoencoders. In ICML. Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. 2021. Conceptual 12M: Pushing WebScale Image-Text Pre-training to Recognize LongTail Visual Concepts. In CVPR, pages 3558–3568. Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. 2024. Sparse Autoencoders Find Highly Interpretable Features in Language Models. In ICLR.

Bartosz Cywiński and Kamil Deja. 2025. SAeUron: Interpretable Concept Unlearning in Diffusion Models with Sparse Autoencoders. In ICML. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In ICLR. Sedigheh Eslami and Gerard de Melo. 2025. Mitigate the Gap: Investigating Approaches for Improving Cross-modal Alignment in CLIP. In ICLR. Eoin Farrell, Yeu-Tong Lau, and Arthur Conmy. 2025. Applying Sparse Autoencoders to Unlearn Knowledge in Language Models. In ICLR. Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. 2025. Scaling and Evaluating Sparse Autoencoders. In ICLR. Roopal Garg, Andrea Burns, Burcu Karagol-Ayan, Yonatan Bitton, Ceslee Montgomery, Yasumasa Onoe, Andrew Bunner, Ranjay Krishna, Jason Michael Baldridge, and Radu Soricut. 2024. ImageInWords: Unlocking Hyper-Detailed Image Descriptions. In EMNLP, pages 93–127. Shashank Goel, Hritik Bansal, Sumit Bhatia, Ryan Rossi, Vishwa Vinay, and Aditya Grover. 2022. CyCLIP: Cyclic Contrastive Language-Image Pretraining. In NeurIPS, volume 35, pages 6704–6719. Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Lixuan Zhu, Samyak Parajuli, Mike Guo, and 1 others. 2021. The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization. In ICCV. Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Xiaodong Song. 2019. Natural Adversarial Examples. In CVPR, pages 15257–15266. Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. 2021. OpenCLIP. Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling Up Visual and Vision-Language Representation Learning with Noisy Text Supervision. In ICML, pages 4904–4916. Sonia Joseph, Praneet Suresh, Ethan Goldfarb, Lorenz Hufe, Yossi Gandelsman, Robert Graham, Danilo Bzdok, Wojciech Samek, and Blake Aaron Richards. 2025. Steering CLIP’s Vision Transformer with Sparse Autoencoders. arXiv preprint arXiv:2504.08729.

Alex Krizhevsky, Geoffrey Hinton, and 1 others. 2009. Learning Multiple Layers of Features from Tiny Images. Technical Report, Computer Science Department, University of Toronto. Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. 2022. Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm. In ICLR. Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. 2022. Mind the Gap: Understanding the Modality Gap in MultiModal Contrastive Representation Learning. In NeurIPS. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context. In ECCV, pages 740– 755. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual Instruction Tuning. In NeurIPS. Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. In ICLR. Marco Mistretta, Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, and Andrew D Bagdanov. 2025. Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion. In ICLR. Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. 2022. SLIP: Self-supervision meets Language-Image pre-training. In ECCV, pages 529– 544. Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg, Alexander Ku, Zarana Parekh, Jordi Pont-Tuset, Garrett Tanzer, and 1 others. 2024. DOCCI: Descriptions of Connected and Contrasting Images. In ECCV, pages 291–309. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation Learning with Contrastive Predictive Coding. arXiv preprint arXiv:1807.03748. Mateusz Pach, Shyamgopal Karthik, Quentin Bouniot, Serge Belongie, and Zeynep Akata. 2025. Sparse Autoencoders Learn Monosemantic Features in Visionlanguage Models. In NeurIPS, volume 38, pages 95706–95742. Seulki Park, Daeho Um, Hajung Yoon, Sanghyuk Chun, and Sangdoo Yun. 2024. RoCOCO: Robustness Benchmark of MS-COCO to Stress-Test Image-Text Matching Models. In ECCV, pages 71–91. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, and 1 others. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In NeurIPS.

Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Imageto-Sentence Models. In ICCV, pages 2641–2649. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. Learning Transferable Visual Models from Natural Language Supervision. In ICML, pages 8748–8763. Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. 2024a. Improving Dictionary Learning with Gated Sparse Autoencoders. In NeurIPS. Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. 2024b. Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders. arXiv preprint arXiv:2407.14435. Sukrut Rao, Sweta Mahajan, Moritz Böhle, and Bernt Schiele. 2024. Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discovery. In ECCV. Anton Razzhigaev, Arseniy Shakhmatov, Anastasia Maltseva, Vladimir Arkhipkin, Igor Pavlov, Ilya Ryabov, Angelina Kuts, Alexander Panchenko, Andrey Kuznetsov, and Denis Dimitrov. 2023. Kandinsky: An Improved Text-to-Image Synthesis with Image Prior and Latent Diffusion. In EMNLP (Demos). Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. 2019. Do ImageNet Classifiers Generalize to ImageNet? In ICML, pages 5389– 5400. PMLR. Simon Schrodi, David T Hoffmann, Max Argus, Volker Fischer, and Thomas Brox. 2025. Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models. In ICLR. Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset for Automatic Image Captioning. In ACL, pages 2556– 2565. Peiyang Shi, Michael C Welle, Mårten Björkman, and Danica Kragic. 2023. Towards Understanding the Modality Gap in CLIP. In ICLR 2023 Workshop on Multimodal Representation Learning: Perks and Pitfalls. Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, and 1 others. 2025. SigLIP

2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features. arXiv preprint arXiv:2502.14786. Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. 2019. Learning Robust Global Representations by Penalizing Local Predictive Power. In NeurIPS, volume 32. Rui Xiao, Sanghwan Kim, Mariana-Iuliana Georgescu, Zeynep Akata, and Stephan Alaniz. 2025. FLAIR: VLM with Fine-grained Language-informed Image Representations. In CVPR. Chunyu Xie, Bin Wang, Fanjing Kong, Jincheng Li, Dawei Liang, Gengshen Zhang, Dawei Leng, and Yuhui Yin. 2025a. FG-CLIP: Fine-grained Visual and Textual Alignment. In ICML. Shaoan Xie, Lingjing Lingjing, Yujia Zheng, Yu Yao, Zeyu Tang, Eric P Xing, Guangyi Chen, and Kun Zhang. 2025b. SmartCLIP: Modular Visionlanguage Alignment with Identification Guarantees. In CVPR, pages 29780–29790. Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. 2022. FILIP: Finegrained Interactive Language-image Pre-training. In ICLR. Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. 2022. CoCa: Contrastive Captioners are Image-Text Foundation Models. TMLR. Vladimir Zaigrajew, Hubert Baniecki, and Przemyslaw Biecek. 2025. Interpreting CLIP with Hierarchical Sparse Autoencoders. In ICML. Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid Loss for Language Image Pre-Training. In ICCV, pages 11975–11986.

TEVI: Text-Conditioned Editing of Visual Representations via Sparse Autoencoders for Improved Vision-Language Alignment Appendix In this appendix, we provide implementation details and additional results. Sec. A covers our controlled synthetic setup and Sec. B provides additional details and results for TEVI applied on CLIP models trained on CC12M. In Sec. C, we briefly discuss broader impact, and in Sec. D, we enumerate artifacts used. (A) Controlled Synthetic Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 (A.1) Implementation Details (A.2)

Additional Results for Concept Disentanglement with CG-SAE

(A.3)

Additional Results for Effectiveness of Conditioning

(B) Evaluation on CLIP Models Trained with Natural Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 (B.1) Implementation Details (B.2)

Baseline Fine-tuning Comparison

(B.3)

Additional Results on RoCOCO

(B.4)

Zeroshot Classification Results

(B.5)

Ablation without Negative Conditioning

(B.6)

Cross-modal Alignment

(B.7)

Comparison against SmartCLIP

(B.8)

Additional Qualitative Examples

(C) Broader Impact . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 (D) Licenses of Artifacts Used . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Controlled Synthetic Setup

More implementation details about the method on the MAD dataset (Schrodi et al., 2025) is as follows. This is in addition to the details provided in Sec. 3.3. A.1

Implementation Details

Dataset. The MAD dataset (Schrodi et al., 2025) is a synthetic dataset that consists of images of digits with colors and morphological transforms. Specifically, it consists of the following attributes: ‘Digit’: {0,1,2,3,4,5,6,7,8,9}, ‘Thickthinning’: {thickening, thinning, no thickthinning}, ‘Scaling’: {large, small}, ‘Fracture’: {fracture, no fracture}, ‘Swelling’: {swelling, no swelling}, and ‘Color’: {gray, red, green, blue, cyan, magenta, yellow}, which gives a total of 6 attribute categories and an aggregate of 26 attribute values. The training dataset consists of 1.44 million images, and the test data consists of 240,000 images. For training CLIP models, captions are generated by using the digit and a random sample of three out of the remaining five attribute categories in the image, placed in a random order with a ‘-’ separator.

spectively for both the SAEs used for CG-SAE and TEVI. Learnt masks. We use a 3-layer MLP with a hidden dimension of 256 and ReLU activations between linear layers. This MLP predicts values to mask the SAE latents. We sweep over learning rates 0.1, 0.01, and 0.001, and choose 0.01 as the optimal learning rate. We train for 25 epochs with 5 warmup epochs using the AdamW optimizer (Loshchilov and Hutter, 2019). A.2

Additional Results for Concept Disentanglement with CG-SAE

In this section, we provide full results for concept disentanglement from our CG-SAEs, as discussed in Sec. 3.3. In Figs. A3 and A4, we show AUC ROC plots for other attribute categories, to supplement the results provided for the ‘Swelling’ category in Fig. 3. In Figs. A1 and A2, we show top activating images for each latent, expanding on Fig. 4. Both quantitatively and qualitatively, we find that our CG-SAE effectively disentangles concepts and each latent activates highly only on the attribute value assigned to it. Top Activating Images of SAE Latents

CLIP models. Following (Schrodi et al., 2025), we train the CLIP model using the AdamW (Loshchilov and Hutter, 2019) optimizer for 200 epochs with a batch size of 128 and a weight decay of 0.1. We sweep over learning rates of {10−5 , 5 × 10−4 , 5 × 10−5 } and pick the learning rate 5 × 10−4 and the final checkpoint with the lowest loss. We use the cosine annealing for the learning rate. For finetuning the text encoder along with the learnt mask, we sweep over learning rates {10−2 , 10−3 , 10−4 , 5 × 10−4 , 10−5 , 10−6 } and pick the learning rate 10−6 . Our code is based on the implementation from OpenCLIP (Ilharco et al., 2021) using PyTorch (Paszke et al., 2019). SAE models. We use the TopK SAE implementation of (Gao et al., 2025). This particular type of SAE chooses the top few (= K) SAE latents to reconstruct the original CLIP embeddings and uses an auxiliary loss that approximates the reconstruction error using the top few (=auxK) dead latents. For our setup, we use K as 12, auxK as 18, after sweeping over these hyperparameters and we train the SAE for 200 epochs. We choose the SAE configuration using with the lowest reconstruction error. We sweep over learning rates {10−1 , 10−2 } and expansion factor 1, 2, 4 and pick 10−2 , 1 re-

Fig. A1: Qualitative examples of top activating images for all the digits of the CG-SAE latents. Each row corresponds to a CG-SAE latent and is labelled with the predefined concept that is assigned to that latent (Sec. 3.2). The columns show examples of images that maximally activate these latents. We find that the CG-SAE learns to disentangle the CLIP image features into concepts as specified by the fixed weight of the latent.

Table A1: Effectiveness of conditioning with CG-SAE for the MAD dataset. For each attribute category (rows), we report the accuracy with the original embeddings (col. 1) and conditioned embeddings, where the attribute of that category is present in the caption (col. 2) and is absent from the caption (col. 3), averaged over five runs. We see that when the attribute is present in the caption, the accuracy is at par with the original embeddings. However, when the attribute is absent from the caption, the accuracy reaches close to random chance (col. 4), which shows that conditioning is effective in only preserving information about attributes present in the caption.

Thickthinning Swelling Fracture Scaling Color

Original (↑)

Attribute in Caption (↑)

Attribute not in Caption

Random Chance

92.9 96.2 92.8 99.7 100.0

97.8 ± 1.3 96.6 ± 6.8 97.5 ± 3.0 99.9 ± 0.1 100.0 ± 0.0

35.4 ± 2.1 50.2 ± 0.1 50.5 ± 0.7 60.7 ± 3.4 34.2 ± 1.7

33.3 30.0 50.0 50.0 14.3

Top Activating Images of SAE Latents

AdamW (Loshchilov and Hutter, 2019) optimizer for 30 epochs with a batch size of 2048 and a weight decay of 0.1. We sweep over learning rates of {10−3 , 5 × 10−4 } and pick the learning rate 10−3 and the final checkpoint with the lowest loss. We use cosine annealing for the learning rate. For fine-tuning the text encoder along with training the masking module, we sweep over learning rates {10−4 , 10−5 , 10−6 }. Our code is based on the implementation from OpenCLIP (Ilharco et al., 2021; Eslami and de Melo, 2025) using PyTorch (Paszke et al., 2019).

No thickthinning

Thickening

No swelling

Swelling

No fracture

Large

Small

Gray

Red

Green

Cyan

Magenta

Yellow

Fig. A2: Qualitative examples of top activating images of the remaining CG-SAE latents. Each row corresponds to a CG-SAE latent and is labelled with the predefined concept that is assigned to that latent (Sec. 3.2). The columns show examples of images that maximally activate these latents. We find that the CG-SAE learns to disentangle the CLIP image features into concepts as specified by the fixed weight of the latent.

A.3

Additional Results for Effectiveness of Conditioning

In this section, we provide additional results on our method’s effectiveness of conditioning (Tabs. A1 and A2). Specifically, for robustness, we report results averaged across five runs, and find that the trends observed in Fig. 5 continue to hold.

Evaluation on CLIP Models Trained with Natural Images

B.1

Implementation Details

SAE models. We use the TopK SAE implementation of (Gao et al., 2025). This particular type of SAE chooses the top few (= K) SAE latents to reconstruct the original CLIP embeddings and uses an auxiliary loss that approximates the reconstruction error using the top few (=auxK) dead latents. For our setup, we vary the sparsity (K) value across values {64, 128, 256}, and discard the configuration where the SAE results in a large number of dead decoding vectors, and finally choose the sparsity value as 128, which gives the lowest reconstruction error and highest zeroshot accurracy on the reconstructed features. We use auxK as 256, after sweeping over these hyperparameters and we train the SAE for 50 epochs. We sweep over learning rates {10−3 , 5 × 10−3 , 10−4 , 5 × 10−4 }, and use an expansion factor of 32 for ViT-B/16 backbones and 16 for ViT-L/14 backbones. We choose the SAE configuration using with the lowest reconstruction error and lowest number of dead nodes for different setups.

In this section, we provide implementation details. Code will be made available on publication.

Learnt masks. We use a 3-layer MLP and ReLU activations between linear layers. This MLP predicts values to mask the SAE latents. We sweep over learning rates {10−4 , 10−5 , 10−6 }. We train for 3 epochs using the AdamW optimizer (Loshchilov and Hutter, 2019).

CLIP models. We train CLIP models using the

Baselines. For the results on SharedCLIP and

Table A2: Effectiveness of conditioning with TEVI for the MAD dataset. For each attribute category (rows), we report the accuracy with the original embeddings (col. 1) and conditioned embeddings, where the attribute of that category is present in the caption (col. 2) and is absent from the caption (col. 3), averaged over five runs. We see that when the attribute is present in the caption, the accuracy is at par with the original embeddings. However, when the attribute is absent from the caption, the accuracy reaches close to random chance (col. 4), which shows that conditioning is effective in only preserving information about attributes present in the caption.

Thickthinning Swelling Fracture Scaling Color

Original (↑)

Attribute in Caption (↑)

Attribute not in Caption

Random Chance

92.9 96.2 92.8 99.7 100.0

99.1 ± 0.7 98.3 ± 0.2 93.5 ± 0.9 95.2 ± 2.7 96.3 ± 1.9

54.3 ± 2.4 57.0 ± 2.7 59.1 ± 1.5 57.1 ± 3.5 22.6 ± 1.9

33.3 30.0 50.0 50.0 14.3

AlignCLIP in Tabs. 1 and 2, we take the checkpoints given by the authors (Eslami and de Melo, 2025) and apply TEVI on them. Note that these checkpoints were trained with a batch size of 512, and in contrast we train our own CLIP models with a batch size of 2048. As a result, the performance of AlignCLIP/SharedCLIP versus our CLIP models cannot be directly compared. Compute usage. We used H100 and A100 GPUs for training the CLIP models on CC12M, training the SAEs, and the masking module. The maximum number of GPU hours needed for training the CLIP model is upper bounded by 18 hours, and for training the masking module is upper bounded by 15 hours. Robust retrieval. Following (Park et al., 2024), in Tab. 3, we report the drop rate and the RSMS metric in addition to the retrieval metrics. Drop rate is calculated as (R@1−Rp@1) , where Rp is the R@1 retrieval score under the perturbed setting. RSMS calculates the percentage of newly added semantically perturbed captions that are retrieved in the first spot by the model. We note that, due to that use of SAE to decompose CLIP image features into concepts, our method is unable to work with altered images as given by the benchmark, where the original image is superimposed with a random patch from a different image. As a result, we only report image-to-text retrieval performance in Tab. 3. B.2

Baseline Fine-tuning Comparison

TEVI involves training a masking module for a small number of extra epochs while also fine-tuning the text encoder. For a fairer comparison accounting for this extra training budget, we report in Tabs. B1 and B2 the performance of CLIP models where the text encoder is fine-tuned for the same number of extra epochs. We find that TEVI nevertheless shows improved retrieval performance across datasets.

B.3

Additional Results on RoCOCO

In addition to the robust retrieval results provided in Tab. 3 on the CLIP ViT-B/16 backbone, here we provide additional results comparing TEVI against SigLIP and AlignCLIP. B.4

Zeroshot Classification Results

Similar to SmartCLIP (Tab. 3 in Xie et al., 2025b), we find (Tab. B4) that the zeroshot classification performance is comparable to the baseline, possibly since zeroshot does not use long captions with multiple attributes, unlike retrieval. Since our method is post hoc, one can achieve the best of both worlds by using the original CLIP embeddings for zeroshot classification. B.5

Ablation without Negative Conditioning

As noted in Sec. 5.1, we use Eq. (11) when applying TEVI on CLIP models trained with CC12M, to additionally provide conditioning signal on negative captions. In Tabs. B5 and B6, we empirically validate this by performing an ablation study against models trained without negative conditioning, i.e. using Eq. (10), and find that the performance degrades significantly. B.6

Cross-modal Alignment

Similar to Fig. 6, Figs. B1 and B2 show crossmodal alignment for SharedCLIP and AlignCLIP across four downstream datasets before and after applying TEVI, and find that it similarly improves alignment. B.7

Comparison against SmartCLIP

In Tabs. B7 and B8, we compare our approach against SmartCLIP (Xie et al., 2025b), applied on the same CLIP models trained on CC12M. We see that TEVI performs comparably on MS COCO and Flickr30k and outperforms on the fine-grained DOCCI and IIW datasets, possibly due to explicit disentanglement from SAEs.

Table B1: Coarse-grained retrieval performance on MSCOCO (Lin et al., 2014) and Flickr30k (Plummer et al., 2015) as compared to fine-tuning CLIP. All the models are trained on CC12M dataset. We report R@1, R@5 and R@10 for both image-to-text and text-to-image retrieval. We see that TEVI improves retrieval performance across models and datasets in comparison to baseline finetuning as well.

R@1

MS COCO (Lin et al., 2014) I→T T→I R@5 R@10 R@1 R@5

R@10

32.98 35.66 35.60

59.02 61.82 60.78

57.03 58.67 58.13

Model CLIP ViT-B/16 +TEVI Fine-tuned

70.04 72.94 71.62

21.38 23.12 22.50

45.30 47.03 46.40

R@1

Flickr30k (Plummer et al., 2015) I→T T→I R@5 R@10 R@1 R@5

R@10

59.66 64.20 63.21

83.73 85.70 86.29

79.31 81.22 80.47

90.14 91.12 90.93

42.46 44.75 44.12

70.33 72.10 71.50

Table B2: Fine-grained retrieval performance on DOCCI (Onoe et al., 2024) and IIW (Garg et al., 2024) as compared to fine-tuning CLIP. All the models are trained on CC12M dataset. We report R@1, R@5 and R@10 for both image-to-text and text-to-image retrieval. We see that TEVI improves retrieval performance across models and datasets in comparison to baseline finetuning as well.

R@1

DOCCI (Onoe et al., 2024) I→T T→I R@5 R@10 R@1 R@5

R@10

R@1

IIW (Garg et al., 2024) I→T T→I R@5 R@10 R@1 R@5

20.38 24.20 23.54

42.36 48.68 46.48

22.86 26.80 25.00

50.98 55.72 53.10

77.94 81.21 80.56

Model CLIP ViT-B/16 +TEVI Fine-tuned

53.52 60.06 56.92

7.16 8.55 7.79

16.96 19.98 18.57

88.89 88.72 88.23

16.88 19.37 18.34

32.66 36.44 35.99

R@10 41.46 45.47 45.20

Table B3: Robust retrieval performance of TEVI on the RoCOCO benchmark (Park et al., 2024). We find that TEVI improves performance over the baseline across settings. Model

COCO Rand-voca Same-concept Diff-concept Danger R@1 R@1 drop rate RSMS R@1 drop rate RSMS R@1 drop rate RSMS R@1 drop rate (↑) (↓) (↓) (↑) (↓) (↓) (↑) (↓) (↓) (↑) (↓)

SigLIP 33.66 +TEVI 36.46 AlignCLIP 32.86 +TEVI 34.70

21.98 26.32 21.46 22.32

11.68 10.14 11.40 12.38

44.16 33.58 43.66 43.70

22.38 26.14 22.72 22.58

11.28 10.32 10.14 12.12

42.12 35.90 40.96 44.16

22.66 27.06 21.94 22.74

11.00 9.40 10.92 11.96

41.84 32.74 41.20 44.48

24.98 29.14 23.44 25.48

8.68 7.32 9.42 9.22

RSMS (↓) 31.96 24.28 37.52 35.20

Table B4: Zeroshot performance across benchmarks. We evaluate on ImageNet-V2 (Recht et al., 2019), ImageNetSketch (Wang et al., 2019), ImageNet-A (Hendrycks et al., 2019), ImageNet-O (Hendrycks et al., 2019), ImageNet-R (Hendrycks et al., 2021), and CIFAR-100 (Krizhevsky et al., 2009). Model

IMN V2

Sketch

-A

-O

-R

CIF100

CLIP ViT-B/16 +TEVI

29.24 28.60

21.93 21.93

7.64 7.42

26.20 25.90

42.69 42.76

28.07 30.66

Table B5: Coarse-grained retrieval performance of CLIP ViT-B/16 when we do not use the negative conditioning on MSCOCO (Lin et al., 2014) and Flickr30k (Plummer et al., 2015). All the models are trained on CC12M dataset. We report R@1, R@5 and R@10 for image-to-text and text-to-image retrieval.

R@1

MS COCO (Lin et al., 2014) I→T T→I R@5 R@5 R@10 R@1

R@10

32.98 10.28

59.02 26.24

57.03 32.84

Model CLIP ViT-B/16 No neg.

70.04 36.48

21.38 9.65

45.30 23.77

R@1

Flickr30k (Plummer et al., 2015) I→T T→I R@5 R@10 R@1 R@5

R@10

59.66 26.04

83.73 53.65

79.31 56.82

90.14 63.81

42.46 22.47

70.33 46.07

Table B6: Fine-grained retrieval performance of CLIP ViT-B/16 when we do not use the negative conditioning on DOCCI (Onoe et al., 2024) and IIW (Garg et al., 2024). All the models are trained on CC12M dataset. We report R@1, R@5 and R@10 for image-to-text and text-to-image retrieval.

R@1

DOCCI (Onoe et al., 2024) I→T T→I R@5 R@10 R@1 R@5

R@10

20.38 5.54

42.36 14.50

22.86 11.90

Model CLIP ViT-B/16 No neg.

B.8

53.52 20.88

7.16 3.07

16.96 8.29

Additional Qualitative Examples

We provide additional randomly sampled qualitative examples in Fig. B3. For each of the four datasets (DOCCI, IIW, Flickr30k, MS COCO), we provide examples of instances both where TEVI

R@1

IIW (Garg et al., 2024) I→T T→I R@5 R@10 R@1 R@5

R@10

50.98 17.65

77.94 38.56

41.46 26.82

88.89 52.12

16.88 7.80

32.66 19.44

retrieves correctly and the baseline does not, and vice-versa. Note that some of the retrieved captions (e.g. by the baseline model for the first MS COCO example) appear correct but do not match the image as per the dataset, which shows the challenging

Table B7: Fine-grained retrieval performance on DOCCI (Onoe et al., 2024) and IIW (Garg et al., 2024) as compared to SmartCLIP. All the models are trained on CC12M dataset. We report R@1, R@5 and R@10 for both image-to-text and text-to-image retrieval.

Model CLIP ViT-B/16 SmartCLIP TEVI

R@1

DOCCI (Onoe et al., 2024) I→T T→I R@5 R@10 R@1 R@5

R@10

20.38 21.76 24.20

42.36 45.04 48.68

22.86 23.80 26.80

53.52 55.64 60.06

7.16 7.47 8.55

16.96 17.83 19.98

R@1

IIW (Garg et al., 2024) I→T T→I R@5 R@10 R@1 R@5

R@10

50.98 54.74 55.72

77.94 82.35 81.21

41.46 42.15 45.47

88.89 89.05 88.72

16.88 17.62 19.37

32.66 33.90 36.44

Table B8: Coarse-grained retrieval performance on MSCOCO (Lin et al., 2014) and Flickr30k (Plummer et al., 2015) as compared to SmartCLIP. All the models are trained on CC12M dataset. We report R@1, R@5 and R@10 for both image-to-text and text-to-image retrieval.

R@1

MS COCO (Lin et al., 2014) I→T T→I R@5 R@10 R@1 R@5

R@10

32.98 35.54 35.66

59.02 61.98 61.82

57.03 59.93 58.67

Model CLIP ViT-B/16 SmartCLIP TEVI

70.04 72.90 72.94

21.38 23.89 23.12

45.30 48.24 47.03

nature of the benchmarks.

Broader Impact

The use of vision-language models for multimodal tasks is widespread, from tasks like image retrieval to visual question answering to image generation. This makes it increasingly important that such models work reliably. Lack of proper semantic alignment can lead to biased outputs (Liang et al., 2022) or poor robustness to distribution shifts (Eslami and de Melo, 2025). Improved visual-textual alignment can enhance downstream performance of such models and help alleviate the above mentioned drawbacks. Our work is fundamental research on addressing vision-language alignment and deals with relatively small-scale models, and therefore has low direct risk. However, like many multimodal frameworks, our methods could be co-opted for malicious purposes if deployed without safeguards.

Licenses of Artifacts Used

We adhere to the licenses provided by the artifacts we use in our work. OpenCLIP and RoCOCO are provided under the MIT license; SharedCLIP and AlignCLIP are released under the CC-BY-NC-ND4.0 license; DOCCI, MS COCO, IIW are under the CC-BY-4.0 license; and SmartCLIP is released under the Apache 2.0 license. Flickr30k permits use for non-commercial research purposes. Our usage of all artifacts are in accordance with their allowed and intended use.

R@1

Flickr30k (Plummer et al., 2015) I→T T→I R@5 R@10 R@1 R@5

R@10

59.66 61.05 64.20

83.73 85.11 85.70

79.31 81.24 81.22

90.14 91.32 91.12

42.46 45.29 44.75

70.33 72.78 72.10

0.4

0.2

Attribute Specificity of ‘Scaling’ SAE Latents

0.8

Large

0.6

ROC AUC Value

N o

0.0

Fracture

0.8

0.6

0.4

0.2

ROC AUC Value

0.0

ROC AUC Value 0.8

No fracture

0.6

0.4

ROC AUC Value

N o

ROC AUC Value 0.8

0.6

0.4

0.2

0.0

ROC AUC Value

N ROC AUC Value 0.8

0.6

0.4

0.2

ROC AUC Value

o 0.0

Thickening 1.0

0.0

Thinning 1.0

Attribute Specificity of ‘Fracture’ SAE Latents 1.0

0.2

1.0

0.8

Small

0.6

0.4

0.2

0.0

0 1 2 3 4 5 6 7 8 9

ck T thin hi n ck in T e g N hi nin o nn g sw in e g N Sw llin o el g fr lin ac g Fr tu ac re tu La re r Smge a G ll ra Re y G d re e B n lu M Cy e ag an en Ye ta llo w

th i

0 1 2 3 4 5 6 7 8 9

ic k T thin hi n ck in T e g N hi nin o nn g sw in e g N Sw llin o el g fr lin ac g Fr tu ac re tu La re r Smge a G ll ra Re y G d re e B n lu M Cy e ag an en Ye ta llo w

N ROC AUC Value

ROC AUC Value

0.2

0 1 2 3 4 5 6 7 8 9

ic k T thin hi n ck in T e g N hi nin o nn g sw in e g N Sw llin o el g fr lin ac g Fr tu ac re tu La re r Smge a G ll ra Re y G d re e B n lu M Cy e ag an en Ye ta llo w

0 1 2 3 4 5 6 7 8 9

th ic k T thin hi n ck in T e g N hi nin o nn g sw in e g N Sw llin o el g fr lin ac g Fr tu ac re tu La re r Smge a G ll ra Re y G d re e B n lu M Cy e ag an en Ye ta llo w 0.4

0 1 2 3 4 5 6 7 8 9

ic k T thin hi n ck in T e g N hi nin o nn g sw in e g N Sw llin o el g fr lin ac g Fr tu ac re tu La re r Smge a G ll ra Re y G d re e B n lu M Cy e ag an en Ye ta llo w

N 0.6

0 1 2 3 4 5 6 7 8 9

th ic k T thin hi n ck in T e g N hi nin o nn g sw in e g N Sw llin o el g fr lin ac g Fr tu ac re tu La re r Smge a G ll ra Re y G d re e B n lu M Cy e ag an en Ye ta llo w

0 1 2 3 4 5 6 7 8 9

ic k T thin hi n ck in T e g N hi nin o nn g sw in e g N Sw llin o el g fr lin ac g Fr tu ac re tu La re r Smge a G ll ra Re y G d re e B n lu M Cy e ag an en Ye ta llo w

1.0

0 1 2 3 4 5 6 7 8 9

ic k T thin hi n ck in T e g N hi nin o nn g sw in e g N Sw llin o el g fr lin ac g Fr tu ac re tu La re r Smge a G ll ra Re y G d re e B n lu M Cy e ag an en Ye ta llo w

ROC AUC Value

No thickthinning

0 1 2 3 4 5 6 7 8 9

ck T thin hi n ck in T e g N hi nin o nn g sw in e g N Sw llin o el g fr lin ac g Fr tu ac re tu La re r Smge a G ll ra Re y G d re e B n lu M Cy e ag an en Ye ta llo w

0.0

th i

0 1 2 3 4 5 6 7 8 9

ic k T thin hi n ck in T e g N hi nin o nn g sw in e g N Sw llin o el g fr lin ac g Fr tu ac re tu La re r Smge a G ll ra Re y G d re e B n lu M Cy e ag an en Ye ta llo w

th 0.8

0 1 2 3 4 5 6 7 8 9

1.0

ROC AUC Value

1.0

ic k T thin hi n ck in T e g N hi nin o nn g sw in e g N Sw llin o el g fr lin ac g Fr tu ac re tu La re r Smge a G ll ra Re y G d re e B n lu M Cy e ag an en Ye ta llo w

N 1.0

ROC AUC Value 1.0

0 1 2 3 4 5 6 7 8 9

ic k T thin hi n ck in T e g N hi nin o nn g sw in e g N Sw llin o el g fr lin ac g Fr tu ac re tu La re r Smge a G ll ra Re y G d re e B n lu M Cy e ag an en Ye ta llo w

th 1.0

N 1.0

ROC AUC Value 1.0

0 1 2 3 4 5 6 7 8 9

ic k T thin hi n ck in T e g N hi nin o nn g sw in e g N Sw llin o el g fr lin ac g Fr tu ac re tu La re r Smge a G ll ra Re y G d re e B n lu M Cy e ag an en Ye ta llo w

Attribute Specificity of ‘Thickthinning’ SAE Latents Attribute Specificity of ‘Colors’ SAE Latents

0.8

Gray

0.6

0.4

0.2

0.0

0.8

Red

0.6

0.4

0.2

0.0

0.8

Green

0.6

0.4

0.2

0.0

0.8

Blue

0.6

0.4

0.2

0.0

0.8

Cyan

0.6

0.4

0.2

0.0

0.8

Magenta

0.6

0.4

0.2

0.0

0.8

Yellow

0.6

0.4

0.2

0.0

Fig. A3: Attribute specificity of CG-SAE latents for attribute categories ‘Thickthinning’, ‘Fracture’, ‘Scaling’, and ‘Color’. We plot the area under the receiver operating characteristic (ROC) curve (AUC) for CG-SAE latents corresponding to the attribute values of each attribute category. We find that the latents are highly attribute-specific, with the AUC being close to 1 for the attribute the latent is assigned to, and 0.5 for unrelated attributes. For results for other attribute categories, see Fig. 3 and Fig. A4.

Attribute Specificity of ‘Digit’ SAE Latents 0

1.0

ROC AUC Value

0.8 0.6 0.4 0.2

0.6 0.4 0.2

th o N

ic kt T hin hi n ck in T e g N hi nin o nn g sw in e g N Sw llin o el g fr lin a Fr ctu g ac re tu La re r Smge a G ll ra y R G ed re e B n lu M Cy e ag an e Ye nta llo w

0 1 2 3 4 5 6 7 8 9

0.0

ic kt T hin hi n ck in T e g N hi nin o nn g sw in e g N Sw llin o el g fr lin ac g Fr tu ac re tu La re r Smge a G ll ra y R G ed re e B n lu M Cy e ag an e Ye nta llo w

0.0

0.8

0 1 2 3 4 5 6 7 8 9

ROC AUC Value

1.0

ROC AUC Value

0.8 0.6 0.4 0.2

0.6 0.4 0.2

th o N

ic kt T hin hi n ck in T e g N hi nin o nn g sw in e g N Sw llin o el g fr lin a Fr ctu g ac re tu La re r Smge a G ll ra y R G ed re e B n lu M Cy e ag an e Ye nta llo w

0 1 2 3 4 5 6 7 8 9

0.0

ic kt T hin hi n ck in T e g N hi nin o nn g sw in e g N Sw llin o el g fr lin ac g Fr tu ac re tu La re r Smge a G ll ra y R G ed re e B n lu M Cy e ag an e Ye nta llo w

0.0

0.8

0 1 2 3 4 5 6 7 8 9

ROC AUC Value

1.0

ROC AUC Value

0.8 0.6 0.4 0.2

0.6 0.4 0.2

th o N

ic kt T hin hi n ck in T e g N hi nin o nn g sw in e g N Sw llin o el g fr lin a Fr ctu g ac re tu La re r Smge a G ll ra y R G ed re e B n lu M Cy e ag an e Ye nta llo w

0 1 2 3 4 5 6 7 8 9

0.0

ic kt T hin hi n ck in T e g N hi nin o nn g sw in e g N Sw llin o el g fr lin ac g Fr tu ac re tu La re r Smge a G ll ra y R G ed re e B n lu M Cy e ag an e Ye nta llo w

0.0

0.8

0 1 2 3 4 5 6 7 8 9

ROC AUC Value

1.0

ROC AUC Value

0.8 0.6 0.4 0.2

0.6 0.4 0.2

th o N

ic kt T hin hi n ck in T e g N hi nin o nn g sw in e g N Sw llin o el g fr lin a Fr ctu g ac re tu La re r Smge a G ll ra y R G ed re e B n lu M Cy e ag an e Ye nta llo w

0 1 2 3 4 5 6 7 8 9

0.0

ic kt T hin hi n ck in T e g N hi nin o nn g sw in e g N Sw llin o el g fr lin ac g Fr tu ac re tu La re r Smge a G ll ra y R G ed re e B n lu M Cy e ag an e Ye nta llo w

0.0

0.8

0 1 2 3 4 5 6 7 8 9

ROC AUC Value

1.0

ROC AUC Value

0.8 0.6 0.4 0.2

0.6 0.4 0.2

th o N

ic kt T hin hi n ck in T e g N hi nin o nn g sw in e g N Sw llin o el g fr lin a Fr ctu g ac re tu La re r Smge a G ll ra y R G ed re e B n lu M Cy e ag an e Ye nta llo w

0 1 2 3 4 5 6 7 8 9

0.0

ic kt T hin hi n ck in T e g N hi nin o nn g sw in e g N Sw llin o el g fr lin ac g Fr tu ac re tu La re r Smge a G ll ra y R G ed re e B n lu M Cy e ag an e Ye nta llo w

0.0

0.8

0 1 2 3 4 5 6 7 8 9

ROC AUC Value

1.0

Fig. A4: Attribute specificity of CG-SAE latents for attribute category ‘Digit’. We plot the area under the receiver operating characteristic (ROC) curve (AUC) for CG-SAE latents corresponding to each digit attribute. We find that the latents are fairly attribute-specific, with the AUC being close to 1 for the attribute the latent is assigned to, and close to 0.5 for unrelated attributes. For results for other attribute categories, see Fig. 3 and Fig. A3.

Image-Text Pairwise Cosine Similarity

400 200 0 0.1

0.2

0.3

0.4

0.5

150 100 50

500 400

80 Original TEVI

300 200 100

0 0.1

0.6

Image-Text Pairwise Cosine Similarity

600 Original TEVI

Frequency

600

Frequency

Image-Text Pairwise Cosine Similarity

200 Original TEVI

Frequency

Image-Text Pairwise Cosine Similarity

800

0.2

Cosine Similarity

0.3

0.4

0.5

0.6

0 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

0.7

Cosine Similarity

Original TEVI

60 40 20 0 0.1

0.2

Cosine Similarity

0.3

0.4

0.5

0.6

0.7

Cosine Similarity

Fig. B1: Cross-modal alignment with SharedCLIP on (1) MSCOCO (Lin et al., 2014), (2) Flickr30k (Plummer et al., 2015), (3) DOCCI (Onoe et al., 2024), (4) IIW (Garg et al., 2024). The alignment is measured by the cosine similarity between the positive image-text pairs, and the y-axis denotes the number of data points for each alignment score. We see the distribution after applying TEVI (blue) shifts to the right as compared to the baseline (orange), showing improved alignment.

Image-Text Pairwise Cosine Similarity

400 200 0 0.1

0.2

0.3

0.4

0.5

Cosine Similarity

0.6

150

Original TEVI

100 50 0 0.1

Image-Text Pairwise Cosine Similarity

600

Frequency

Original TEVI

Frequency

600

Image-Text Pairwise Cosine Similarity

200

500 400

80 Original TEVI

300 200 100

0.2

0.3

0.4

0.5

Cosine Similarity

0.6

0.7

0 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

Cosine Similarity

Frequency

Image-Text Pairwise Cosine Similarity

800

Original TEVI

40 20 0 0.1

0.2

0.3

0.4

0.5

0.6

0.7

Cosine Similarity

Fig. B2: Cross-modal alignment with AlignCLIP on (1) MSCOCO (Lin et al., 2014), (2) Flickr30k (Plummer et al., 2015), (3) DOCCI (Onoe et al., 2024), (4) IIW (Garg et al., 2024). The alignment is measured by the cosine similarity between the positive image-text pairs, and the y-axis denotes the number of data points for each alignment score. We see the distribution after applying TEVI (blue) shifts to the right as compared to the baseline (orange), showing improved alignment.

TEVI

The ramp of the utility trailer is black and has mesh over the thick black supports

A side view of a black Diamond T utility trailer placed on the side of a road next to a park with trees

Image

Only TEVI correct

Original

A yellow label with blue border is affixed to the side of the container with illegible writing

The stamp commemorates the 6th World Festival of Youth and Students, Moscow, 1957

These ornaments flank a vibrant red sign prominently displaying the whimsical message NO Humbugging written in an elegant silver cursive font, accompanied by decorative silver snowflakes scattered around the text

A close-up shot presents a round red wooden disk on which the words "BIG RED" are vertically stacked in the center in a large white all-caps serif font, with the scripted word "Deliciously" above those words in white and the word "Different!" below those words in white, all on a brown plank background

An outside overhead view of three sport balls

An outdoor top-down view of 2 white soccer balls sitting on a lawn of grass, The balls have a pentagon pattern all around, a blue and white design, some faded black marks, and the word "SELECT" at the center

Shadows cover the statue

The figure is seen mostly in silhouette due to the lighting, and few details can be made out beyond its pose

A side view of a white crayfish on top of a dark gray piece of a fossilized branch in an aquarium

The tail fin of another white-tipped re-shark is visible in the upper right corner between the middle and right large rocks

A vertically-oriented overhead image shows an orange and black Monarch butterfly next to a small, downy, white feather on green and tan grass

On the ground and behind the bird, dirt covers the ground, littered with pebbles and twigs

A vertical teal neon light is visible behind the cards

The blue light is surrounding the inner border of the square

Centrally featured is the iconic black Apple logo, partially overlaid with a colorful Google logo sticker

In the center of the quadrants is a map of the racetrack with colored circles to indicate the positions of the characters

Original

(b) IIW TEVI

Image

Original

TEVI

A young man playing out side with a disc.

A guy gets ready to throw a Frisbee during a game.

a building and some signs and street light and phone lines

a black and white photo of a street signs and buildings

Man and boy play with dog at sunset on a beach.

Young adults dressed casually walk around in a field.

Four people in a sandy area holding black trash bags.

A mother and her sons look at an information kiosk.

A little girl donates to the Salvation Army near a storefront with white, plastic containers.

A red car on street with utility vehicles in background.

Three similar cars are stopped at traffic lights.

A small dog jumping on a street.

A brown and white dog is running down a ramp.

some windows a fireplace books tables and a couch

A living room with windows, chairs, table, lamps, and a person sitting in a chair with a walker beside it.

A group of children sitting in chairs with monitors over them.

A large group of computer gamers are arranged in a row while playing games.

An orange bowl filled with lots of noodles and beef.

A bowl of Asian Cuisine with beef, noodles and broccoli.

Only Original correct

Only TEVI correct

A man at the beach holding a pail, standing next to a child.

Only Original correct

Only TEVI correct

TEVI

To the left of the black-and-white picture, there is a brown wall that is part of the cave and runs vertically

(a) DOCCI Image

Original

A close up view of a limestone cave wall that extends up to a dark opening in the top right corner of the image

Only Original correct

Only TEVI correct

Image

(d) MS COCO

Fig. B3: Additional qualitative examples. For each dataset, we show examples of both instances where TEVI retrieves correctly when the baseline does not, and vice-versa. All examples are sampled randomly.

Record · ID 266164 · SHA-256 70c6b01e99452d1d
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.