R EVISITING C OMPOSITIONALITY IN D UAL -E NCODER V ISION -L ANGUAGE M ODELS : T HE ROLE OF I NFERENCE
arXiv:2604.11496v1 [cs.CV] 13 Apr 2026
Imanol Miranda Ander Salaberria Eneko Agirre Gorka Azkune HiTZ Center – Ixa, University of the Basque Country (UPV/EHU) {imanol.miranda, ander.salaberria, e.agirre, gorka.azcune}@ehu.eus
A BSTRACT Dual-encoder Vision-Language Models (VLMs) such as CLIP are often characterized as bag-of-words systems due to their poor performance on compositional benchmarks. We argue that this limitation may stem less from deficient representations than from the standard inference protocol based on global cosine similarity. First, through controlled diagnostic experiments, we show that explicitly enforcing fine-grained region–segment alignment at inference dramatically improves compositional performance without updating pretrained encoders. We then introduce a lightweight transformer that learns such alignments directly from frozen patch and token embeddings. Comparing against full finetuning and prior end-to-end compositional training methods, we find that although these approaches improve in-domain retrieval, their gains do not consistently transfer under distribution shift. In contrast, learning localized alignment over frozen representations matches full fine-tuning on indomain retrieval while yielding substantial improvements on controlled out-of-domain compositional benchmarks. These results identify global embedding matching as a key bottleneck in dual-encoder VLMs and highlight the importance of alignment mechanisms for robust compositional generalization. Keywords Vision-Language Models · Compositional Reasoning · Inference Mechanisms
1
Introduction
Dual-encoder Vision-Language Models (VLMs), pioneered by CLIP Radford et al. [2021], have become foundational models for vision-language tasks such as image–text retrieval Cao et al. [2022], zero-shot image recognition Radford et al. [2021], and image–text scoring Hessel et al. [2021]. However, several studies report that these models struggle with compositional reasoning Yuksekgonul et al. [2022], Thrush et al. [2022], Hsieh et al. [2024], Miranda et al. [2024], often behaving like bag-of-words systems. For instance, they fail to reliably distinguish between “a black dog and a white cat” and “a black cat and a white dog”, leading to low performance on Vision-Language Compositionality (VLC) benchmarks. At the same time, recent analyses caution against drawing definitive conclusions about the compositional limitations of VLMs Diwan et al. [2022], Campbell et al. [2024]. We argue that these limitations may stem not from the learned representations themselves, but from the standard inference protocol used to evaluate them: global cosine similarity between pooled image and text embeddings. This global matching mechanism collapses multimodal information into a single vector comparison, potentially discarding region–token correspondences required for compositional binding. In contrast, compositional reasoning requires aligning textual elements describing objects, attributes, and relations with their corresponding image regions (Figure 1). To test this hypothesis, we first conduct a controlled diagnostic study in which we enforce structured region–segment matching at inference time while keeping pretrained VLM encoders frozen. This experiment isolates the role of alignment from representation learning. We evaluate models on a new controlled out-of-distribution dataset, B I SC O RC TRL, specifically designed to measure compositional reasoning under reduced spurious correlations. Across multiple VLMs, enforcing structured alignment dramatically improves compositional performance, demonstrating that finegrained region-segment alignments are beneficial for compositional reasoning. Building on this diagnostic evidence, we then investigate whether such fine-grained alignment can be learned directly from pretrained representations. We introduce a lightweight transformer operating over patch and token embeddings
Revisiting Compositionality in Dual-Encoder Vision-Language Models: The Role of Inference
Standard Inference Approach cos
Visual Encoder
Alignment reduced to one operation
Fine-grained alignments
Text Encoder
A white bird with a brown head flys over a large body of water.
Vision-Language Compositional Reasoning Figure 1: Vision-language compositional reasoning requires fine-grained alignment between textual segments describing objects, attributes, and relations and their corresponding image regions (bottom). In contrast, standard dual-encoder VLM inference relies on global embedding matching, reducing image–text similarity to a single cosine operation between pooled representations (top). This discrepancy highlights a structural mismatch between the localized alignment required for compositional binding and the global matching mechanism typically used at inference. extracted from frozen encoders. To disentangle alignment learning from representation updates, we compare against two controlled alternatives: (i) applying the same transformer to global embeddings only, and (ii) fully fine-tuning the original VLMs while retaining cosine-based inference. We further compare against prior end-to-end compositional training methods that modify the contrastive objective to encourage fine-grained reasoning. Models are evaluated on in-domain compositional benchmarks, including S UGAR C REPE Hsieh et al. [2024] and B I VLC Miranda et al. [2024], as well as on B I SC O R-C TRL for out-of-distribution generalization. Our results show that: 1. Enforcing structured region–segment alignment at inference substantially improves compositional reasoning without updating pretrained encoders. 2. Learning fine-grained alignment over frozen patch and token representations matches full fine-tuning on in-domain retrieval and yields large gains in out-of-domain compositional generalization, outperforming cosine-based inference. 3. In contrast, prior end-to-end compositional training methods and full fine-tuning improve in-domain benchmarks but do not consistently transfer under distribution shift. 4. Increasing modeling capacity alone—e.g., applying a transformer to global embeddings—does not improve compositional performance, highlighting global embedding matching as a key bottleneck. Together, these findings suggest that compositional failures in dual-encoder VLMs stem less from missing representational capacity and more from the global embedding matching mechanism used at inference. Even models trained with compositional objectives show limited robustness under distribution shift when relying on global embedding similarity. By contrast, enabling localized alignment over pretrained representations leads to substantially stronger generalization on out-of-distribution benchmarks. These results highlight the importance of revisiting inference protocols in dual-encoder architectures and motivate evaluation settings that better reflect the structured, region–token information already encoded in modern VLMs. Code1 and datasets2 will be publicly released. 1 2
https://github.com/IMirandaM/revisiting-vl-compositional-inference https://huggingface.co/datasets/imirandam/BiSCoR
2
Revisiting Compositionality in Dual-Encoder Vision-Language Models: The Role of Inference
2
Related work
Dual-encoder Vision-Language Models consist of a vision encoder and a text encoder trained jointly to align visual and textual representations in a shared embedding space. The paradigm was popularized by CLIP Radford et al. [2021], which enabled large-scale contrastive pretraining and zero-shot transfer. Subsequent models, including SigLIP and SigLIP 2 Zhai et al. [2023], Tschannen et al. [2025] and Perception Encoder Bolya et al. [2025], refine the training objective, scale data and model size, and incorporate additional learning signals, consistently improving standard benchmarks such as image recognition and image–text retrieval. Despite these differences, dual-encoder VLMs share a common inference mechanism: images and texts are independently encoded into global embeddings, and similarity is computed via cosine matching between pooled representations. This design underlies their success in retrieval, scoring Hessel et al. [2021], and as visual backbones for multimodal large language models such as LLaVA Liu et al. [2023], PaliGemma Beyer et al. [2024], and Qwen3-VL Bai et al. [2025]. In this work, we examine whether this shared global matching paradigm is sufficient for assessing and exploiting compositional reasoning in dual-encoder VLMs. Vision-Language Compositionality (VLC) refers to a model’s ability to distinguish image-text pairs containing the same elements arranged in different configurations (e.g., “a red sphere and a blue cylinder” vs. “a blue sphere and a red cylinder”), requiring correct cross-modal binding of objects, attributes, and relations. VLC is commonly evaluated through retrieval benchmarks that construct hard negatives by minimally perturbing captions or images Ma et al. [2023], Hsieh et al. [2024], Thrush et al. [2022], Ray et al. [2024]. Models must rank correct pairs above compositionally altered alternatives, typically using global cosine similarity between pooled embeddings. However, this setup allows models to exploit linguistic or dataset biases Lin et al. [2024a], Miranda et al. [2024] and entangles compositional reasoning with the limitations of global matching. To address these issues, recent work advocates bidirectional retrieval and controlled dataset design Miranda et al. [2024]. We follow these practices and additionally introduce a controlled out-of-distribution benchmark to better isolate compositional reasoning from spurious correlations. End-to-end training for VL compositionality. The limited compositional performance of dual-encoder VLMs has motivated approaches that modify the training objective to encourage stronger cross-modal binding. One line of work introduces fine-grained alignment signals during pretraining, as in FILIP Yao et al. [2021], X-VLM Zeng et al. [2022], PyramidCLIP Gao et al. [2022], FSC-CLIP Oh et al. [2024], and FineCLIP Jing et al. [2024], which promote token- or region-level interactions alongside global contrastive learning. Another direction emphasizes hard-negative mining to reduce shortcut learning, including NegCLIP Yuksekgonul et al. [2022], GNM Sahin et al. [2024], and TripletCLIP Patel et al. [2024]. Despite differing training strategies, these methods retain the standard inference paradigm: image and text representations are pooled into global embeddings and compared via cosine similarity. In contrast, we investigate whether restructuring inference—while keeping pretrained encoders frozen—can better exploit fine-grained information already encoded in dual-encoder representations.
3
Diagnosing the Inference Bottleneck in VLMs
Standard evaluations of dual-encoder VLMs rely on global cosine similarity between pooled image and text embeddings, without explicitly modeling interactions between image regions and textual components Hsieh et al. [2024], Miranda et al. [2024], Yuksekgonul et al. [2022]. Whether this global matching mechanism adequately captures the fine-grained correspondences required for compositional reasoning remains unclear. To investigate this question, we conduct a controlled diagnostic study that isolates the role of inference. We introduce B I SC O R-C TRL, a benchmark designed to evaluate compositional robustness under controlled conditions, and apply a structured matching protocol to frozen VLM encoders at inference time. This setup allows us to assess whether enforcing region-segment alignment, without modifying the underlying representations, improves compositional performance. 3.1
B I SC O R-C TRL: A Controlled Compositional Benchmark
Evaluating compositional reasoning in VLMs requires benchmarks that minimize confounding factors unrelated to visual grounding. Prior work shows that bidirectional retrieval settings provide a more balanced assessment of compositional abilities by mitigating language-based biases, where models may exploit caption plausibility or prior likelihood instead 3
Revisiting Compositionality in Dual-Encoder Vision-Language Models: The Role of Inference
A purple cylinder and a brown sphere
A small cube and a large sphere
A metal cylinder and a rubber sphere
Two cylinders and six cubes
Color
Size
Material
Quantity
A brown cylinder and a purple sphere
A large cube and a small sphere
A rubber cylinder and a metal sphere
Six cylinders and two cubes
Figure 2: Examples from the B I SC O R-C TRL dataset. From left to right: instances from the C OLOR, S IZE, M ATERIAL, and Q UANTITY categories (the last containing 8 objects). Each instance consists of two image–caption pairs: a correct pair (top image and caption) and a hard negative pair (bottom image and caption). of image evidence Lin et al. [2024b,a], Miranda et al. [2024], Udandarao et al. [2025]. Accordingly, B I SC O R-C TRL is constructed as a symmetric bidirectional retrieval benchmark, requiring joint discrimination of images and captions under controlled conditions. B I SC O R-C TRL focuses exclusively on S WAP cases, in which candidate captions contain the same objects, attributes, and relations but differ in their bindings (e.g., swapped attributes or object roles). Such cases are particularly challenging because success cannot rely on detecting individual elements in isolation Miranda et al. [2024], Hsieh et al. [2024], Thrush et al. [2022]. Instead, models must correctly bind attributes and relations to their corresponding objects. By restricting the dataset to this category, we focus on the core binding problem while eliminating easier compositional variations. The benchmark is designed as an out-of-distribution (OOD) evaluation. It is built from CLEVR-based synthetic 3D scenes with controlled captions Johnson et al. [2017], contrasting with the natural images and free-form text used in large-scale VLM pretraining Radford et al. [2021], Tschannen et al. [2025], Bolya et al. [2025]. The use of synthetic scenes enables precise control over object attributes and bindings, ensuring that positive and negative pairs differ only in the targeted compositional factor. Such control is essential for isolating whether models capture fine-grained compositional structure rather than dataset-specific correlations. Dataset Construction: B I SC O R-C TRL contains four compositional categories: C OLOR, S IZE, M ATERIAL, and Q UANTITY, with 1,000 independent examples per category. Each example consists of two images and two captions forming a symmetric bidirectional retrieval pair: each caption correctly describes only one image, while the alternative caption corresponds to a minimally modified version of the same scene. The dataset includes development and test splits of equal size, constructed from disjoint CLEVR scene sets (training split for development and validation split for test) to prevent scene-level overlap. Examples are generated automatically as follows: 1. Scene Sampling. Sample a base CLEVR scene. 2. Object Selection. For color, size, and material, select two objects; for quantity, select multiple objects sharing attributes to enable controlled counting variations. 3. Caption Generation. Generate structured captions from predefined templates using scene annotations. 4. Hard Negative Caption Creation (S WAP). Create a compositional negative by swapping object identities or attribute bindings while preserving the same elements. 5. Hard Negative Image Generation. Modify the original scene via controlled re-rendering, altering only the swapped attribute or binding. The resulting image pair preserves global layout and rendering conditions, differing only in the targeted compositional factor. This construction yields tightly controlled compositional contrasts in which captions share identical elements and images share the same global structure. Consequently, success on B I SC O R-C TRL cannot rely on element detection or caption plausibility, but requires correctly binding object–attribute relationships across modalities. Example pairs are shown in Figure 2 (see Appendix A for more). 3.2
Enforcing Fine-Grained Alignment at Inference
To assess whether explicit region–segment correspondences improve compositional reasoning, we design a controlled inference protocol that enforces such alignments while keeping pretrained VLM encoders frozen. The objective is 4
Revisiting Compositionality in Dual-Encoder Vision-Language Models: The Role of Inference
diagnostic rather than methodological: by modifying only the inference procedure and leaving model parameters unchanged, we isolate the role of alignment independently of representation learning. Unlike standard evaluation based on global embedding similarity, our protocol decomposes captions into structured segments and matches them to image regions before aggregating similarities into a final score. We refer to this procedure as Structure-Guided Inference (SGI), described below. [Step 1] Generate image crops: Given an image I, we generate a set of crops {ci }N i=0 using predefined scales, aspect ratios, and strides. These crops approximate candidate object-level regions without requiring supervision. [Step 2] Generate text segments: Given a caption T , we deterministically decompose it into object–attribute phrases plus the full caption using predefined parsing rules, obtaining segments {tj }M j=0 . For example, for T = “a black cat and a white dog”, the segments are “black cat”, “white dog”, and the full caption. This decomposition is fixed across experiments. [Step 3] Match crops and text segments: Let Venc and Tenc denote the frozen visual and textual encoders. For each crop ci and segment tj , we compute embeddings vi = Venc (ci ) and lj = Tenc (tj ) and their cosine similarity. This yields a similarity matrix of size N × M . For each segment tj , we select the crop with maximum similarity, producing one matched pair per segment mj . [Step 4] Aggregate alignment score: The final image–text score is computed as the average similarity over the selected matches: 1 X Sim(I, T ) = sim(mj ) (1) M j 3.3
Experimental Setup
Models: We evaluate representative dual-encoder VLMs: CLIP Radford et al. [2021], SigLIP 2 Tschannen et al. [2025], and Perception Encoder (PE) Bolya et al. [2025], covering early contrastive models, modern multi-objective training, and recent scalable architectures. For CLIP we use ViT-B/32 (2242 input); for SigLIP 2, ViT-B/32 and giantopt-ViT/16 (2562 ); for PE, ViT-B/16 (2242 ) and ViT-G/14 (4482 ). All models are evaluated using both standard global embedding similarity and the proposed Structure-Guided Inference (SGI). SGI hyperparameters (crop configuration and segment granularity) are selected on the B I SC O R-C TRL development split and fixed thereafter (see Appendix B for details). Text segments are derived from B I SC O R-C TRL scene annotations to ensure controlled decomposition. To verify robustness, we also evaluate automatically generated segments using SpaCy Honnibal et al. [2020] (see AppendixB.5 for details), obtaining nearly identical results. Throughout the experiments, encoders remain frozen and no additional training is performed. Benchmark and performance metrics: For this diagnostic study, we use B I SC O R-C TRL as the primary benchmark. Performance is measured with the group score Thrush et al. [2022]: a perfect group score requires both images to be correctly matched to their corresponding captions, and both captions to be correctly matched to their images (formal definitions are in Appendix C). This metric discourages solutions based solely on linguistic priors and provides a balanced assessment of compositional reasoning. 3.4
Diagnostic Results and Analysis
Table 1 reports the results on B I SC O R-C TRL. Using standard global cosine similarity, all models obtain extremely low group scores (1–9 points on average), confirming that global embedding matching fails to capture compositional structure. Enforcing fine-grained region–segment alignment at inference through SGI substantially improves performance, raising average group scores to 25–56 points. Gains are consistent across compositional categories, with color and material swaps benefiting most, while size and quantity remain more challenging. Importantly, SGI operates on frozen encoders and introduces no additional training, modifying only the inference procedure. Similar results obtained with automatically generated text segments (see Appendix B.5) indicate that improvements stem from better exploitation of pretrained representations rather than privileged annotation. Overall, these experiments indicate that explicitly enforcing fine-grained region–segment alignment at inference substantially improves compositional performance compared to standard global cosine similarity. However, because SGI relies on externally guided structural decomposition, it does not establish whether such alignment can emerge 5
Revisiting Compositionality in Dual-Encoder Vision-Language Models: The Role of Inference
Table 1: Results for the diagnostic experiments on the B I SC O R-C TRL dataset. We provide the average group score obtained by each model using global embedding similarity and SGI. We also provide the group score for all the categories of the dataset. C OLOR
S IZE
Categories M ATERIAL
Q UANTITY
1.4 24.9
1.5 76.5
0.2 4.7
0.7 15.7
3.3 2.5
PE +SGI
8.5 44.7
5.9 91.3
10.9 16.9
7.7 55.5
9.4 15.3
PE-G +SGI
3.8 52.9
5.8 97.4
1.0 6.7
2.6 92.1
5.8 15.5
SigLIP 2 +SGI
4.5 47.6
3.7 94.9
9.6 7.1
1.5 68.3
3.1 20.2
SigLIP 2-G +SGI
4.8 56.4
6.0 96.2
6.6 18.1
2.4 91.7
4.4 19.7
Model
Average
CLIP +SGI
directly from pretrained representations. The results instead suggest that frozen VLM embeddings contain information that can support compositional alignment when appropriately structured. We therefore ask whether this alignment can be learned directly from patch- and token-level embeddings, without imposing external structure at inference time.
4
Learning Fine-Grained Alignment from Frozen VLMs
4.1
Motivation and Overview
Building on the diagnostic findings of Section 3, we investigate whether fine-grained alignment can be learned directly from pretrained VLM representations. Our hypothesis is that patch-level visual embeddings and token-level textual embeddings already encode localized information relevant for compositional reasoning, but lack an explicit mechanism to align these elements in a structured manner. To this end, we introduce a lightweight transformer that operates on frozen patch and token embeddings, learning cross-modal correspondences while keeping the pretrained VLM intact. This design isolates the role of alignment learning from representation learning. Unlike full fine-tuning, which entangles both effects, our approach explicitly tests whether compositional improvements can be obtained by learning to exploit existing representations. If successful, this would provide further evidence that compositional reasoning in VLMs is primarily limited by alignment rather than encoding. 4.2
Alignment Transformer Architecture
To test whether compositional improvements can be obtained by learning fine-grained correspondences, we introduce a lightweight alignment module operating on top of frozen VLM encoders. The visual and textual backbones remain unchanged throughout training. Frozen Patch and Token Embeddings: Given an image I and a caption T , we extract patch embeddings {vi }N i=1 from the visual encoder and token embeddings {ti }M i=1 from the text encoder. We discard the projection heads used for standard embedding similarity inference. All encoder parameters are frozen. Cross-Modal Alignment Transformer: We concatenate the visual and textual sequences and feed them into a lightweight transformer encoder. We learn new image and text projections on top of those embeddings. We also add learnable positional embeddings to all inputs, an additional embedding to separate visual and textual representations, and a [CLS] token to the transformer encoder to learn the final matching score between the image and the text. The transformer allows: (i) cross-modal attention between patches and tokens, (ii) contextual refinement of region-level and token-level features, (iii) implicit learning of fine-grained correspondences. Importantly, the number of parameters in this module is small compared to the frozen backbone, ensuring that improvements can be attributed to alignment learning rather than large-scale representation updates. 6
Revisiting Compositionality in Dual-Encoder Vision-Language Models: The Role of Inference
Training Objective: The alignment transformer is trained using a contrastive retrieval objective. Given a batch of image–text pairs, the model is optimized to assign higher scores to matching pairs than to mismatched ones. Crucially, only the alignment transformer parameters are updated; the visual and textual encoders remain frozen. Design Rationale: This architecture isolates the effect of alignment learning: (i) if improvements arise from full fine-tuning, compositional gains should only appear when updating the encoders, and (ii) if pretrained representations already encode compositional cues, learning to align patch- and token-level features should suffice. By keeping the backbone fixed and introducing only a lightweight cross-modal module, we directly test whether fine-grained alignment is the missing ingredient for compositional reasoning in VLMs. 4.3
Experimental Setup
Models: We conduct experiments using CLIP Radford et al. [2021] and Perception Encoder (PE) Bolya et al. [2025]. These models were selected to cover both a widely studied contrastive VLM and a more recent large-scale alternative. For both CLIP and PE, we use the same pretrained checkpoints as in Section 3, discarding PE-G. In all alignment-learning experiments, the visual and textual encoders remain frozen. Training Data: To study how data properties affect alignment learning, we consider two training datasets: (i) COCO Lin et al. [2014]: a standard image–caption dataset containing clean positive pairs but no explicit hard negatives, and (ii) TROHN-Img Miranda et al. [2024]: a dataset derived from COCO that introduces automatically generated hard negative captions and images, resulting in a substantially noisier training distribution. This setup allows us to investigate whether exposure to hard negatives facilitates the learning of fine-grained correspondences, or whether alignment learning primarily depends on the structure of pretrained representations rather than negative sampling difficulty. Evaluation Data: We evaluate both in-domain and out-of-domain compositional reasoning. In-domain benchmarks: (i) B I VLC Miranda et al. [2024]: a bidirectional retrieval benchmark containing R EPLACE, S WAP, and A DD compositional categories, and (ii) S UGAR C REPE Hsieh et al. [2024]: an image-to-text retrieval benchmark with the same three compositional transformations. These datasets test whether alignment learning improves compositional reasoning under training-distribution conditions. Note that both datasets are derived from COCO and thus are in-domain relative to our training datasets. Out-of-domain benchmark: B I SC O R-C TRL, a controlled diagnostic benchmark designed to isolate compositional reasoning under distribution shifts (Section 3.1). Evaluation on B I SC O R-C TRL assesses whether learned alignments generalize beyond the training data. Ablation Design: To isolate the role of alignment learning, we compare the proposed lightweight alignment transformer with patch-token embeddings with two other training settings: (i) Full fine-tuning: the original dual-encoder VLM is fine-tuned end-to-end (visual and textual encoders) on COCO or TROHN-Img using a standard contrastive retrieval objective, without any additional alignment transformer. This setting entangles representation learning and alignment implicitly through encoder updates. (ii) Global-alignment transformer: the same lightweight transformer is trained on top of frozen global image and text embeddings only. This ablation controls for the additional modeling capacity of the transformer while removing access to patch- and token-level representations. This controlled comparison enables us to answer two key questions: (i) Are compositional gains primarily driven by updating representations? (ii) Or can they be achieved by learning to align existing fine-grained features? Implementation Details and Hyperparameter Selection: For the lightweight alignment transformer, we select architectural hyperparameters using development splits of the training datasets. In particular, we explore the number of transformer layers under each training configuration (COCO and TROHN-Img, global vs. fine-grained representations). For CLIP, we perform an exhaustive search over depths ranging from 1 to 4 layers (less than 10% of the total parameters). The optimal number of layers is 4 for all the variants (see full results in Appendix D.2). This configuration contains 13.3M parameters, corresponding to only 8.8% of the parameters of the original CLIP backbone, highlighting the lightweight nature of the alignment module. For PE, to reduce computational cost, we evaluate only the configurations that performed best for CLIP (4 layers). This controlled strategy allows us to assess whether alignment learning generalizes across backbones without repeating the full hyperparameter search. All hyperparameters are fixed before final evaluation on the test sets (see Appendix D.1 and E). 4.4
Results and Discussion
Table 2 presents the main results (results per compositional category in Appendix F). We analyze in-domain retrieval performance first, followed by out-of-domain compositional generalization. 7
Revisiting Compositionality in Dual-Encoder Vision-Language Models: The Role of Inference
Table 2: Main results comparing full fine-tuning (FT), a transformer over frozen global embeddings (TFGlobal ), and a transformer over frozen patch/token embeddings (TFLocal ). Results are reported as accuracy on S UGAR C REPE and group score on bidirectional retrieval benchmarks (B I VLC, B I SC O R-C TRL); S WAP performance for S UGAR C REPE and B I VLC is shown in brackets. While FT and TFLocal improve in-domain performance, TFLocal yields markedly stronger gains under distribution shift. Backbone
Method
Training data
Base
Pretrained
73.0(63.3)
36.8(13.7)
1.4
FT
COCO TROHN-Img
81.0(68.9) 85.5(72.0)
47.5(20.9) 57.5(27.9)
1.4 1.9
TFGlobal
COCO TROHN-Img
73.2(63.5) 82.2(68.8)
38.4(11.4) 49.0(15.9)
1.2 1.2
TFLocal
COCO TROHN-Img
80.9(76.3) 86.3(77.1)
45.7(24.0) 61.3(39.0)
15.1 13.2
Base
Pretrained
84.4(76.8)
41.5(13.4)
8.5
FT
COCO TROHN-Img
85.7(76.1) 90.1(80.3)
56.3(31.8) 68.2(43.7)
8.3 9.8
TFGlobal
COCO TROHN-Img
78.0(68.0) 86.9(74.9)
44.6(15.6) 58.3(24.5)
1.9 1.2
TFLocal
COCO TROHN-Img
84.4(80.6) 89.2(80.8)
53.3(33.7) 67.1(42.9)
30.0 24.0
CLIP
PE
In-domain S UGAR C REPE B I VLC
Out-of-domain B I SC O R-C TRL
In-Domain Retrieval: Fine-tuning substantially improves performance on both S UGAR C REPE and B I VLC. For instance, CLIP fine-tuned on TROHN-Img increases from 73.0 to 85.5 on S UGAR C REPE and from 36.8 to 57.5 on B I VLC. Similar trends are observed for PE. These results confirm that standard fine-tuning is effective for improving retrieval under the training distribution. The lightweight alignment transformer operating on global representations (TFGlobal ) yields modest gains in some in-domain settings, indicating that additional modeling capacity alone is insufficient. When operating on patch- and token-level representations (TFLocal ), the alignment transformer matches or exceeds full fine-tuning performance. Notably, CLIP+TFLocal trained on TROHN-Img achieves the strongest in-domain results overall (86.3 on S UGAR C REPE and 61.3 on B I VLC), despite keeping the backbone frozen. Since our hypothesis concerns compositional binding, we additionally analyze the S WAP subsets of S UGAR C REPE and B I VLC (Table 2, brackets), which isolate minimal attribute–object reversals and reduce reliance on linguistic shortcuts. In these subsets, the relative performance gaps between TFLocal , fine-tuning, and TFGlobal become even more pronounced across both backbones, with TFLocal consistently achieving the strongest results. This amplification on the most binding-sensitive cases reinforces that localized alignment particularly benefits examples requiring precise cross-modal correspondence. Out-of-Domain Compositional Generalization: A strikingly different picture emerges on B I SC O R-C TRL. Full fine-tuning provides negligible improvements over the frozen baseline (e.g., CLIP: 1.4 → 1.9; PE: 8.5 → 9.8). Similarly, the global transformer fails to improve compositional generalization and in some cases reduces it. In contrast, the finegrained alignment transformer yields substantial gains. CLIP+TFLocal improves from 1.4 to 15.1 (COCO training), while PE+TFLocal improves from 8.5 to 30.0 — nearly a fourfold increase over the pretrained baseline. These improvements are obtained without updating the visual or textual encoders. This result provides strong evidence that compositional reasoning benefits primarily from learning fine-grained alignments rather than from updating global representations. Alignment vs. Representation Learning: The controlled comparison between full fine-tuning and frozen-backbone alignment learning allows us to disentangle representation learning from alignment learning. While fine-tuning improves in-domain retrieval, it fails to enhance compositional generalization. Conversely, learning to align patch- and token-level representations significantly improves out-of-domain compositional performance. These findings suggest that pretrained VLMs already encode rich localized information, but standard global objectives fail to exploit it. Alignment learning is sufficient to unlock this latent compositional capability. 8
Revisiting Compositionality in Dual-Encoder Vision-Language Models: The Role of Inference
Table 3: Comparison between end-to-end compositional training methods and alignment learning. We report accuracy on S UGAR C REPE and group score on B I VLC and B I SC O R-C TRL; S WAP results for S UGAR C REPE and B I VLC are shown in brackets. Training data is indicated for each method († denotes dataset extensions with hard negatives or similarity-based sampling). While several end-to-end approaches improve in-domain benchmarks, gains do not consistently transfer to B I SC O R-C TRL, whereas alignment learning over frozen representations yields stronger improvements under distribution shift. Model
Training data
S UGAR C REPE
B I VLC
B I SC O R-C TRL
CLIP
Pretrained
73.0(63.3)
36.8(13.7)
1.4
NegCLIP TripletCLIP FSC-CLIP X-VLM FineCLIP
COCO† CC3M/12M COCO† COCO+Others COCO†
83.6(76.6) 82.6(71.9) 85.1(77.6) 81.9(67.4) 80.6(68.2)
44.9(18.7) 35.2(9.8) 46.5(19.2) 40.9(12.3) 39.4(8.6)
1.8 1.2 1.2 1.7 1.4
CLIP+TFLocal
COCO TROHN-Img
80.9(76.3) 86.3(77.1)
45.7(24.0) 61.3(39.0)
15.1 13.2
Effect of Hard Negatives: Training with TROHN-Img consistently improves in-domain retrieval across all methods. However, for fine-grained alignment learning, COCO-trained models often achieve stronger out-of-domain compositional performance (e.g., PE+TFLocal : 30.0 vs 24.0). This suggests that excessive noise or automatically generated hard negatives may bias the model toward shortcut strategies, reducing robustness under distribution shift. Summary: Overall, the results demonstrate that: (i) Fine-tuning improves in-domain retrieval but not compositional generalization; (ii) Additional modeling capacity alone is insufficient; (iii) Learning fine-grained alignment over frozen representations dramatically enhances compositional reasoning. Together with the diagnostic findings in Section 3, these results support our central claim: compositional failures in VLMs stem primarily from insufficient alignment mechanisms rather than missing representational capacity. 4.5
Comparison with End-to-End Compositional Training
Recent work has sought to improve compositional reasoning by modifying CLIP-style training objectives, either through fine-grained alignment supervision (e.g., FSC-CLIP Oh et al. [2024], X-VLM Zeng et al. [2022]) or by introducing hard negatives (e.g., NegCLIP Yuksekgonul et al. [2022], TripletCLIP Patel et al. [2024]). These methods alter the training signal but retain standard global cosine similarity at inference. Table 3 compares these approaches with our alignment-learning model built on frozen CLIP representations (detailed results per category in Appendix F), explicitly indicating the training data used by each method. While several end-toend approaches improve performance on natural-image compositional benchmarks such as S UGAR C REPE and B I VLC, gains on the S WAP subsets—shown in brackets—remain limited, particularly on B I VLC. Their improvements also do not transfer to the controlled OOD benchmark B I SC O R-C TRL. Despite being trained on additional compositional objectives or extended datasets (e.g., COCO variants with hard negatives or CC3M/12M), their performance on B I SC O R-C TRL remains close to that of the original CLIP. In contrast, CLIP+TFLocal yields a substantial improvement on B I SC O R-C TRL (from 1.4 to 15.1) without modifying the pretrained encoders. It also achieves stronger performance on the S WAP subsets, particularly on B I VLC, reflecting improved binding under challenging perturbations. These gains stem from learning to align patch- and token-level embeddings over frozen representations. Taken together with Section 4.4, these results suggest that improved compositional generalization does not necessarily require altering large-scale representation learning. Rather, the way multimodal similarity is computed plays a critical role. Even models trained with compositional objectives show limited gains if inference relies solely on global embedding similarity, whereas learning localized alignment mechanisms leads to markedly stronger improvements under distribution shift. Notably, the magnitude of these gains varies across backbones: alignment learning over Perception Encoder yields substantially higher compositional performance than over CLIP, indicating that representational quality and training scale still influence the effectiveness of structured inference. While our findings highlight the central role of alignment mechanisms, they do not preclude that improved representation learning objectives may further enhance compositional robustness when combined with structured inference. 9
Revisiting Compositionality in Dual-Encoder Vision-Language Models: The Role of Inference
5
Conclusion
In this work, we revisited the compositional limitations of dual-encoder VLMs. Rather than attributing these failures solely to deficient representations, we provided evidence that they are strongly influenced by the global cosine similarity typically used at inference time. Through controlled diagnostic experiments, we demonstrated that enforcing structured region–segment alignment—while keeping pretrained encoders frozen—substantially improves compositional performance. Moreover, learning fine-grained alignment from frozen patch and token embeddings outperforms both the pretrained backbone and its fully fine-tuned counterpart, which rely on global cosine similarity at inference, particularly under distribution shift. In contrast, increasing model capacity alone by applying a transformer to global embeddings yields limited gains. These findings suggest that dual-encoder VLMs encode richer compositional information than commonly assumed, but standard global matching underutilizes it. We hope this work encourages renewed attention to inference protocols and evaluation practices when assessing compositional reasoning in VLMs.
Acknowledgements This work is partially supported by Ministry of Science, Innovation, and Universities of the Spanish Government MCIN/AEI/10.13039/501100011033 by means of the projects: (i) MOLVI (PID2024-157855OB-C32) and by FEDER, EU; (ii) HumanAIze (AIA2025-163322-C61). The CHIST-ERA grant (Project Geo-R2LLM, CHIST-ERA-23-MultiGIS04) funded by the Ministry of Science, Innovation, and Universities of the Spanish Government (PCI2025-163286), the Basque Government (IXA excellence research group IT1570-22 and IKER-GAITU project), and the European Union under Horizon Europe (Project LUMINOUS, grant number 101135724).
References Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021. Min Cao, Shiping Li, Juntao Li, Liqiang Nie, and Min Zhang. Image-text retrieval: A survey on recent research and development. arXiv preprint arXiv:2203.14713, 2022. Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference-free evaluation metric for image captioning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7514– 7528, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi:10.18653/v1/2021.emnlp-main.595. URL https://aclanthology.org/2021.emnlp-main.595/. Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In The Eleventh International Conference on Learning Representations, 2022. Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5238–5248, 2022. Cheng-Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality. Advances in Neural Information Processing Systems, 36, 2024. Imanol Miranda, Ander Salaberria, Eneko Agirre, and Gorka Azkune. Bivlc: Extending vision-language compositionality evaluation with text-to-image retrieval. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 101880–101904. Curran Associates, Inc., 2024. Anuj Diwan, Layne Berry, Eunsol Choi, David Harwath, and Kyle Mahowald. Why is winoground hard? investigating failures in visuolinguistic compositionality. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2236–2250, 2022. Declan Campbell, Sunayana Rane, Tyler Giallanza, Camillo Nicolò De Sabbata, Kia Ghods, Amogh Joshi, Alexander Ku, Steven Frankland, Tom Griffiths, Jonathan D Cohen, et al. Understanding the limits of vision language models through the lens of the binding problem. Advances in Neural Information Processing Systems, 37:113436–113460, 2024. 10
Revisiting Compositionality in Dual-Encoder Vision-Language Models: The Role of Inference
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023. Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786, 2025. Daniel Bolya, Po-Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Rasheed, et al. Perception encoder: The best visual embeddings are not at the output of the network. arXiv preprint arXiv:2504.13181, 2025. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916, 2023. Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726, 2024. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631, 2025. Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna. Crepe: Can vision-language foundation models reason compositionally? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10910–10921, 2023. Arijit Ray, Filip Radenovic, Abhimanyu Dubey, Bryan Plummer, Ranjay Krishna, and Kate Saenko. Cola: A benchmark for compositional text-to-image retrieval. Advances in Neural Information Processing Systems, 36, 2024. Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. arXiv preprint arXiv:2404.01291, 2024a. Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. FILIP: fine-grained interactive language-image pre-training. CoRR, abs/2111.07783, 2021. URL https://arxiv.org/abs/2111.07783. Yan Zeng, Xinsong Zhang, and Hang Li. Multi-grained vision language pre-training: Aligning texts with visual concepts. In International Conference on Machine Learning, pages 25994–26009. PMLR, 2022. Yuting Gao, Jinfeng Liu, Zihan Xu, Jun Zhang, Ke Li, Rongrong Ji, and Chunhua Shen. Pyramidclip: Hierarchical feature alignment for vision-language model pretraining, 2022. URL https://arxiv.org/abs/2204.14095. Youngtaek Oh, Jae Won Cho, Dong-Jin Kim, In So Kweon, and Junmo Kim. Preserving multi-modal capabilities of pre-trained vlms for improving vision-linguistic compositionality. arXiv preprint arXiv:2410.05210, 2024. Dong Jing, Xiaolong He, Yutian Luo, Nanyi Fei, Guoxing Yang, Wei Wei, Huiwen Zhao, and Zhiwu Lu. Fineclip: Selfdistilled region-based clip for better fine-grained understanding. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 27896–27918. Curran Associates, Inc., 2024. URL https://proceedings.neurips.cc/paper_files/paper/ 2024/file/3122aaa22b2fe83f9cead1a696f65ceb-Paper-Conference.pdf. Ugur Sahin, Hang Li, Qadeer Khan, Daniel Cremers, and Volker Tresp. Enhancing multimodal compositional reasoning of visual language models with generative negative mining. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5563–5573, 2024. Maitreya Patel, Abhiram Kusumba, Sheng Cheng, Changhoon Kim, Tejas Gokhale, Chitta Baral, and Yezhou Yang. Tripletclip: Improving compositional reasoning of clip via synthetic vision-language negatives. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 32731–32760. Curran Associates, Inc., 2024. URL https://proceedings.neurips.cc/paper_files/paper/2024/file/ 39781da4b5d05bc2908ce08e43bc6404-Paper-Conference.pdf. Zhiqiu Lin, Xinyue Chen, Deepak Pathak, Pengchuan Zhang, and Deva Ramanan. Revisiting the role of language priors in vision-language models. In International Conference on Machine Learning, pages 29914–29934. PMLR, 2024b. Vishaal Udandarao, Mehdi Cherti, Shyamgopal Karthik, Jenia Jitsev, Samuel Albanie, and Matthias Bethge. A good crepe needs more than just sugar: Investigating biases in compositional vision-language benchmarks. arXiv preprint arXiv:2506.08227, 2025. Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901–2910, 2017. 11
Revisiting Compositionality in Dual-Encoder Vision-Language Models: The Role of Inference
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spaCy: Industrial-strength Natural Language Processing in Python. https://doi.org/10.5281/zenodo.1212303, 2020. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online, October 2020. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/2020.emnlp-demos.6.
12
Revisiting Compositionality in Dual-Encoder Vision-Language Models: The Role of Inference
Appendix of Revisiting Compositionality in Dual-Encoder Vision-Language Models: The Role of Inference A
B I SC O R-C TRL dataset information
We host B I SC O R at HuggingFace3 . We provide a summary below. Dataset documentation B I SC O R-C TRL is a benchmark of Bidirectional S WAPS for Compositional Reasoning development. Each instance consists of two images and two captions. Using each of the images and captions as a base, a model is asked to select the pair that correctly represents the base versus the hard negative distractor with minor compositional changes. Thus, we can measure image-to-text and text-to-image retrieval with hard negative pairs. To obtain good results on the dataset, it is necessary that the model performs well in both directions for the same instance. B I SC O R-C TRL is designed to have full control of the scenes and their compositions, allowing for a more detailed assessment of compositionality. It is based on CLEVR Johnson et al. [2017], where we build different variants of S WAP instances: i) C OLOR, ii) S IZE, iii) M ATERIAL and iv) Q UANTITY. Each variant consists of two splits, Development and Test, with 1,000 instances each. • image: New positive image rendered by us. • caption: Caption obtained from the scene used to render the positive image. • negative_image: New negative image rendered by us. • negative_caption: Caption obtained from the scene used to render the negative images. An example of an instance can be seen in Figure 3.
Figure 3: Example of a B I SC O R instance after loading the dataset. Maintenance plan We are committed to maintaining the dataset to resolve any technical issues. We actively track issues in the HuggingFace or GitHub repositories. Licensing Our work is licensed under the MIT License4 for the code and a Creative Commons Attribution 4.0 International License (CC BY 4.0) for the data5 . Author statement
B
We, the authors, assume full responsibility in case of violation of rights.
Structure-Guided Inference development Results
In this section, we present the development experiments and results of Structure-Guided Inference (SGI) for CLIP, Perception Encoder and SigLIP 2 models. B.1
SGI development experiments
We evaluate six different configurations of Structure-Guided Inference. For image crops, we always use crops of sizes (32, 32), (56, 56), (112, 112), (224, 224), (56, 112) and (112, 56), combining different scales and aspect ratios. We resize all the crops to the input size of the model and we deploy those crops in two different ways: i) grid, avoiding any overlap of crops of the same size, and ii) overlap, using a stride of crop_size/2. This means that we process 86 crops per image with the grid configuration, and 270 crops with overlap. 3
https://huggingface.co/datasets/imirandam/BiSCoR https://github.com/IMirandaM/revisiting-vl-compositional-inference/blob/main/LICENSE 5 https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/cc-by-4.0.md 4
1
Revisiting Compositionality in Dual-Encoder Vision-Language Models: The Role of Inference
Regarding text segmentation, we consider two different strategies: i) fine-grained, where text segments are of the form [object], [attribute + object] and [attribute + object, relation, attribute + object] and ii) coarse-grained text segments, where text segments are of the form [attribute + object] and [attribute + object, relation, attribute + object] (see Figure 4). To obtain those text segments in B I SC O R-C TRL we directly use the ground-truth scene graphs, so we know that the created text segments are perfect. We have also used SpaCy to obtain the segments, see Appendix B.5 for more details.
Figure 4: An example for our two text segmenting strategies. As can be seen, fine-grained adds two more segments to coarse-grained text segments.
B.2
CLIP development results
Table 4 shows the results of the development dataset for global embedding similarity and for different variants of Structure-Guided Inference for CLIP. Table 4: Development results for CLIP base model and different SGI configurations. FG for fine-grained and CG for coarse-grained text segments. Bold for best results. B I SC O R-C TRL Model C OLOR S IZE M ATERIAL Q UANTITY
B.3
I2T
T2I
Gr.
I2T
T2I
Gr.
I2T
T2I
Gr.
I2T
T2I
Gr.
Random
25.0
25.0
16.7
25.0
25.0
16.7
25.0
25.0
16.7
25.0
25.0
16.7
CLIP
12.8
5.3
2.0
2.8
8.1
0.4
8.8
2.7
0.5
11.9
7.8
3.5
Grid + FG Grid + CG
70.1 70.1
61.6 67.4
53.8 58.3
18.0 18.0
9.0 15.8
3.7 6.6
23.6 23.6
13.8 16.6
6.9 7.7
8.5 6.9
11.5 10.7
2.5 2.1
Over + FG Over + CG
84.9 84.9
75.0 81.5
70.4 75.6
20.0 20.0
10.2 16.2
4.5 7.2
32.9 32.9
19.4 22.7
10.3 12.9
10.2 8.6
16.4 14.5
4.7 3.6
Perception Encoder development results
Table 5 shows the results of the development dataset for global embedding similarity and for different variants of Structure-Guided Inference for Perception Encoder model. B.4
SigLIP 2 development results
Table 6 shows the results of the development dataset for global embedding similarity and for different variants of Structure-Guided Inference for SigLIP 2 model. B.5
SGI SpaCy results
To evaluate the robustness of SGI, we transitioned from manual ground-truth labels to a simplified, automated extraction pipeline using the SpaCy Honnibal et al. [2020] noun chunks utility. This shift was specifically designed to determine whether the performance gains were a result of a better exploitation of pretrained representations or merely a reflection of high-fidelity text segments. By employing a standard, non-LLM parser (i.e. "en_core_web_sm") to decompose 2
Revisiting Compositionality in Dual-Encoder Vision-Language Models: The Role of Inference
Table 5: Development results for Perception Encoder base model and different SGI configurations. FG for fine-grained and CG for coarse-grained text segments. Bold for best results. B I SC O R-C TRL Model C OLOR S IZE M ATERIAL Q UANTITY
I2T
T2I
Gr.
I2T
T2I
Gr.
I2T
T2I
Gr.
I2T
T2I
Gr.
Random
25.0
25.0
16.7
25.0
25.0
16.7
25.0
25.0
16.7
25.0
25.0
16.7
PE
23.8
9.5
4.5
36.0
25.7
11.4
23.3
12.7
5.9
24.8
21.0
11.8
Grid + FG Grid + CG
91.2 91.3
80.9 85.2
78.2 82.5
41.7 43.1
15.1 22.2
7.6 11.8
46.2 46.3
66.8 72.3
35.8 38.8
29.0 23.9
26.6 28.6
16.4 14.0
Over + FG Over + CG
96.8 96.8
91.5 94.2
90.3 92.8
46.7 47.5
20.6 34.6
12.5 21.3
51.5 51.4
82.0 85.9
47.0 47.7
33.7 26.2
30.9 33.2
21.2 16.9
Table 6: Development results for SigLIP 2 base model and different SGI configurations. FG for fine-grained and CG for coarse-grained text segments. Bold for best results. B I SC O R-C TRL Model C OLOR S IZE M ATERIAL Q UANTITY
I2T
T2I
Gr.
I2T
T2I
Gr.
I2T
T2I
Gr.
I2T
T2I
Gr.
Random
25.0
25.0
16.7
25.0
25.0
16.7
25.0
25.0
16.7
25.0
25.0
16.7
SigLIP 2
17.1
5.6
2.4
39.0
16.0
8.3
6.7
6.9
1.0
21.4
10.4
5.2
Grid + FG Grid + CG
97.1 97.1
89.6 93.0
88.6 92.0
17.7 17.7
12.4 14.7
4.1 5.0
68.9 68.9
63.4 71.5
49.7 55.5
35.0 30.7
28.1 28.0
17.7 17.1
Over + FG Over + CG
82.0 98.6
95.3 96.1
95.1 95.9
19.4 19.4
18.6 21.4
5.8 5.9
74.1 74.1
78.4 82.7
62.3 64.9
37.3 32.6
31.9 30.6
21.3 18.6
captions into entity-attribute pairs (e.g., a red cube) and global captions (e.g., a red cube and a blue sphere), allows us to verify if SGI can maintain its efficacy when decoupled from gold annotations. Table 7 shows the results for CLIP, PE, and SigLIP2 with SpaCy text segments, which are similar to those obtained with the gold annotations.
C
Detailed evaluation metrics
The I2T score measures the performance for image-to-text retrieval. For each instance in development and test datasets, we actually have two image-to-text retrieval examples. To obtain a perfect I2T score, the correct captions for both images have to be selected. Thus, assuming C0 , C1 refer to positive and negative caption respectively, I0 , I1 to positive and negative image, and we use s(Ci , Ii ) as the similarity function for a caption and an image, I2T score I2T (C0 , I0 , C1 , I1 ) is defined in Equation 2: ( 1 if s (C0 , I0 ) > s (C1 , I0 ) and s (C1 , I1 ) > s (C0 , I1 ) I2T (C0 , I0 , C1 , I1 ) = (2) 0 otherwise The T2I score T 2I(C0 , I0 , C1 , I1 ) is similarly defined for text-to-image retrieval (Equation 3): (
1
if s (C0 , I0 ) > s (C0 , I1 ) and s (C1 , I1 ) > s (C1 , I0 ) otherwise
T 2I (C0 , I0 , C1 , I1 ) = 0
(3)
Finally, the Group score G(C0 , I0 , C1 , I1 ) is the main metric, since it combines the performance for image-to-text and text-to-image retrieval. To obtain a perfect group score for a given instance, both images have to be matched with the suitable captions and both captions with the suitable images. The group score is defined in Equation 4: 3
Revisiting Compositionality in Dual-Encoder Vision-Language Models: The Role of Inference
Table 7: Results for SGI with text segments obtained with SpaCy on the B I SC O R-C TRL dataset. We provide the average group score obtained by each model using global embedding similarity, SGI with gold annotations and SGI with SpaCy text segments. We also provide the group score for all the categories of the dataset. Model
Average
C OLOR
S IZE
Categories M ATERIAL
Q UANTITY
CLIP +SGI +SGISpaCy
1.4 24.9 25.5
1.5 76.5 74.9
0.2 4.7 5.1
0.7 15.7 18.0
3.3 2.5 3.9
PE +SGI +SGISpaCy
8.5 44.7 44.7
5.9 91.3 91.5
10.9 16.9 12.3
7.7 55.5 60.8
9.4 15.3 14.1
SigLIP 2 +SGI +SGISpaCy
4.5 47.6 45.7
3.7 94.9 94.4
9.6 7.1 3.6
1.5 68.3 68.3
3.1 20.2 16.5
(
1
if I2T (C0 , I0 , C1 , I1 ) and T 2I (C0 , I0 , C1 , I1 ) otherwise
G (C0 , I0 , C1 , I1 ) = 0
D
(4)
Learning Fine-Grained Alignment development
In this section you can find the information about the training hyperparameters and development results of Section 4 of the main paper. D.1
Training hyperparameters
Hyperparameters for TFGlobal and TFLocal training: • Learning rate: 1e-4. • Scheduler: Cosine scheduler with a warm-up of 10% of the total number of training steps. • Optimizer: AdamW optimizer with beta1 = 0.9, beta2 = 0.95, eps = 1e-08 and weight decay = 1e-7. • Loss function: Contrastive Loss. • Batch size: We define a batch size of 50 (50 images x 50 captions) for COCO dataset. For TROHN-Img, as we have hard negatives, we define a batch size of 25, and then we add negatives, obtaining 25 positive pairs and 25 negative pairs in the same batch. • Epochs: We fine-tune all models over 5 epochs and we use validation accuracy as the model selection criterion, i.e. we selected the model with the highest accuracy on the corresponding validation set. • Embedding layer: For TFGlobal , because it relies on the same embeddings used for global embedding similarity, we extract representations from the final layer. For CLIP TFLocal , following the established practice in the literature, e.g. LLaVA Liu et al. [2023] and VQAScore Lin et al. [2024b], we use embeddings from the penultimate layer for both encoders, vision and text. For PE TFLocal we use embeddings from the last layer. For all PE full fine-tunings we have used the same hyperparameters as in Miranda et al. [2024] for CLIPCOCO and CLIPTROHN-Img . Detailed hyperparameters: • Learning rate: 1e-6. • Scheduler: Cosine scheduler with 50 warmup steps. • Optimizer: AdamW optimizer with beta1 = 0.9, beta2 = 0.98, eps = 1e-6 and weight decay = 0.1. • Loss function: InfoNCE Loss. • Batch size: We define a batch size of 400 (400 images x 400 captions) with COCO dataset and for TROHN-Img a batch size of 200, and then we add the 200 negatives, resulting in 400 images x 400 captions. 4
Revisiting Compositionality in Dual-Encoder Vision-Language Models: The Role of Inference
Table 8: Development results for each variant and dataset with different number of layers. For validation, we provide batch accuracy. In bold, the best for each variant. Model
VALIDATION
Dataset
Nº layers
COCO
1 2 3 4
91.7 93.0 93.2 93.2
TROHN-Img
1 2 3 4
80.5 83.8 84.0 84.3
COCO
1 2 3 4
91.5 94.2 94.3 94.4
TROHN-Img
1 2 3 4
81.8 86.4 86.8 87.2
Accuracy
TFGlobal
TFLocal
• Epochs: We fine-tune all models over 10 epochs and we used validation accuracy as the model selection criterion, i.e. we selected the model with the highest accuracy on the corresponding validation set. More information about training data, models, software and hardware can be found in Appendix E D.2
Learning Fine-Grained Alignment development results
Table 8 shows the results of our four learnable variants with different number of layers, from 1 to 4, in the validation set of each training data.
E
Implementation details
This appendix contains all the information related to the implementation of the experiments. All the source can be found at https://github.com/IMirandaM/revisiting-vl-compositional-inference. E.1
Source datasets
We obtain all source datasets directly from the original sources published by the authors. To the best of our knowledge, all data sources we use are open to non-commercial use, do not contain personally identifiable information and do not contain offensive content. • CLEVR Lin et al. [2014]: We obtain CLEVR scenes from the official project website6 under a Creative Commons Attribution 4.0 License. E.2
Training datasets
To train TFGlobal and TFLocal , we used the following two datasets: • COCO Lin et al. [2014]: We obtain COCO 2017 from the official project website7 under a Creative Commons Attribution 4.0 License8 . It contains 591,753 captions and 118,287 images, i.e., 591,753 instances formed by an image and a caption. 6
https://cs.stanford.edu/people/jcjohns/clevr/ https://cocodataset.org/#download 8 https://cocodataset.org/#termsofuse 7
5
Revisiting Compositionality in Dual-Encoder Vision-Language Models: The Role of Inference
• TROHN-Img Miranda et al. [2024]: We obtain TROHN-Img from the official Hugging Face repository 9 under the MIT license. It contains 296,070 instances formed by two images and two captions, i.e. 592,140 pairs, an amount similar to that of the COCO 2017 train. E.3
Evaluation datasets
We obtain all evaluation datasets directly from the original sources published by the authors. • B I VLC Miranda et al. [2024]: We obtain B I VLC from the official Hugging Face repository10 . • S UGAR C REPE Hsieh et al. [2024]: We obtain S UGAR C REPE from the official GitHub repository11 . E.4
Software information
Models
We detail the sources of models we used.
• CLIP: We obtain the pretrained baseline VIT-B-32 OpenAI’s CLIP model Radford et al. [2021] from Hugging Face12 . • SigLIP 2: We obtain all SigLIP 2 Tschannen et al. [2025] models from Hugging Face collection13 . – SigLIP 2: We obtain siglip2-base-patch32-256 from the official Hugging Face repository14 . – SigLIP 2-Giant: We obtain siglip2-giant-opt-patch16-256 from the official Hugging Face repository15 . • Perception Encoder: We obtain all Perception Encoder Bolya et al. [2025] models from Hugging Face collection16 . – Pe: We obtain PE-Core-B16-224 from the official Hugging Face repository17 . – PE-Giant: We obtain PE-Core-G14-448 from the official Hugging Face repository18 . • NegCLIP: We obtain the NegCLIP model Yuksekgonul et al. [2022] from the official GitHub repository.19 • TripletCLIP: We obtain the TripletCLIP model Patel et al. [2024] from the official GitHub repository.20 • FSC-CLIP: We obtain the FSC-CLIP model Oh et al. [2024] from the official GitHub repository.21 • FineCLIP: We obtain the FineCLIP model Jing et al. [2024] from the official GitHub repository.22 • X-VLM: We obtain the X-VLM model fine-tuned for retrieval in COCO Zeng et al. [2022] from the official GitHub repository.23 • CLIPCOCO : We obtain the CLIPCOCO model Miranda et al. [2024] from the official Hugging Face repository.24 • CLIPTROHN-Img : We obtain the CLIPTROHN-Img model Miranda et al. [2024] from the official Hugging Face repository.25 9
https://huggingface.co/datasets/imirandam/TROHN-Img https://huggingface.co/datasets/imirandam/BiVLC 11 https://github.com/RAIVNLab/sugar-crepe 12 https://huggingface.co/openai/clip-vit-base-patch32 13 https://huggingface.co/collections/google/siglip2-67b5dcef38c175486e240107 14 https://huggingface.co/google/siglip2-base-patch32-256 15 https://huggingface.co/google/siglip2-giant-opt-patch16-256 16 https://huggingface.co/collections/facebook/perception-encoder 17 https://huggingface.co/facebook/PE-Core-B16-224 18 https://huggingface.co/facebook/PE-Core-G14-448 19 https://github.com/mertyg/vision-language-models-are-bows 20 https://github.com/tripletclip/TripletCLIP/ 21 https://github.com/ytaek-oh/fsc-clip 22 https://github.com/Timsty1/FineCLIP 23 https://github.com/zengyan-97/X-VLM 24 https://huggingface.co/imirandam/CLIP_COCO 25 https://huggingface.co/imirandam/CLIP_TROHN-Img 10
6
Revisiting Compositionality in Dual-Encoder Vision-Language Models: The Role of Inference
Implementation decisions We have decided to keep the preprocessing of the images constant based on the model with the lowest resolution, i.e. CLIP, controlling that all models receive the same original image and same crops. For this, all images are preprocessed in the same way, resize to 224 and center crop. For SGI all hyperparameters described in appendix B are identical across all models. The only difference is the batch size: 25 for all the models, except 10 for FineCLIP and X-VLM and and 5 for SigLIP 2-Giant. The exact values of all the hyperparameters for full fine-tunings, TFGlobal and TFLocal can be found in Appendix D.1 Evaluation: We base our evaluations on the Transformers library Wolf et al. [2020], except for X-VLM26 , FineCLIP27 and PE28 where we use the code provided in the official repository of each project. Rendering images: E.5
For rendering the images from B I SC O R-C TRL we have used Blender 4.4.3.
Hardware information
Development experiments: All development experiments have been performed on one NVIDIA A100-SXM4-80GB GPU and 64 GB of RAM. Evaluation: As in the development experiments, the evaluation was performed on one NVIDIA A100-SXM4-80GB GPU and 64 GB of RAM. Rendering images: For rendering the images we have used an NVIDIA RTX A1000 6GB Laptop GPU. Each rendering takes around 1.5 seconds.
F
Detailed results of Learning Fine-Grained Alignment
In this appendix you can find the detailed results of all the variants for each model, CLIP and PE, and the results for end-to-end models in the three test datasets. F.1
S UGAR C REPE detailed results
Table 9 shows the results of all the variants for each model, CLIP and PE, and the results for end-to-end models in S UGAR C REPE. F.2
B I VLC detailed results
Table 10 shows the results of all the variants for each model, CLIP and PE, and the results for end-to-end models in B I VLC. F.3
B I SC O R-C TRL Detailed results
Table 11 shows the results of all the variants for each model, CLIP and PE, and the results for end-to-end models in B I SC O R-C TRL.
26
https://github.com/zengyan-97/X-VLM https://github.com/Timsty1/FineCLIP 28 https://github.com/facebookresearch/perception_models 27
7
Revisiting Compositionality in Dual-Encoder Vision-Language Models: The Role of Inference
Table 9: Average group score and per category for S UGAR C REPE divided in CLIP and PE families. In bold, the best for each family and underlined the second best. S UGAR C REPE Model AVERAGE R EPLACE S WAP A DD
CLIP FTCOCO FTTROHN-img TFGlobal-COCO TFGlobal-TROHN-img TFLocal-COCO TFLocal-TROHN-img NegCLIP TripletCLIP FSC-CLIP X-VLM FineCLIP
73.0 80.9 85.5 73.2 82.2 80.9 86.3 83.6 82.6 85.1 81.9 80.6
80.8 86.1 89.4 80.7 85.3 83.2 86.9 85.5 88.8 88.1 88.2 86.0
63.3 68.9 72.0 63.5 68.8 76.3 77.1 76.6 71.9 77.6 67.4 68.2
75.1 87.8 95.2 75.5 92.5 83.2 94.8 88.8 87.2 89.5 90.1 87.7
PE FTCOCO FTTROHN-img TFGlobal-COCO TFGlobal-TROHN-img TFLocal-COCO TFLocal-TROHN-img
84.4 85.7 90.1 78.0 86.9 84.4 89.2
87.9 88.2 92.4 84.5 89.9 85.2 91.3
76.8 76.1 80.3 68.0 74.9 80.6 80.8
88.4 92.9 97.5 81.4 95.9 87.4 95.6
Table 10: Average group score and per category for B I VLC divided in CLIP and PE families. In bold, the best for each family and underlined the second best. B I VLC Model AVERAGE R EPLACE S WAP A DD
CLIP FTCOCO FTTROHN-img TFGlobal-COCO TFGlobal-TROHN-img TFLocal-COCO TFLocal-TROHN-img NegCLIP TripletCLIP FSC-CLIP X-VLM FineCLIP
36.8 47.5 57.5 38.4 49.0 45.7 61.3 44.9 35.2 46.5 40.9 39.4
57.3 69.9 76.6 59.9 69.1 62.8 74.1 68.0 54.9 68.8 64.1 63.5
13.7 20.9 27.9 11.4 15.9 24.0 39.0 18.7 9.8 19.2 12.3 8.6
39.6 51.6 68.0 44.0 61.9 50.3 70.7 48.0 40.8 51.6 46.3 46.1
PE FTCOCO FTTROHN-img TFGlobal-COCO TFGlobal-TROHN-img TFLocal-COCO TFLocal-TROHN-img
41.5 56.3 68.2 44.6 58.3 53.3 67.1
64.8 75.8 82.2 66.8 77.5 69.7 80.9
13.4 31.8 43.7 15.6 24.5 33.7 42.9
46.3 61.3 78.7 51.4 72.8 56.4 77.7
8
Revisiting Compositionality in Dual-Encoder Vision-Language Models: The Role of Inference
Table 11: Average group score and per category for B I SC O R-C TRL divided in CLIP and PE families. In bold, the best for each family and underlined the second best. B I SC O R-C TRL Model AVERAGE C OLOR S IZE M ATERIAL Q UANTITY
CLIP FTCOCO FTTROHN-img TFGlobal-COCO TFGlobal-TROHN-img TFLocal-COCO TFLocal-TROHN-img NegCLIP TripletCLIP FSC-CLIP X-VLM FineCLIP
1.4 1.4 1.9 1.2 1.2 15.1 13.2 1.8 1.2 1.2 1.7 1.4
1.5 1.9 5.1 1.8 1.8 49.5 46.4 2.4 1.4 1.6 3.7 2.2
0.2 1.3 0.0 1.4 0.0 1.6 0.1 0.1 0.7 0.8 0.0 1.5
0.7 0.4 0.0 0.2 1.6 0.1 0.4 0.5 0.7 0.0 0.1 0.2
3.3 1.9 2.3 1.2 1.4 9.2 5.8 4.2 2.0 2.5 3.1 1.8
PE FTCOCO FTTROHN-img TFGlobal-COCO TFGlobal-TROHN-img TFLocal-COCO TFLocal-TROHN-img
8.5 8.3 9.8 1.9 1.2 30.0 24.0
5.9 6.8 9.7 1.5 1.2 72.4 60.4
10.9 9.9 7.9 1.6 1.4 19.8 8.6
7.7 8.9 10.0 2.1 0.9 15.8 20.5
9.4 7.7 11.5 2.4 1.3 12.1 6.5
9