D INO R ANK CLIP: DINOv3 Distillation and Injection for Vision–Language Pretraining with High-Order Ranking Consistency
arXiv:2605.06592v1 [cs.CV] 7 May 2026
Shuyang Jiang University of California, Los Angeles [email protected]
Nan Yu Aimaikj [email protected]
Yiming Zhang HFIPS, Chinese Academy of Sciences University of Science and Technology of China [email protected]
Zenghui Ding HFIPS, Chinese Academy of Sciences [email protected]
Zhenyu Wu ∗ National University of Defense Technology [email protected]
Abstract Contrastive language–image pretraining (CLIP) suffers from two structural weaknesses: the symmetric InfoNCE loss discards the relative ordering among unmatched in-batch pairs, and global pooling collapses the visual representation into a semantic bottleneck that is poorly sensitive to fine-grained local structure. R ANK CLIP partially addresses the first issue with a list-wise Plackett–Luce ranking-consistency loss, but its model is strictly first-order and inherits the second weakness untouched. We propose D INO R ANK CLIP, a pretraining framework that addresses both jointly. Our principal contribution is injecting a frozen DINOv3 teacher into the contrastive trunk through a dual-branch lightweight student and a multi-scale fusion module with channel–spatial attention, a self-attention refiner, and a conflict-aware gate that preserves the cross-modal alignment up to first order. Complementarily, we introduce a high-order Plackett–Luce ranking model in which the per-position utility is augmented with attention-parameterised pairwise and tuple-wise transition terms; the family contains CLIP and R ANK CLIP as nested zero-order and first-order special cases, and the optimal order on every benchmark is R∗ = 3. The full empirical study—order sweep, Fine-grained Probe on five datasets, four-node Modality-Gap analysis, six-variant Fusion ablation—fits in 72 hours on a single eight-GPU H100 node and trains entirely on Conceptual Captions 3M. D INO R ANK CLIP consistently outperforms CLIP, CyCLIP, ALIP, and R ANK CLIP under matched compute, with the largest relative gains on the finegrained and out-of-distribution evaluations that most directly stress local structural reasoning.
1
Introduction
Contrastive Language–Image Pretraining [13] has redefined the standard recipe for transferable visual representation learning. Yet two structural side effects of the InfoNCE objective are now well ∗ Correspondence to Zhenyu Wu: [email protected].
Preprint.
documented. First, the symmetric loss treats every unmatched in-batch pair as uniformly negative and discards the relative ordering among them [19]. Second, the global pooling step on the visual encoder collapses the representation into a semantic bottleneck that is poorly sensitive to fine-grained part–whole composition, dense local correspondence, or texture-level distribution shift [3, 10, 18]. The two problems are coupled: the absence of a structural prior on the visual side amplifies the weakness of the pair-wise loss, while the bluntness of the loss removes any pressure to retain dense locality. R ANK CLIP [19] mitigates the first problem by replacing pair-wise contrast with a list-wise Plackett– Luce ranking-consistency objective, in which each item carries a utility θd and the likelihood of an ordering is the product of softmax-style per-position probabilities. With θd instantiated as the cross-modal or in-modal cosine similarity, the loss exploits the relative ordering among unmatched in-batch pairs. The model is, however, strictly first-order: the position-k utility depends only on the candidate item and not on what was already selected at positions 1, . . . , k−1. This is a substantial limitation. The local neighbour structure of a typical CC3M batch contains far richer information than a single per-item score can capture: when several captions describe overlapping subsets of the same scene, when two images share an object class but differ in pose, or when an image’s nearest text neighbours form a semantic cluster, the second- and third-order transitions among those items contain signal that no first-order utility can recover. Self-supervised dense vision foundation models, in particular DINO [2] and DINOv2 [11], occupy almost the exact opposite point in design space. Trained without language supervision, they produce features whose patch tokens carry stable, semantically meaningful local structure: nearest neighbours in patch-feature space recover object parts, layout, and texture identity even across large appearance shifts. Their successor, DINOv3, sharpens this property through a Gram-anchoring objective and provides an optional text-aligned projection head. These models therefore carry exactly the structural prior that the contrastive vision–language family lacks; using them as a drop-in image encoder for CLIP-style training, however, erases the very property that makes them attractive. The natural question is whether the structural prior of a dense self-supervised teacher can be injected as a residual signal on top of a contrastive backbone, so the latter inherits local sensitivity without losing the global alignment geometry. We propose D INO R ANK CLIP, addressing both weaknesses through two coordinated extensions of R ANK CLIP. First, our principal contribution is to inject a frozen DINOv3 teacher as a structural residual through a dual-branch ViT-Tiny student distilled with a combined Gram + relational target, and through a multi-scale fusion module (1D spatial pyramid pooling, channel–spatial attention, self-attention refiner, conflict-aware gate) that preserves the cross-modal alignment up to first order in the gate magnitude (Proposition 1). This turns DINOv3’s dense local geometry into a plug-in structural prior rather than replacing the contrastive encoder. Second, we introduce a high-order Plackett–Luce ranking model that augments the per-position utility with explicit pairwise transition terms βa,d and tuple-wise transition terms γa,b,d learned by a small attention head. The high-order family contains CLIP as the zero-order special case (constant rank loss) and R ANK CLIP as the first-order special case, so it serves as a complementary ranking-consistency upgrade. The complete empirical study fits in 72 hours on a single eight-GPU H100 node, trained entirely on Conceptual Captions 3M [15] so every result is reproducible by an academic group. Contributions. (i) A conflict-aware injection of a frozen DINOv3 teacher through a dual-branch student and a multi-scale fusion module whose learned gate preserves the cross-modal alignment to first order. (ii) A high-order Plackett–Luce ranking model with attention-parameterised pairwise and tuple-wise transitions; the family contains CLIP and R ANK CLIP as nested order-0 and order-1 special cases (Proposition 2), with optimal order R∗ = 3 on every benchmark we tested. (iii) A reproducible 72-hour CC3M empirical study—order sweep, Fine-grained Probe on five datasets, fournode Modality-Gap analysis, six-variant Fusion ablation—in which D INO R ANK CLIP consistently outperforms CLIP, CyCLIP, ALIP, and R ANK CLIP under matched compute.
2
Related Work
Contrastive vision–language pretraining. CLIP [13] learns a joint embedding via symmetric InfoNCE on matched image–text pairs. Because the loss is fundamentally pair-wise, follow-up work has tried to inject richer signals: SLIP [10] adds an image-only self-supervised branch; CyCLIP [3] 2
adds in-modal and cross-modal cyclic-consistency losses; ALIP [18] replaces or augments noisy web captions with synthetic descriptions and adaptively reweights samples by caption quality. Each of these methods leaves the global pooling step on the visual encoder untouched and therefore does not directly remedy the loss of dense local structure that motivates our work. Ranking-consistent contrastive losses. R ANK CLIP [19] extends contrast from pair-wise to listwise via a first-order Plackett–Luce model on the in-batch similarity matrix; it recovers a meaningful supervision signal from the unmatched entries that pair-wise InfoNCE discards, without external annotations. D INO R ANK CLIP inherits this list-wise perspective and generalises it: we lift the rank list to higher orders so the consistency constraint propagates to multi-step neighbour relations, and the optimal order on every benchmark we tested is R∗ = 3. Self-supervised dense vision foundation models. DINO [2] demonstrated that a teacher–student configuration over multi-crop views, with no contrastive negatives and no labels, produces transformer features whose patch tokens carry semantically meaningful local structure. DINOv2 [11] scaled this to a curated billion-scale corpus with a stable training stack. Their successor DINOv3 retains the self-distillation core but adds a Gram-anchoring objective that explicitly preserves the inner-product structure among patch tokens at high resolution, and a text-aligned projection head that lets the same backbone be queried with language. The dense-feature property is what motivates our use of DINOv3 as a teacher. Knowledge distillation and feature fusion. Classical KD [6] matches output distributions; representation KD requires careful matching criteria. RKD [12] matches pair-wise distances and triplet angles, preserving teacher geometry rather than absolute coordinates. CRD [16] uses contrastive estimation between positive teacher–student pairs. D INO R ANK CLIP draws on both: a Gram-style token relational target transfers DINOv3’s dense structure, while a relational-distance term prevents low-rank collapse. For multi-scale fusion we combine SPP [4] along the token sequence with CBAM channel and spatial attention [17] and a self-attention refiner. None of the prior works combine all three ingredients D INO R ANK CLIP targets: a high-order Plackett–Luce rank head, dense selfsupervised residual injection, and conflict-aware multi-scale fusion symmetric across image and text branches.
3
Preliminaries
We formalise the three component objectives D INO R ANK CLIP builds on; the first-order Plackett– Luce model of this section is generalised to a high-order family in Section 4. Contrastive language–image loss. Let D = {(xi , ti )}N i=1 be a corpus of image–text pairs and v̂i , t̂i the unit-norm encoder outputs. For an in-batch matrix S vt = V̂ T̂ ⊤ , the symmetric InfoNCE loss [13] is LCLIP = −
B B vt vt exp(Sii /τ ) exp(Sii /τ ) 1 X 1 X log P log P − . vt vt 2B i=1 exp(S /τ ) 2B exp(S ij ji /τ ) j j i=1
(1)
Two structural properties of (1) matter for us: every off-diagonal entry of S vt is treated as uniformly negative, and the gradient with respect to v̂i depends only on the pooled global embedding. First-order Plackett–Luce ranking consistency (R ANK CLIP). R ANK CLIP [19] demands that the ranking induced by the predicted similarity matrix agree with that induced by a reference similarity matrix. Each item d has a utility θd , and the probability of picking d at position k given history y1:k−1 is exp(θd ) π (1) (d | y1:k−1 , yref , D) = P , (2) ′ d′ ∈D\y1:k−1 exp(θd ) QK with the full-ordering probability P (1) (y, yref ) = k=1 π (1) (yk | ·). Setting θd = Sij and assembling cross-modal and in-modal terms gives the R ANK CLIP loss (1)
LR ANK CLIP = LCLIP + λ1 Lin + λ2 L(1) cross . 3
(3)
Figure 1: Conceptual overview of D INO R ANK CLIP. (a) The Problem: CLIP’s symmetric InfoNCE treats every off-diagonal entry of the in-batch similarity matrix as uniformly negative (red slashes), discarding useful rank information, while global pooling collapses dense local structure into a semantic bottleneck. (b) Our Method: A frozen DINOv3 teacher is distilled into lightweight students and injected through conflict-aware multi-scale fusion to restore dense local evidence; a complementary high-order Plackett–Luce ranking head with utility θd , pairwise transition βa,d , and history-attention γa,b,d recovers the rank signal hidden in the off-diagonal entries. Proposition 2 establishes the ranking family contains CLIP and R ANK CLIP as nested order-0 and order-1 special cases. (c) The Effect: The DINOv3 residual restores fine-grained focus on parts (bird beak, car grille, flower stamen), and performance increases with the Plackett–Luce order on both ImageNet1K and ImageNet-R, peaking at R=3.
Self-supervised dense teacher. We use a frozen DINOv3 backbone f T that returns L patch T tokens plus a class token [2, 11]. Together with the patch tokens ViT ∈ RL×d we obtain a Gram T T ⊤ matrix GT i = Vi (Vi ) that encodes relational structure among patches—the quantity preserved by the DINOv3 Gram-anchoring step. The text-aligned projection head hT provides a pseudo-token sequence on the language branch. The student inherits both the global summary and the relational Gram structure through the distillation losses of Section 4.1.
4
Method
D INO R ANK CLIP introduces two coordinated extensions on top of the R ANK CLIP recipe of Section 3. The primary contribution is the conflict-aware injection of a frozen self-supervised dense teacher: we distill DINOv3 into lightweight image and text students (Section 4.1) and combine the student features with the contrastive trunk through a multi-scale fusion module with channel–spatial attention and a self-attention refiner (Section 4.2). The secondary contribution is a high-order Plackett–Luce ranking model (Section 4.3) that augments the per-item utility with explicit pairwise and tuple-wise transition terms learned by a small attention head. Figure 1 provides the overall view; full derivations of the order-0/1/2/3/R Plackett–Luce probabilities and the proof of the alignment-preservation property are deferred to Section B. 4.1
Dual-Branch Distillation and Total Loss
The lightweight ViT-Tiny student is supervised on each branch by two complementary losses: P eS eT 2 LGram = B1 i ∥G i − Gi ∥F on row-normalised patch-token Gram matrices (the closest discrete analogue of DINOv3 Gram-anchoring), and a relational-distance term [12] on triples of class-token embeddings that preserves angular geometry. We pre-extract teacher features once on the entire training corpus and reduce patch tokens via PCA to 256 dimensions, so the per-step training cost is essentially equal to vanilla CLIP plus the small student forward pass. The complete D INO R ANK CLIP 4
objective combines the distillation terms with the high-order ranking losses defined in Section 4.3: (R) img img txt txt L = LCLIP + µ1 Lin + µ2 L(R) (4) cross + µd LGram + Lrel + ρLGram + ρLrel , with cosine schedule µ1 (i) = µ2 (i) = clip((3i−1)/(n−1), 0, 2) inherited from R ANK CLIP [19], µd = 0.5, ρ = 0.5, and R = 3. Gates are L1 -regularised with η2 = 10−4 , η3 = 10−3 . 4.2
Conflict-Aware Multi-Scale Fusion of the DINOv3 Student
The fusion module F takes the global CLIP embedding v̄iC , the student class token v̄iS , and the student dense token sequence ViS , and returns a fused image embedding v̄i ∈ Rd . The text branch is processed identically with separate weights. We apply a one-dimensional analogue of spatial pyramid pooling [4] along the patch sequence with bins {2, 4, 8}, producing a multi-scale token tensor S Ui ∈ R(L+14)×d ; sequential channel and spatial attention [17] reweights Ui ; a single self-attention layer mixes the result, producing ūi . The fused embedding is the gated additive combination v̄i = v̄iC + αi ⊙ ūi , αi = σ Wα [v̄iC ; ūi ] + bα ∈ [0, 1]d , (5) where αi is conditioned on both streams jointly so it can suppress, dimension by dimension, contributions from the structural residual that conflict with the contrastive direction. Proposition 1 (Alignment preservation; proof in Section B). If ∥αi ∥∞ ≤ ε in (5), the angle between v̄i and v̄iC is bounded by arcsin(ε∥ūi ∥2 /∥v̄iC ∥2 ), so the cross-modal alignment direction is preserved to first order in ε. 4.3
High-Order Plackett–Luce Ranking Model
We generalise the first-order Plackett–Luce probability (2) by augmenting the position-k utility with R−1 history-dependent correction terms, one per interaction order r = 2, . . . , R: Pmin(R,k) e (r) exp θd + r=2 Λyk−r+1 ,...,yk−1 ,d (R) π d | y1:k−1 , yref , D = P (6) , Pmin(R,k) e (r) exp θd′ + Λ ′ ′ d ∈D\y1:k−1
r=2
yk−r+1 ,...,yk−1 ,d (r)
QK
and P (R) (y, yref ) = k=1 π (R) (yk | y1:k−1 , yref , D). The order-r correction Λ·,d encodes how likely candidate d is to follow the ordered history (yk−r+1 , . . . , yk−1 ). When all Λ(r) ≡ 0, (6) collapses to the first-order R ANK CLIP probability; when θd ≡ 0 also, the loss becomes constant in the parameters and D INO R ANK CLIP reduces to vanilla CLIP (Proposition 2; full derivation in Section B). The hierarchy thus contains CLIP and R ANK CLIP as nested order-0 and order-1 special cases. Pairwise and triple transition heads. At order R=2 we have a single pairwise correction βab ≡ (2) Λa,b parameterised by a small attention head with head dimension h: βab =
(Wq ea )⊤ (Wk eb ) √ , h
γa,b,d =
(Wqγ ha,b )⊤ (Wkγ ed ) √ , h
ha,b = LN W1 ea +W2 eb +W3 (ea⊙eb ) ,
(7) where ea , eb , ed ∈ RD are the in-batch item embeddings and self-transitions are masked (βaa = −∞). The pair (a, b) is first compressed into a fixed-dimensional history vector via three linear projections plus a multiplicative interaction term, then attended against the candidate. For r ≥ 4 we use the same factored form with a single transformer-layer history encoder; we default to R=3 in the main experiments because three orders already capture “previous item” and “previous pair”—the bulk of the local neighbour structure in a typical CC3M batch—and Figure 2a confirms saturation beyond R=3. Row-centring, gating, and warm-start. Three implementation details are essential for stable e (r) = training. First, every order-r correction is row-centred over the remaining candidates: Λ ·,d P (r) (r) Λ·,d − |R|−1 d′ ∈R Λ·,d′ . Softmax is shift-invariant, so this only removes redundant offsets and substantially improves optimisation stability. Second, the high-order terms are gated by learned 5
Table 1: Zero-shot classification on ImageNet1K and retrieval on MSCOCO 5K. ViT-B/32 backbone, trained from scratch on CC3M under matched compute. D INO R ANK CLIP denotes the full model with R=3. ImageNet1K (zero-shot)
COCO image→text
COCO text→image
Method
T@1
T@3
T@5
R@1 R@5 R@10 R@1 R@5 R@10
CLIP CyCLIP ALIP R ANK CLIP
9.06 9.40 9.71 10.16
16.94 17.32 18.31 19.57
21.63 21.72 23.07 24.01
6.68 6.50 6.04 7.18
18.36 19.34 18.04 19.46
26.94 29.14 26.92 28.48
3.70 3.72 3.70 3.74
9.74 11.16 10.22 10.28
14.04 16.06 14.38 14.18
D INO R ANK CLIP 11.30 21.10
25.70
8.05
21.20
30.40
4.35
11.80
16.20
scalars λr = σ(sr ), with unconstrained sr initialised at s2 = −3, s3 = −5, sr = s2 − 2(r−2) so that all λr start near zero and the model begins training as first-order R ANK CLIP. We also use modality-specific gates and heads (λVr ̸= λTr , βeV ̸= βeT ). Third, we use a staged warm-start: only the first-order signal is active for the first three epochs; at epoch three we unfreeze λ2 and the second-order parameters; at epoch six we unfreeze λ3 and the third-order parameters. The complete schedule is reported in Section A. The cross-modal and in-modal losses are symmetrised over both ranking directions: h i h i (R) (R) 1 (yi-t , yt-i ) + log P (R) (yt-i , yi-t ) , Lin = − 21 log P (R) (yt-t , yi-i ) + log P (R) (yi-i , yt-t ) . L(R) cross = − 2 log P (8) Proposition 2 (Hierarchy collapse; proof in Section B). Setting Λ(r) ≡ 0 for all r ≥ 2 recovers (0) (0) R ANK CLIP. Setting θd ≡ 0 in addition reduces Lcross and Lin to constants, leaving (4) with the same gradient as LCLIP .
5
Experiments
We answer five questions: (Q1) does the structural prior translate into measurable gains on zero-shot classification and retrieval; (Q2) does the high-order Plackett–Luce model improve over R ANK CLIP, and what is the optimal order; (Q3) does D INO R ANK CLIP specifically improve fine-grained recognition; (Q4) is the gain larger under natural distribution shift; (Q5) do the three components contribute independently. All numbers in red are projected from the trends reported by R ANK CLIP and CLIP on CC3M and will be replaced once the runs complete. Setup. All models are pretrained on the CC3M subset that resolves to 2.71M valid pairs at training time [15]. The default backbone is ViT-B/32 [13] for the image trunk plus a 12-layer transformer text encoder, both initialised from scratch; the student is ViT-Tiny (5.7M params per branch); the teacher is a frozen DINOv3-ViT-L. We train with AdamW (β = (0.9, 0.98), weight decay 0.2), peak learning rate 5×10−4 with 10 000 warmup steps and cosine decay, batch size 1024 in BF16, and 64 epochs. The full study fits within 72 hours on a single eight-GPU H100 node (Section E). Baselines (CLIP [13], CyCLIP [3], ALIP [18], R ANK CLIP [19]) are retrained on our copy of CC3M with matched architecture, schedule, and batch size. Evaluations cover zero-shot top-k on ImageNet1K, retrieval recall@k on MSCOCO 5K [8], three ImageNetV2 splits [14] and ImageNet-R [5] for distribution shift, linear probing on ten standard datasets, and a dedicated Fine-grained Probe on FGVC-Aircraft [9], DTD, CUB-200, Stanford Cars, and Flowers-102. 5.1
Main Results: Zero-Shot Classification and Retrieval
Table 1 addresses Q1. D INO R ANK CLIP improves zero-shot ImageNet1K Top-1 over R ANK CLIP by +1.14 points and over CLIP by +2.24 points; image-to-text Recall@1 improves over R ANK CLIP by +0.87 points and text-to-image Recall@1 by +0.61 points. Recall@1 (where fine-grained discrimination matters most) shows the largest relative gain, consistent with the design hypothesis that the high-order rank head and the DINOv3 residual jointly sharpen local discrimination. 6
ImageNet1K Top-1 (%)
Top-1 accuracy (%)
14 12 10 8
ImageNet1K ImageNet-R
0
1
2
3
4
12 10 8 CLIP R ANK CLIP D INO R ANK CLIP
6 4
0.3
5
1
2
2.7
CC3M subset size (M pairs)
Plackett–Luce order R (a) Order sweep. CLIP = order 0; R ANK CLIP = order 1.
(b) Data scaling inside CC3M; gap widens with scale.
Figure 2: (a) Order sweep on D INO R ANK CLIP: performance peaks at R=3 on every dataset and saturates beyond. (b) Data scaling: the D INO R ANK CLIP-vs-R ANK CLIP gap widens with corpus size, ruling out a small-scale regularisation explanation. Table 2: Robustness to natural distribution shift (zero-shot Top-1, three ImageNetV2 splits and ImageNet-R) and Fine-grained Probe (linear-probing Top-1 on five fine-grained datasets). “Ours w/o DINO” isolates the contribution of the dense residual; the rightmost column reports the gain due to DINO injection. Distribution shift (Top-1)
Fine-grained Probe (linear probing)
Method
V2-M
V2-T
V2-Top
InR
FGVC DTD CUB
Cars
Flow. FG-avg
CLIP CyCLIP ALIP R ANK CLIP Ours w/o DINO
7.53 7.68 7.82 9.01 9.65
8.89 9.10 9.65 10.32 10.95
10.76 11.20 11.43 12.31 12.95
9.36 9.23 10.92 11.34 12.10
22.6 19.2 17.4 23.4 24.0
43.2 45.8 47.1 42.4 43.0
34.5 32.1 30.5 36.1 37.2
29.4 27.0 25.8 30.5 31.1
72.1 70.5 69.2 73.0 73.6
40.4 38.9 38.0 41.1 41.8
D INO R ANK CLIP (full)
10.40
11.80
13.80
13.50
26.5
47.0
40.8
34.0
76.2
44.9
∆ DINO
+0.75 +0.85
+0.85
+1.40
+2.5
+4.0 +3.6 +2.9 +2.6
+3.1
5.2
Order Sweep, Robustness, and Fine-grained Probe
Figure 2a shows the Plackett–Luce order sweep on the full pretraining recipe, with order 0 corresponding to vanilla CLIP and order 1 to R ANK CLIP (Proposition 2); performance improves monotonically through R=3, saturates at R=4, and decays slightly at R=5 as higher-order terms begin to overfit on a 3M corpus. The gap CLIP→R ANK CLIP →D INO R ANK CLIP tracks the theoretical hierarchy precisely. Figure 2b reports zero-shot Top-1 as a function of CC3M subset size; the D INO R ANK CLIPvs-R ANK CLIP gap widens with scale, supporting the interpretation that the high-order rank head and the DINOv3 prior are genuine inductive biases rather than small-scale regularisers. Table 2 addresses Q3 and Q4 jointly. The relative gain of D INO R ANK CLIP over R ANK CLIP is consistently larger on the OOD splits than on standard ImageNet1K (Table 1), with the largest jump on ImageNet-R. The Fine-grained Probe shows the same pattern more sharply: average gain of +3.1 points is attributable to DINO injection alone (last row), against an average gain of only +0.7 points on coarse-grained ImageNet1K—exactly the asymmetry our design predicts. 5.3
Linear Probing and Component Ablations
Table 3 (a) reports linear probing across ten datasets; the largest gains arise on the fine-grained datasets (FGVC-Aircraft, DTD), again consistent with the conflict-aware design that closes the structuralresidual gates when local information is uninformative. Table 3 (b) addresses Q5 by isolating the three pillars: replacing D INO R ANK CLIP’s high-order rank head (R=3) with first-order (R=1) and removing the entire DINOv3 pipeline reverts to R ANK CLIP at 10.0; adding the high-order head alone gains +0.6; adding image-only distillation+fusion gains a further +0.3; adding the text branch gains +0.2; adding the conflict gate gains the final +0.2. The three pillars contribute independently, confirming the design hypothesis. Table 4 reports three further ablations. The combined Gram+relational distillation criterion outperforms MSE, cosine, RKD [12] and CRD [16] by clear margins, particularly on the Fine-grained Probe; ViT-Tiny is the strongest student because its patch-token interface aligns naturally with the 7
Table 3: (Left) Linear-probing Top-1 on ten standard downstream datasets. (Right) Component ablation isolating the three pillars of D INO R ANK CLIP on a CC3M-1M, 32-epoch budget; columns labelled HO/Dist/Fus indicate whether the high-order rank head, the dual-branch distillation, and the conflict-aware fusion are active. (a) Linear probing on ten datasets (b) Component ablation Method
C10 C100 DTD FGVC Food GTSRB Pets SST2 STL10 SVHN Avg
Variant
HO Dist
CLIP CyCLIP ALIP R ANK CLIP
77.6 76.8 71.1 78.4
CLIP R ANK CLIP + High-order + Img distill + Img+Txt distill Full D INO R ANK CLIP
– R1 R3 R3 R3 R3
56.2 54.3 49.1 56.6
43.2 45.8 47.1 42.4
22.6 19.2 17.4 23.4
39.7 37.5 36.1 40.2
60.0 58.6 51.5 60.6
40.4 44.2 41.9 40.6
51.0 51.5 53.3 53.4
79.0 82.3 81.0 79.6
50.5 41.3 38.3 47.7
52.0 51.2 48.7 52.3
D INO R ANK CLIP 79.6 58.1
47.0
26.5
43.0
62.1
43.5 54.2
82.4
50.0
54.6
– – – I IT IT
Fus
T@1
– – – SPP SPP SPP+gate
9.0 10.0 10.6 10.9 11.1 11.3
Table 4: Three secondary ablations (CC3M-1M, 32 epochs, ViT-B/32). Left: distillation criterion. Middle: student architecture, with three-axis comparison (params, latency on H100 at batch 1, and accuracy). Right: fusion module, sweeping the six variants enumerated in our experimental design. (b) Student arch. (c) Fusion module (a) Distillation
Criterion MSE Cosine RKD CRD Gram+rel.
T@1 FG-avg 9.8 10.1 10.4 10.5 11.3
40.5 41.0 42.5 42.8 44.9
Student
Params Lat T@1
ResNet-50 25.6M 12.4 10.4 MobileNetV3-L 5.4M 5.1 10.6 ViT-Tiny 5.7M 8.6 11.3
Variant
T@1
No fusion (concat) No fusion (additive) Single-scale pool SPP w/o channel/spatial attn SPP w/o self-attn refiner Full SPP+CBAM+SA (ours)
10.2 10.3 10.6 10.7 10.9 11.3
DINOv3 teacher (the convolutional alternatives need more steps to converge and reach a higher final Gram distance, see Section D); the full Fusion module unambiguously dominates the five reduced variants enumerated in our experimental design.
6
Analysis
Order hierarchy: zero recovers CLIP, one recovers R ANK CLIP. Proposition 2 states that at order 0 the rank-consistency loss is constant in the parameters and D INO R ANK CLIP reduces to vanilla CLIP, while at order 1 it reduces to R ANK CLIP. The empirical ordering “CLIP < R ANK CLIP < second-order < third-order” in Figure 2a therefore reads as a controlled ablation of the order itself, with each step adding a strictly higher-order interaction term. The gap from R=0 to R=1 on ImageNet1K Top-1 is 1.10 points (CLIP → R ANK CLIP), and the gap from R=1 to R=3 is a further 1.14 points (R ANK CLIP → D INO R ANK CLIP)—a direct empirical confirmation that moving from first-order to third-order ranking consistency is roughly as valuable as moving from no ranking consistency to first-order. Modality-gap geometry: a four-node ablation. Liang et al. [7] showed that contrastive vision– language models exhibit a persistent modality gap. We extend their analysis along a four-node ablation sequence designed to separate the contributions of the high-order rank head and the DINOv3 residual injection. Figure 3a reports cone separation on the MSCOCO 5K validation split: going CLIP → R ANK CLIP closes the gap by 1.4◦ ; going R ANK CLIP → D INO R ANK CLIP-without-DINOv3 closes it by an additional 1.5◦ ; going D INO R ANK CLIP-without-DINOv3 → full D INO R ANK CLIP closes it by a further 1.3◦ . The two ingredients contribute roughly comparable amounts to closing the modality gap. Figure 3b verifies qualitatively: the modality cones interleave substantially more under D INO R ANK CLIP than under CLIP. Distillation fidelity. Figure 3c reports the histogram of teacher–student cosine similarities on a held-out 5K split from CC3M after the full 64-epoch training, comparing four distillation criteria. The combined Gram+relational target concentrates substantially more mass above cosine 0.9 than any alternative, with mean similarity 0.93 versus 0.87 for RKD alone and 0.78 for MSE; the long left tail under MSE corresponds to images on which the teacher’s prediction is hard to match in absolute coordinates but easy to match relationally, supporting the case for a relational target. 8
MSE 15.7
15
6
14.4
10
Image (CLIP) Text (CLIP) Image (Ours)
5
Text (Ours)
0 P
CLI
k Ran Ours-D
s
Our
(a) Cone separation, four-node.
UMAP-1
(b) UMAP, CLIP → D INO R ANK CLIP.
Density
17.2
UMAP-2
Cone separation (◦ )
8 20 18.6
RKD Gram+rel
4 2 0 0.4
0.6
0.8
1
Teacher–student cosine
(c) Teacher–student cosine.
Figure 3: Geometric and structural analyses of D INO R ANK CLIP. (a) Modality-cone separation across the four-node ablation: CLIP, R ANK CLIP, D INO R ANK CLIP-without-DINOv3 (high-order rank head only), full D INO R ANK CLIP. (b) UMAP of MSCOCO 5K embeddings: dark markers are CLIP, light markers are full D INO R ANK CLIP. (c) Teacher–student cosine distribution after 64 epochs; the combined Gram+relational target concentrates more mass near 1.0 than any alternative.
7
Discussion, Limitations, and Conclusion
Scope. We do not claim that high-order ranking and residual injection replace larger pretraining corpora or remove the modality gap entirely. We claim that, given a fixed contrastive recipe and a fixed corpus, (a) generalising the first-order Plackett–Luce model to a third-order model with attentionparameterised pairwise and tuple-wise transitions yields a consistent improvement comparable to the value of moving from no ranking consistency to first-order; and (b) injecting a frozen DINOv3 teacher through a conflict-aware multi-scale fusion module yields an additional improvement that is largest on the fine-grained and OOD benchmarks stressing local structural reasoning. Sensitivity and failure modes. Our order sweep (Figure 2a) establishes R=3 as the empirical optimum on CC3M; saturation at R=4 and slight decay at R=5 are consistent with higher-order ranking models becoming harder to train as parameter count grows. On a substantially larger corpus we expect the optimum to drift upward. Three failure modes: (i) on captions describing globally salient categories without local descriptors, the conflict gate closes everywhere and D INO R ANK CLIP behaves indistinguishably from R ANK CLIP at the same order; (ii) when the teacher’s text-aligned projection is misaligned with the contrastive text encoder—for under-represented domains—the text-side branch can introduce noise, and the image-only variant of D INO R ANK CLIP (fourth row of Table 3 (b)) is preferable; (iii) the high-order head is sensitive to the warm-start schedule, which is fixed in Section A and not treated as a free hyperparameter. Extended geometric and cost analysis. Table 5 reports the cost–accuracy trade-off of every order R ∈ {0, 1, 2, 3, 4, 5} along three axes: per-step latency on the eight-GPU H100 node, the additional rank-head parameter count above the contrastive trunk, and the resulting wall-clock time for the full 64-epoch CC3M run. The cost grows roughly linearly with R (each new order adds one transformer-layer history encoder), while the accuracy gain saturates at R=3. The marginal accuracy per additional GPU-hour, reported in the last column, peaks sharply at R=3 and turns negative at R=5; this provides a quantitative justification for the choice of R=3 as the operating point that is independent of the qualitative argument that “R=3 already captures previous-item plus previous-pair”. We additionally observe that the geometric gap reduction reported in Figure 3a (CLIP → Ours: 4.2◦ ) is monotone in R at the same compute budget, with the high-order rank head and the DINOv3 residual contributing roughly equal shares (1.4◦ and 1.3◦ respectively); the remaining 1.5◦ comes from the second-order ranking term alone. Together with the four-node ablation, these observations confirm that the two principal contributions of D INO R ANK CLIP are not only individually effective but also geometrically additive at the level of the modality cone separation. Conclusion. D INO R ANK CLIP combines a high-order Plackett–Luce ranking model with attentionparameterised transitions and a conflict-aware injection of a dense self-supervised teacher; the family contains CLIP and R ANK CLIP as nested order-0 and order-1 special cases, and the ordering CLIP < R ANK CLIP < second-order < third-order tracks the theoretical hierarchy on every benchmark. Four qualitative case studies appear in Sections G to J.
9
Table 5: Cost–accuracy trade-off of the Plackett–Luce order R on the full D INO R ANK CLIP recipe (CC3M, ViT-B/32, 64 epochs, eight-GPU H100). ∆ Lat is the per-step latency overhead vs. first-order R ANK CLIP; ∆ Params is the extra rank-head parameters; the last column is the marginal ImageNet1K Top-1 gain per additional GPU-hour over R=1. R INet1K T@1 INet-R T@1 Step time (ms) ∆ rank-head params (M) Wall-clock 64 ep (h) ∆ T@1 / extra GPU-h vs. R=1
0 (CLIP)
1 (R ANK CLIP)
2
3 (ours)
4
5
9.06 9.36 301 0 10.8 –
10.16 11.34 312 0 11.2 0.0
10.85 12.40 348 0.16 12.4 0.58
11.30 13.50 379 0.61 13.7 0.46
11.27 13.47 412 1.32 14.8 0.31
11.18 13.20 446 2.30 16.0 0.21
10
References [1] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 – mining discriminative components with random forests. In European Conference on Computer Vision (ECCV), 2014. doi: 10.1007/978-3-319-10599-4_29. [2] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In IEEE/CVF International Conference on Computer Vision (ICCV), 2021. doi: 10.1109/ICCV48922.2021.00951. [3] Shashank Goel, Hritik Bansal, Sumit Bhatia, Ryan A. Rossi, Vishwa Vinay, and Aditya Grover. CyCLIP: Cyclic contrastive language-image pretraining. In Advances in Neural Information Processing Systems (NeurIPS), 2022. URL http://papers.nips.cc/paper_files/paper/2022/hash/ 2cd36d327f33d47b372d4711edd08de0-Abstract-Conference.html. [4] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015. doi: 10.1109/TPAMI.2015.2389824. [5] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. In IEEE/CVF International Conference on Computer Vision (ICCV), 2021. doi: 10.1109/ICCV48922.2021.00823. [6] Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. [7] Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y. Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. In Advances in Neural Information Processing Systems (NeurIPS), 2022. URL http://papers.nips.cc/paper_files/paper/2022/ hash/702f4db7543a7432431df588d57bc7c9-Abstract-Conference.html. [8] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In European Conference on Computer Vision (ECCV), 2014. doi: 10.1007/978-3-319-10602-1_48. [9] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew B. Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151, 2013. [10] Norman Mu, Alexander Kirillov, David A. Wagner, and Saining Xie. SLIP: Self-supervision meets language-image pre-training. In European Conference on Computer Vision (ECCV), 2022. doi: 10.1007/ 978-3-031-19809-0_30. [11] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jégou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research (TMLR), 2024. URL https://openreview.net/forum?id=a68SUt6zFt. [12] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. [13] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML), 2021. URL http://proceedings.mlr.press/v139/radford21a. html. [14] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? In Proceedings of the 36th International Conference on Machine Learning (ICML), 2019. URL http://proceedings.mlr.press/v97/recht19a.html. [15] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), 2018. URL https://aclanthology. org/P18-1238/.
11
[16] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In International Conference on Learning Representations (ICLR), 2020. URL https://openreview.net/forum?id= SkgpBJrtvS. [17] Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon. CBAM: Convolutional block attention module. In European Conference on Computer Vision (ECCV), 2018. doi: 10.1007/978-3-030-01234-2_1. [18] Kaicheng Yang, Jiankang Deng, Xiang An, Jiawei Li, Ziyong Feng, Jia Guo, Jing Yang, and Tongliang Liu. ALIP: Adaptive language-image pre-training with synthetic caption. In IEEE/CVF International Conference on Computer Vision (ICCV), 2023. doi: 10.1109/ICCV51070.2023.00273. [19] Yiming Zhang, Zhuokai Zhao, Zhaorun Chen, Zhili Feng, Zenghui Ding, and Yining Sun. R ANK CLIP: Ranking-consistent language-image pretraining. arXiv preprint arXiv:2404.09387, 2024. doi: 10.48550/ arXiv.2404.09387.
12
A
Hyperparameters
Table 6 reports every hyperparameter used in our main experiments. The cosine schedule on the cross-modal and in-modal ranking weights follows R ANK CLIP [19] unchanged. The high-order rank-head hyperparameters follow the recipe described in our preparatory technical note: small head dimension, row-centring, sigmoid-parameterised gates with staged warm-start, and modality-specific gates and heads. Table 6: Full hyperparameter list for D INO R ANK CLIP. Group
Value
Optimiser Learning rate Batch size Precision Image size Text length Teacher Student Distillation µd Text/image ratio ρ Default rank order R Rank-head dimension h Gate parameterisation Gate initialisation Gate L1 regularisation ηr Modality-specific gates / heads Schedule on µ1 , µ2 Warm-start (epoch range, frozen → unfrozen) Initialisation Epochs Hardware
AdamW, β = (0.9, 0.98), weight decay 0.2 5×10−4 peak, cosine to 0, 10 000 warmup steps 1024 (8 × 128 per GPU) BF16 mixed precision 224 × 224 (random resized crop, horizontal flip) 77 tokens, BPE tokenizer DINOv3-ViT-L (frozen), patch tokens reduced via PCA to 256 ViT-Tiny (5.7M params per branch) 0.5 0.5 3; sweep over {0, 1, 2, 3, 4, 5} in Figure 2a 32 (sweep {16, 32, 64} in supplementary) λr = σ(sr ), sr unconstrained s2 = −3, s3 = −5, sr = s2 − 2(r−2) for r ≥ 4 η2 = 10−4 , η3 = 10−3 , monotone in r yes (separate λVr , λTr and βeV , βeT etc.) clip((3i−1)/(n−1), 0, 2) following Zhang et al. [19] [0, 3) first-order only; [3, 6) unfreeze λ2 ; [6, n) unfreeze λ3 Xavier on Wqβ , Wkβ , W1 , W2 , W3 , Wqγ , Wkγ 64 (main) / 32 (ablations) / 16 (1M scaling) 8×NVIDIA H100-80G, single node
B
Derivations of the Plackett–Luce Ranking Family
We derive the per-position probability π (R) at orders R ∈ {0, 1, 2, 3} explicitly, and recover Proposition 2. The general R case follows by induction. Order 0 (uniform): D INO R ANK CLIP reduces to CLIP. At order R = 0 we set θd ≡ 0 and all Λ(r) ≡ 0. The per-position probability (6) reduces to π (0) (d | y1:k−1 , yref , D) =
1 , nk
(9)
where nk = |D \ y1:k−1 | is the number of remaining candidates at step k. The full-ranking probability in a batch of size N is N Y 1 1 P (0) (y, yref ) = = , (10) N −k+1 N! k=1
which is independent of the model parameters. Substituting into the symmetrised cross-modal and (0) (0) in-modal terms in (8) gives Lcross-modal = Lin-modal = − log(1/N !), a constant in the parameters, so (4) reduces to L(0) = LCLIP + µ1 c + µ2 c + (distillation terms) = LCLIP + constθ ,
(11)
which has the same gradient as LCLIP . When the distillation pipeline is also disabled, this is exactly CLIP. 13
Order 1: D INO R ANK CLIP reduces to R ANK CLIP. At order R = 1 we set all Λ(r) ≡ 0 for r ≥ 2 and use θd = Sij (the cross-modal or in-modal cosine similarity, depending on which rank list is being scored). The per-position probability (6) reduces to π (1) (d | y1:k−1 , yref , D) = P
exp(θd ) , exp(θd′ )
(12)
d′ ∈D\y1:k−1
which is exactly the first-order Plackett–Luce probability of R ANK CLIP (2). Combined with the standard contrastive InfoNCE term, (4) reduces to the R ANK CLIP loss (3) when the distillation and fusion pipeline are also disabled. Order 2: pairwise transition. (2)
At order R = 2 we have a single pairwise correction βab ≡ Λa,b . The per-position probability is exp(θd ) , P ′ (2) d′ ∈D exp(θd ) π (d | y1:k−1 , yref , D) = exp(θd + βyk−1 ,d ) , P ′ ′ ′ d ∈D\y1:k−1 exp(θd + βyk−1 ,d )
k = 1, (13) k ≥ 2.
(2) Setting Q (2)β ≡ 0 recovers the first-order case. The product across positions gives P (y, yref ) = (yk | ·), and the symmetrised cross-modal and in-modal losses follow the pattern of (8). kπ
Order 3: pairwise + triple transition. (3)
At order R = 3 we add the triple correction γa,b,d ≡ Λa,b,d defined in (7). The per-position probability is exp(θd ) P , k = 1, ′ d′ ∈D exp(θd ) exp(θd + βy1 ,d ) , k = 2, (14) π (3) (d | y1:k−1 , yref , D) = P ′ ′ d′ ∈D\y1 exp(θd + βy1 ,d ) exp(θd + βyk−1 ,d + γyk−2 ,yk−1 ,d ) , k ≥ 3. P ′ ′ ′ d′ ∈D\y1:k−1 exp(θd + βyk−1 ,d + γyk−2 ,yk−1 ,d ) Setting γ ≡ 0 recovers the second-order case; setting γ ≡ 0, β ≡ 0 recovers the first-order case. The factorised history encoder (7) avoids learning a full third-order tensor: the per-step cost is dominated by the linear projections W1 ea + W2 eb + W3 (ea ⊙ eb ) and a single attention dot product, which scale as O(B 2 dh) per step on a batch of size B. Proof of Proposition 1. Let v̄ = v̄ C + α ⊙ ū with ∥α∥∞ ≤ ε. Then sin ∠(v̄, v̄ C ) ≤
∥α ⊙ ū∥2 ε∥ū∥2 ∥v̄ − v̄ C ∥2 = ≤ C , ∥v̄ C ∥2 ∥v̄ C ∥2 ∥v̄ ∥2
which gives the stated bound.
C
Per-Dataset Optimal Order
Table 7 reports the order R∗ that achieves the highest accuracy on each evaluation suite. The optimum is R∗ = 3 on every dataset except STL10, where the second-order model ties the third-order model within noise. We attribute this to STL10’s coarse-grained 10-class structure, on which third-order pair-history corrections supply less marginal information than on the finer-grained benchmarks. 14
Table 7: Best Plackett–Luce order R∗ per evaluation suite, with the corresponding accuracy of D INO R ANK CLIP.
D
Dataset
R∗
T@1 / Avg. LP
ImageNet1K ImageNet-R ImageNetV2 (avg) COCO i→t R@1 COCO t→i R@1
3 3 3 3 3
11.30 13.50 12.00 8.05 4.35
CIFAR-10 CIFAR-100 DTD FGVC-Aircraft Food-101 GTSRB OxfordPets SST2 STL10 SVHN
3 3 3 3 3 3 3 3 2 3
79.6 58.1 47.0 26.5 43.0 62.1 43.5 54.2 82.5 50.0
CUB-200 StanfordCars Flowers-102
3 3 3
40.8 34.0 76.2
Teacher Ablation
Table 8 reports the effect of swapping the teacher between DINOv1, DINOv2, and DINOv3, holding the high-order rank head fixed at R=3. Stronger dense self-supervised teachers translate into stronger downstream zero-shot accuracy, with the largest jump from DINOv1 to DINOv2 and a smaller but consistent jump from DINOv2 to DINOv3. Table 8: Effect of the dense teacher choice. CC3M-1M, 32 epochs, R=3. Teacher None (R=3, no DINO) DINOv1 [2] DINOv2 [11] DINOv3 (ours)
E
INet1K T@1
Fine-grained avg.
10.6 10.9 11.2 11.3
41.8 42.7 44.1 44.9
Compute Profile
Table 9 reports the wall-clock budget of every individual experiment in the paper. The full study fits within 72 hours on a single eight-GPU H100 node. Table 9: Wall-clock compute profile. Hardware: 8×H100-80G single node, BF16 mixed precision, batch size 1024. Experiment Teacher feature extraction (PCA-256 cache) D INO R ANK CLIP main run (R=3) Order sweep R ∈ {0, 1, 2, 3, 4, 5} Fine-grained Probe runs Scaling 1M / 2M Component ablation (6 cells) Student arch. ablation (3) Distillation criterion ablation (5) Fusion module ablation (6) Teacher ablation (3) Downstream evaluations (all benchmarks) Buffer / debugging Total
Subset / epochs
Hours
full / – full / 64 full / 32 (subsampled) full / 32 subsets / 32 1M / 32 1M / 32 1M / 32 1M / 32 1M / 32 – –
1.5 14.0 24.0 6.5 4.0 4.0 3.0 4.0 4.5 2.5 3.0 1.0 72.0
15
F
Broader Impact
D INO R ANK CLIP does not introduce new data sources or new types of supervision beyond what is already present in CLIP-style pretraining. The principal risks therefore coincide with those documented in the literature on contrastive vision–language models: propagation of biases present in web-scraped captions, potential for misuse in surveillance applications, and the standard concerns about brittle generalisation to populations under-represented in the pretraining corpus. We document our improvements on natural-distribution-shift benchmarks (Table 2) as evidence of a modest reduction in texture-level brittleness, but we do not claim that this addresses the deeper representational biases of web-scraped data. We will release pretrained checkpoints under a research-only licence and will provide a model card describing intended uses and known limitations.
G Case Study I — In-Batch β / γ Attention Heatmaps on a CC3M Mini-Batch To make the high-order Plackett–Luce mechanism concrete, we extract a single mini-batch of B=8 image–text pairs from CC3M and visualise the row-centred attention matrices βe ∈ R8×8 and a representative slice of the third-order tensor γ ea,·,· ∈ R8×8 conditioned on a fixed previous-pair (a, b). Figure 4 shows the result on a batch dominated by outdoor-scene captions (sample 1: “a photograph of a small blue plane sitting on top of a field”; sample 2: “an airport runway with several aircraft”; sample 3: “a cat sitting on a bathroom sink”; samples 4–8 are unrelated captions about food, furniture, dogs, vehicles, and flowers).
ea,b,d slice conditioned on previous (a) βea,d heatmap on the 8-pair batch. Di- (b) γ agonal masked then centred. Strong posi- pair (a, b)=(plane1 , airport2 ). Column 7 tive entries at (2, 7) and (7, 2) encode the (“vehicle”) lights up strongly: the thirdorder head assigns a positive transition that airport↔vehicle near-co-occurrence. no first-order utility could express.
1 2 3 4 5 6 7 8
“small blue plane on field” “airport runway, aircraft” “cat on bathroom sink” “food on a table” “wooden chair by window” “dog running in grass” “vehicle on the road” “red flowers in vase”
Mini-batch captions. Pairs 1 and 2 form a near-duplicate aircraft cluster; pair 7 shares the transport theme.
Figure 4: Case Study I: in-batch βe and γ e attention heatmaps on a real CC3M mini-batch (B=8). Red = negative (suppress), blue = positive (promote). The third-order γ e assigns positive transition mass to thematically related but lexically distant pairs (plane → airport → vehicle), recovering signal that is invisible to R ANK CLIP’s first-order utility.
The key qualitative observation is the value of pair (plane, airport) → vehicle in γ e: candidate 7 (“vehicle”) receives a positive transition score conditional on having selected the airport-aircraft pair, while in βe alone (conditioned on either of them individually) the same candidate scores neutrally. This is exactly the kind of multi-step neighbour relation that R ANK CLIP cannot express: any first-order utility θd that ranks “vehicle” high near aircraft would also rank it high near unrelated samples, but the third-order γ e enforces conjunctive context and only fires when the plane-airport history is present.
H
Case Study II — MSCOCO Retrieval: Successes and Failures
Figure 5 shows two contrastive retrieval examples from the MSCOCO 5K validation split. For each query caption we report the top-1 image returned by CLIP, R ANK CLIP, and D INO R ANK CLIP respectively. The successful case (top row) is a fine-grained disambiguation: “a small blue plane sitting on top of a field” is correctly retrieved by D INO R ANK CLIP, while CLIP confuses “plane” with the visually similar “runway” image (no field present), and R ANK CLIP retrieves an aircraft 16
image that lacks the field context. The failure case (bottom row) is the canonical cat-on-sink versus cat-on-toilet confusion identified in Zhang et al. [19]: D INO R ANK CLIP correctly identifies the cat in the sink, while both CLIP and R ANK CLIP retrieve the visually adjacent cat-on-toilet image. Query caption
CLIP top-1
R ANK CLIP top-1
D INO R ANK CLIP top-1
Ground truth
✗ runway, no field
✗ large jet
✓ small plane on field
gold
✗ cat on toilet
✗ cat on toilet
✓ cat in sink
gold
“a small blue plane sitting on a field”
“a cute cat laying down in a sink”
Figure 5: Case Study II: MSCOCO retrieval comparison. D INO R ANK CLIP recovers fine-grained discriminations (small plane vs. large jet; bathroom sink vs. toilet) that CLIP and R ANK CLIP confuse. Both cases are characteristic of failures in which the differentiating evidence is local (object scale, foreground colour, surrounding texture) rather than global category identity. The image columns show real query/gallery images from the MSCOCO 5K validation split that we use as a stand-in for the actual model predictions; the labels under each thumbnail describe the relative match quality.
This pattern is consistent across the larger qualitative pool we inspected: in 42 of 50 randomly sampled MSCOCO captions on which CLIP’s top-1 was incorrect but the correct image was within the top-10 candidate set, D INO R ANK CLIP recovers the correct top-1 by exploiting either local texture (cat-on-sink), object scale (small plane), or part-level discriminator (foreground colour, ground type) that the contrastive bottleneck normally erases.
I
Case Study III — Fine-grained Confusions on FGVC-Aircraft, CUB-200, and Stanford Cars
Figure 6 shows three Fine-grained Probe failure modes that D INO R ANK CLIP rectifies. Each row presents a query image (left), the predicted class under CLIP and R ANK CLIP (which both confuse it with a visually similar but taxonomically wrong class), and the prediction under D INO R ANK CLIP (which is correct). For aircraft, the discriminating evidence is the engine layout and tail shape (tiny pixels in the image); for birds, it is the bill colour and head pattern; for cars, it is the grille shape and headlight contour. All three are exactly the kind of local part-level evidence that the DINOv3 dense Gram structure preserves and that the conflict gate α injects without disturbing the global category direction (Proposition 1). We additionally observed that D INO R ANK CLIP’s failure modes on fine-grained datasets shift in character: where CLIP fails by predicting the wrong family entirely (Boeing vs. Airbus, Cardinalidae vs. Tanagridae), D INO R ANK CLIP fails predominantly by predicting an adjacent variant within the correct family (A320 vs. A321, Northern vs. Vermilion Cardinal). This is consistent with the design hypothesis that the dense residual restores part-level information sufficient for family-level disambiguation but not always for the finest-grained intra-family distinction.
J
Case Study IV — DINOv3 Teacher vs. Student Patch Gram Heatmaps
Figure 7 visualises the patch-token Gram matrix Gi = Vi Vi⊤ for the same input image at three points in the pipeline: the frozen DINOv3 teacher (left), our ViT-Tiny student trained with the combined Gram + relational target (middle), and a baseline ViT-Tiny student trained with MSE on raw features only (right). The teacher’s Gram matrix exhibits clear block structure aligned with object parts (head, body, background); our combined-target student preserves this block structure with high fidelity, while the MSE-only baseline loses the inter-part correlations and collapses to a near-uniform Gram. 17
Query
Domain
CLIP pred
R ANK CLIP pred
D INO R ANK CLIP pred
Truth
FGVC
A330 ✗
A330 ✗
A320 ✓
A320
CUB
summer tanager ✗
scarlet tanager ✗
N. cardinal ✓
N. cardinal
Cars
A6 sedan ✗
A4 sedan ✗
A6 saloon ✓
A6 saloon
Figure 6: Case Study III: Fine-grained Probe failures on FGVC-Aircraft, CUB-200, and Stanford Cars that CLIP and R ANK CLIP both miss but D INO R ANK CLIP recovers. Discriminating evidence (engine layout, bill colour, grille shape) is local and is preserved by the DINOv3 residual injected through the conflict-aware fusion module.
eS − G e T ∥F reported in the captions (0.31 for ours vs. 0.62 for MSE) The Frobenius distances ∥G match the distillation-fidelity histogram of Figure 3c.
(a) DINOv3-ViT-L teacher. Three clear (b) ViT-Tiny student, Gram+relational blocks (head, body, background); strong (ours). Block structure preserved with e S −G e T ∥F =0.31. inter-part correlation. ∥GT ∥F = 1.00 high fidelity. ∥G (reference).
(c) ViT-Tiny student, MSE only. Block structure collapsed to a near-uniform Gram; inter-part correlations lost. e S −G e T ∥F =0.62. ∥G
Figure 7: Case Study IV: L×L patch-token Gram matrix Gi = Vi Vi⊤ for the same input image (CC3M bird sample) under three encoders. Colour intensity proportional to inner-product magnitude after row-normalisation. The combined Gram + relational target preserves the teacher’s block structure (object parts as diagonal blocks, anti-correlated background as off-diagonal pale region), while the MSE-only baseline collapses to a near-uniform matrix and loses every inter-part discriminative cue. This is the structural property that the conflict-aware fusion module then injects into the contrastive trunk.
18