Advancing Relevance Measurement with Vision-Language Models for Web-Scale Search

arXiv:2608.02446v1 [cs.IR] 3 Aug 2026

Advancing Relevance Measurement with Vision–Language Models for Web-Scale Search Han Wang∗

Alex Whitworth∗

Pak Ming Cheung

Pinterest San Francisco, CA, USA hanwang@pinterest.com

Pinterest San Francisco, CA, USA awhitworth@pinterest.com

Pinterest San Francisco, CA, USA pcheung@pinterest.com

Zhenjie Zhang

Krishna Kamath†

Xi Chen

Pinterest San Francisco, CA, USA zhenjiezhang@pinterest.com

Pinterest San Francisco, CA, USA krishna.kamath@gmail.com

Pinterest San Francisco, CA, USA xichen@pinterest.com

Roberto Konow

Kurchi Subhra Hazra

Pinterest San Francisco, CA, USA rkonow@pinterest.com

Pinterest San Francisco, CA, USA ksubhrahazra@pinterest.com

Abstract Relevance evaluation plays a crucial role in personalized search systems, serving as a guardrail alongside user engagement metrics to ensure that search results align with user queries and intent. While human annotation is the traditional method for relevance evaluation, its high cost and long turnaround time limit its scalability. In this work, we present a VLM-based automated relevance evaluation pipeline deployed within Pinterest Search for online A/B experiments. We rigorously validate the alignment between VLM-generated judgments and human annotations, demonstrating that VLMs can provide reliable relevance measurement for experiments while greatly improving the evaluation efficiency. Leveraging VLM-based labeling further unlocks opportunities to expand the query set, optimize sampling design, and efficiently assess a wider range of search experiences at scale. This approach leads to higherquality relevance metrics and significantly reduces the Minimum Detectable Effects (MDEs) in online experiment measurements.

CCS Concepts • Information systems → Retrieval models and ranking; Web searching and information discovery.

Keywords Search Recommendation Systems, Vision-Language Models, Relevance Measurement ∗ Equal contribution † Work done at Pinterest

This work is licensed under a Creative Commons Attribution 4.0 International License. RecSys ’26, Minneapolis, MN, USA © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2284-4/2026/09 https://doi.org/10.1145/3773078.3831891

ACM Reference Format: Han Wang, Alex Whitworth, Pak Ming Cheung, Zhenjie Zhang, Krishna Kamath, Xi Chen, Roberto Konow, and Kurchi Subhra Hazra. 2026. Advancing Relevance Measurement with Vision–Language Models for Web-Scale Search. In 20th ACM Conference on Recommender Systems (RecSys ’26), September 27-October 02, 2026, Minneapolis, MN, USA. ACM, New York, NY, USA, 6 pages. https://doi.org/10.1145/3773078.3831891

1

Introduction

Search relevance measures how well search results align with a user’s query. For personalized search systems, evaluating relevance is crucial to ensure that displayed results are pertinent to the user’s information needs, rather than over-relying on prior engagement. Online A/B experiments are widely used in industry to measure the impact of ranking changes. For search systems, it is essential to track changes in semantic relevance alongside engagement metrics. Engagement metrics can be collected from real-time user actions (e.g., click, save), but user engagement does not always reflect the semantic relevance of the search feed: position, presentation, and attention effects all influence user actions independently of content quality. A personalization change may therefore drive strong engagement gains while introducing irrelevant content into top slots, degrading overall search relevance. Relevance evaluation therefore serves as a guardrail to detect such tradeoffs. Relevance evaluation typically relies on human annotations. However, this approach suffers from high costs, long turnaround time, and limited scalability, which can ultimately compromise the quality of relevance metrics. In recent years, the emergence of Large Language Models (LLMs) has sparked growing interest in automating relevance assessment in the Information Retrieval (IR) community. There have been several works exploring the feasibility of using LLMs to supplement laborious human labeling efforts [8, 21, 24]. Extending LLM-based evaluation to multimodal settings, recent advances in Vision-Language Models (VLMs) have enabled native, long-context multimodal reasoning [3, 20], making them well-suited for automated relevance evaluation [28].

RecSys ’26, September 27-October 02, 2026, Minneapolis, MN, USA

At Pinterest, search is one of the key surfaces where users discover content that meets their information needs. Beyond user engagement metrics, we also track semantic relevance of top slots to ensure a high-quality search experience. In this paper, we present our methodology for leveraging VLMs to assess semantic relevance in online A/B experiments at Pinterest Search. To address the multimodal nature of Pinterest content, we fine-tune open-source VLMs on relevance prediction tasks using human-annotated labels, then apply the fine-tuned VLMs to label search results in online A/B experiments.1 This approach not only significantly reduces labeling costs and achieves over 20× improvement in labeling turnaround time, but also improves metric quality by scaling up query sets and refining sampling design. Since launch, the system has scaled to more than 4× the relevance measurement jobs, enabling broader and more frequent relevance evaluation across Pinterest Search. The primary contribution of this work lies in the end-to-end design and deployment of a VLM-based relevance evaluation pipeline for live A/B experimentation, rather than in a novel modeling architecture. To the best of our knowledge, this is the first work to comprehensively address this problem in an industry search system. We hope this work offers practical insights for industry practitioners seeking to improve the efficiency of relevance measurement. Our contributions are as follows: • We present an end-to-end pipeline for VLM-based relevance evaluation in a production search system, covering practical considerations and validation steps before deploying automated relevance labeling at scale. • We highlight that VLM-based relevance assessment enables expanding the query set and improving the query sampling design, which leads to higher-quality relevance metrics and a 6× reduction in Minimum Detectable Effect (MDE), substantially improving sensitivity for online experiment evaluation. • We demonstrate that fine-tuned VLMs produce high-quality relevance metrics closely aligned with human judgments. The mean error in the query-level 𝑠𝐷𝐶𝐺@𝐾 metric (defined in Section 3.4) remains within 0.03, and paired differences exhibit even lower bias, confirming robustness for A/B experiment evaluation.

2

Related Work

Recent studies have shown the potential of using LLMs for automating relevance assessments in Information Retrieval (IR), as part of the broader “LLM-as-a-judge” paradigm [11, 30]. Thomas et al. [21] and Upadhyay et al. [24] demonstrated that LLMs can accurately predict searcher preferences and generate human-quality relevance labels. A significant line of work investigates prompting techniques for LLM-based relevance labeling, including zero-shot [2, 23, 24] and few-shot [13, 17] approaches. Beyond text-only settings, Yang and Lin [28] benchmarked VLMs for zero-shot relevance evaluation on image-text retrieval tasks. More recently, Meng et al. [14] and Abbasiantaeb et al. [1] showed that fine-tuning open-source LLMs on human-labeled relevance judgments yields more reliable predictions than few-shot prompting with significantly larger models. 1 A preliminary, non-archival version of this work appeared as a workshop paper

[26], which used a text-only model. The present paper extends it with a VLM-based multimodal pipeline, revises and expands the reported results, and includes additional ablation studies.

Wang et al.

Unlike prior work that primarily compares LLMs/VLMs and prompting strategies [1, 14] or explores human-LLM collaboration [19], our work presents a real-world deployment of VLM-based relevance measurement within an industry-scale A/B experimentation pipeline at Pinterest Search. While VLMs have been applied to enhance multimodal representations for retrieval and ranking [9, 29], our work addresses the complementary problem of automated relevance evaluation.

3 Methodology 3.1 Problem Statement At Pinterest, we measure the semantic relevance between search queries and Pins2 using a 5-level rating scale: Highly Relevant (L5), Relevant (L4), Marginally Relevant (L3), Irrelevant (L2), and Highly Irrelevant (L1). In online experiments, assessing relevance across experimental groups is essential for detecting the impact of ranking changes on top-slot relevance, beyond what engagement metrics alone can reveal. The relevance measurement has historically been limited by the low availability of human labels and the high per-label cost. This led to measurement designs and sample sizes that could only detect large topline metric movements, but were insufficient to measure heterogeneous treatment effects or small topline effects. Next, we describe how we leverage VLMs to scale relevance labeling and address these measurement bottlenecks.

3.2

Fine-tuned VLM as Relevance Model

We fine-tune the open-source Qwen3-VL models [3] on humanannotated query-Pin pairs to optimize relevance prediction performance. The training dataset contains approximately 0.8 million human-annotated pairs, with model selection performed on a separate held-out set of approximately 20K pairs. To support search queries across multiple languages, we leverage the cross-lingual transfer capabilities of multilingual Qwen3-VL. We prompt the model to generate a relevance score from "1" to "5" following the 5-level rating guideline (Section 3.1), minimizing cross-entropy loss during training. At inference, we take the argmax over the logits of output tokens "1" through "5" as the predicted relevance level. Beyond the Pin image, we leverage a comprehensive set of textual metadata to represent each Pin: title and description, landing page title and description, titles of user-curated boards the Pin has been saved to, and historically highly-engaged queries for the Pin. Although derived from engagement signals, these queries are aggregated across all users and thus serve as complementary content annotations. Together, these textual features and the Pin image form a rich multimodal representation for relevance prediction, as illustrated in Figure 1.

3.3

Stratified Sampling Design

VLM labeling significantly reduces both the cost and turnaround time of relevance labeling, enabling much larger sampling designs. We therefore propose a stratified query sampling design that enables measurement of heterogeneous treatment effects and reduces MDEs 2 Pins on Pinterest are rich multimedia entities that feature images, videos, and other

content, often linked to external webpages or blogs.

Advancing Relevance Measurement with Vision–Language Models for Web-Scale Search

Predicted label

… Vision-Language Model

Pin Image

title / description landing page title / description board titles highly-engaged queries

Figure 1: The fine-tuned VLM takes query text, Pin text metadata, and Pin image as input, and outputs a graded relevance label on a 5-point scale via argmax over the output logits.

by 6 times. Prior to VLM labeling, stratified query sampling with human annotations was impractical, as representing each stratum adequately required a large number of queries. Stratification plays an important role in sampling-based measurement. First, stratification ensures the sample population is representative of the whole population. In addition, when the strata are chosen such that stratum means are heterogeneous, variance reduction can be achieved [15]. Formally, let the relevance score be a random variable 𝑌 with mean 𝜇 and variance 𝜎 2 . The variance of the sample mean 𝑌¯ is 𝑉 (𝑌¯ ) = 𝜎 2 /𝑁 , where 𝑁 is the total sample size. When 𝐾 strata are defined such that each stratum has a different mean, 𝑉 (𝑌¯ ) can be decomposed into within-strata and between-strata variance (see Equation 1). Stratified sampling eliminates the between-strata term [27], yielding: 𝑉 (𝑌¯𝑆𝑅𝑆 ) =

𝐾 ∑︁ 𝑛

𝑘

𝜎𝑘2 +

𝑁 | {z } 𝑘=1

within-strata

𝐾 ∑︁ 𝑛

𝑘

(𝜇𝑘 − 𝜇) 2 ≥

𝐾 ∑︁ 𝑛

𝑁 |

𝑘

𝑁

𝑘=1

𝜎𝑘2 = 𝑉 (𝑌¯𝑠𝑡𝑟𝑎𝑡 ),

𝑘=1

{z

would compromise the generalizability of resulting experimental estimates. Even where such observations exist, their utility is limited: Pinterest is a highly dynamic system in which engagement signals and inventory evolve continuously, making pre-treatment scores a weak proxy for current relevance. Finally, labeling repeated queries would double annotation cost. Stratified sampling avoids all three issues while targeting the dominant variance structure directly.

3.4

Pin Text

Query Text

RecSys ’26, September 27-October 02, 2026, Minneapolis, MN, USA

}

between-strata

(1) where 𝑛𝑘 , 𝜇𝑘 , 𝜎𝑘2 denote the sample size, mean, and variance of each stratum. The resulting variance reduction directly translates to reduced MDEs. The primary source of variance in our relevance measurement system is between-query variation: strata defined on query characteristics show large mean differences driven by query intent, content quality, and inventory depth. Stratification directly targets this variance structure, yielding design effects (DEFF) well below 1. To determine the query strata, we evaluated multiple stratification choices. While BERTopic [10] achieved the largest variance reduction, we adopted an in-house query-to-interest model based on DistilBERT [18] for its ease of integration with existing systems and scalability. For completeness, we note why CUPED [7] is less suitable here. CUPED requires pre-treatment covariate observations, which in this context would mean restricting the sampling frame to queryuser pairs with recent repeated queries, a substantial exclusion that

Relevance Measurement with VLMs

To measure the relevance impact of an A/B experiment on search ranking, we take a stratified sample of paired search queries from control and treatment experiment groups3 , ensuring that the sample is representative of the overall query distribution [22]. The use of paired samples blocks between-query differences, an important source of variation in experiment measurement. For each query in our paired sample, we retain the top 𝐾 search results and generate VLM-based relevance labels. We then compute 𝑠𝐷𝐶𝐺@𝐾 for each query and aggregate query-level metrics to derive experiment-level topline metrics. The 𝑠𝐷𝐶𝐺@𝐾 metric is a variant of the standard 𝑛𝐷𝐶𝐺@𝐾, where we assume an infinite supply of highly relevant (L5) documents (see Equation 2). We use 𝐾 = 25 throughout our evaluation. Í𝐾 𝑙𝑘 /log2 (1 + 𝑘) 𝑠𝐷𝐶𝐺@𝐾 = Í𝑘=1 , 𝑙𝑘 ∈ {1, 2, 3, 4, 5}. (2) 𝐾 𝑘=1 5/log2 (1 + 𝑘) Finally, we calculate heterogeneous treatment effects stratified by query popularity and query interest category (e.g., beauty, women’s fashion, art), applying a Benjamini-Hochberg procedure [4] to control the false discovery rate. The full VLM-based relevance measurement procedure is illustrated in Figure 2.

4

Results

We fine-tune the open-source Qwen3-VL-4B model for automated relevance evaluation. The fine-tuned model labels hundreds of thousands of query-Pin pairs within two hours on a single A100 GPU, achieving over 20× improvement in labeling turnaround time compared to human annotation. We evaluate the fine-tuned model across three research questions (RQs): • RQ1: Can VLM-based relevance assessment provide metrics that reliably align with human judgments for online A/B experiment measurements? • RQ2: Does VLM-based relevance assessment offer greater metric sensitivity compared to human labeling? • RQ3: Is multilingual VLM-based relevance assessment effective for non-English queries?

4.1

RQ1: Alignment with Human Labels

We validate the metrics derived from VLM labeling by comparing against human labels using query-Pin pairs from live experiment traffic. The human labels were collected by instructing annotators to rate the relevance of each query-Pin pair using the same 5-level rating scale. The exact match rate between VLM and human labels is 82.9%, with 94.2% of ratings differing by no more than one point, 3 Excluding queries that appear in only one experimental group is a deliberate restric-

tion of the target population, not a structural bias toward high-volume queries.

RecSys ’26, September 27-October 02, 2026, Minneapolis, MN, USA

Wang et al.

search query

control

Data Collection

Relevance Labeling

Metrics Calculation

search log sampling

VLM labeling

sDCG@K

stratified sample queries

search log sampling

treatment

(query, Pin)

VLM labeling

Δ, Δ, paired paired t-test t-test

sDCG@K

Figure 2: The VLM-based relevance measurement pipeline at Pinterest Search, consisting of search result sampling with stratified queries, VLM-based relevance labeling, and metric calculation for A/B experiment evaluation. which is comparable to the inter-annotator agreement. Quadratic Weighted Kappa (QWK) [6] is a chance-corrected ordinal agreement statistic that penalizes larger disagreements more heavily. Our model achieves a QWK of 0.507, indicating good alignment between VLM-generated and human relevance labels. Following established practices in the literature [1, 14], we calculate the rank-based correlation metrics, Kendall’s 𝜏 and Spearman’s 𝜌, to assess agreement between VLM and human rankings on the query-level sDCG@K metric. Overall, we achieve Kendall’s 𝜏 = 0.534 and Spearman’s 𝜌 = 0.668, indicating strong alignment between VLM and human rankings across all popularity segments.

differences, we additionally validate paired difference errors. According to Table 1, the mean paired difference error has negligible magnitude across all segments, and notably eliminates the slight positive bias observed in query-level errors, confirming that the paired experimental design is robust to VLM label bias. We visualize both error distributions in Figure 3. Both are tightly centered around 0, with paired differences exhibiting a tighter distribution, supporting the use of VLM-based labeling for reliable online A/B experiment evaluation.

Table 1: Alignment between VLM and human labels in querylevel sDCG@K for relevance evaluation across different query popularity segments in the US market.

Segment

Query-level Error Mean P10 P90

Paired Difference Error Mean P10 P90

Overall

0.021

-0.011

0.073

0.000

-0.013

0.013

Head Torso Tail Single

0.018 0.022 0.020 0.021

0.000 0.000 -0.003 -0.042

0.045 0.060 0.015 0.084

0.001 0.000 0.000 0.001

-0.012 -0.009 -0.009 -0.016

0.009 0.008 0.010 0.017

Beyond ranking alignment, we further validate the query-level sDCG@K error distribution, defined as VLM-derived sDCG@K minus human-derived sDCG@K. We focus on query-level rather than experiment-level error because each experiment is a single estimate with limited statistical power, whereas query-level evaluation substantially increases sample size and operates at the actual estimation unit. To assess the performance across query popularity levels, we categorize the queries into four popularity segments based on search volume [12]: head, torso, tail, and single4 . As shown in Table 1, the mean error is below 0.03 across all popularity segments, with P10 and P90 within [-0.1, 0.1], indicating low bias and well-bounded variance. Since A/B evaluation relies on paired metric 4 Single queries are defined as those that receive fewer than 10 searches.

Figure 3: Query-level 𝒔𝑫𝑪 𝑮@𝑲 error distribution for single group (left) and paired differences (right) in the US market relevance evaluation. We next evaluate Qwen3-VL across different model sizes and compare against a text-only XLM-RoBERTa-large baseline augmented with BLIP-generated image captions [25], to assess the accuracy-efficiency tradeoff and the value of visual inputs. Results for the US market are summarized in Table 2. Qwen3-VL-4B and Qwen3-VL-8B achieve comparable performance across all metrics, with near-identical accuracy and rank correlations within 0.002 of each other. Notably, the text-only XLM-RoBERTa model has a low mean query-level error, but its wider P10–P90 range indicates higher variance, meaning more queries where model and human judgments diverge substantially. For reliable relevance measurement, a tight error distribution is preferable. We therefore adopt Qwen3-VL-4B as our production model, as it offers strong relevance prediction quality, a tight error distribution, and lower computational cost than the 8B variant.

Advancing Relevance Measurement with Vision–Language Models for Web-Scale Search

RecSys ’26, September 27-October 02, 2026, Minneapolis, MN, USA

Table 2: Performance of VLMs and text-only models on automated relevance evaluation in the US market. Relative gains over the XLM-RoBERTa-large baseline are shown for Accuracy, QWK and rank correlation metrics. Model

QWK

0.808

0.429

0.490

0.824 (+2.0%) 0.829 (+2.6%) 0.829 (+2.6%)

0.490 (+14.2%) 0.507 (+18.2%) 0.514 (+19.8%)

0.522 (+6.5%) 0.534 (+9.0%) 0.532 (+8.6%)

XLM-RoBERTa-large (text-only) Qwen3-VL-2B Qwen3-VL-4B Qwen3-VL-8B

sDCG Rank Correlation Kendall’s 𝜏 Spearman’s 𝜌

Accuracy

Query-level Error Mean P10 P90

Paired Difference Error Mean P10 P90

0.626

0.015

-0.032

0.074

0.000

-0.015

0.015

0.658 (+5.1%) 0.668 (+6.7%) 0.670 (+7.0%)

0.019 0.021 0.018

-0.016 -0.011 -0.016

0.074 0.073 0.069

0.000 0.000 0.000

-0.013 -0.013 -0.013

0.014 0.013 0.013

Table 3: Alignment between VLM and human labels in query-level sDCG@K for relevance evaluation across France (FR), Germany (DE), and Brazil (BR) markets. sDCG Rank Correlation Kendall’s 𝜏 Spearman’s 𝜌

Country FR DE BR

4.2

0.430 0.362 0.409

0.544 0.460 0.493

RQ2: Metrics Sensitivity and MDEs

We evaluate the impact of these changes on experiment sensitivity by measuring MDEs for our experimentation system. The MDE is the smallest change in a metric that an experiment can reliably detect given the sample size, statistical power, and significance level chosen for the test. Since most online experiments produce small effect sizes, achieving small MDEs is essential for detecting meaningful improvements and maintaining experimentation velocity.

Table 4: Improvement in metric sensitivity (MDE) with proposed sampling design and estimator. Sample Type

Estimator

𝑛

𝜎ˆ

Reduction in 𝜎ˆ

SRS Stratified Stratified Stratified

SRS Stratified SRS Stratified

2000 2000 5000 5000

0.184 0.096 0.094 0.061

52% 51% 67%

Query-level Error Mean P10 P90

Paired Difference Error Mean P10 P90

0.035 0.023 0.040

-0.002 -0.004 0.000

-0.002 -0.021 0.000

0.098 0.087 0.140

0.046 0.044 0.070

assuming standard values for 𝛼 = 0.05 and 𝛽 = 0.8, √︁ (𝑧 1−𝛼/2 + 𝑧 𝛽 ) × 2𝜎ˆ 2 /𝑛 . (3) MDE = Lift%Detectable = 𝜇ˆ We can therefore decompose MDE reduction into contributions from variance reduction and sample size increase. We present these results in Table 4. The largest share of MDE reduction is attributable to variance reduction from stratification, consistent with our observation at Pinterest that most variance in relevance occurs across queries, largely due to differences in query interest and popularity. Beyond metric quality, VLM labeling also achieves over a 20-fold improvement in turnaround time and a greater than 99% reduction in per-label cost, as shown in Table 5. Table 5: Comparison of label volume, per-label cost, and turnaround time before and after adopting VLM-based labeling.

Baseline VLM Labeling

Before the introduction of VLM labeling, relevance measurement had large MDEs, typically ranging from 1.3% to 1.5%. These large MDEs were primarily the result of the constraints on our sampling designs imposed by the high cost and turnaround time of human labeling. The introduction of VLM labeling removed these constraints, enabling us to redesign our sampling approach. We increased our sample sizes, moved from simple random sampling (SRS) to stratified sampling, and adopted a stratified sampling estimator [22]. Optimal allocation [16] is used to allocate sample units to strata, which are defined as the cross-product of query interest category and popularity segment. These changes enabled us to reduce our MDEs to ≤ 0.25%. To attribute the MDE reduction to each source of change, we follow a standard derivation of MDE [5] as a function of the number ˆ and variance (𝜎ˆ 2 ), of queries in the sample (𝑛), metric mean (𝜇),

-0.051 -0.049 -0.065

4.3

Label Volume

Per-label Cost ($)

Turnaround Time

𝑂 (10𝐾) 𝑂 (100𝐾)

0.1 2 × 10−5 (-99.98%)

2 days 2 hours (-96%)

RQ3: Performance on Non-English Queries

Since the fine-tuning data consists predominantly of English queryPin pairs, careful validation is required before extending VLMbased assessment to non-English queries. For this analysis, we focus on the France (FR), Germany (DE), and Brazil (BR) markets as representative non-English markets, using query-Pin pairs from live experiment traffic for validation. The query-level metric alignment is summarized in Table 3. While Kendall’s 𝜏 and Spearman’s 𝜌 are lower than those observed for English queries (Table 1), they are generally interpreted as moderate-to-strong correlations according to existing literature [14, 28]. Notably, query-level errors show a slight positive bias; however, this bias is effectively eliminated in paired differences,

RecSys ’26, September 27-October 02, 2026, Minneapolis, MN, USA

with errors tightly concentrated around 0. These results suggest that the VLM-based relevance assessment generalizes effectively to non-English markets.

5

Conclusion and Future Work

In this work, we present a VLM-based relevance labeling framework for generating query-level relevance metrics in online A/B experiment evaluation. We demonstrate that fine-tuned VLMs achieve low bias on query-level 𝑠𝐷𝐶𝐺@𝐾 metrics and paired differences, with strong alignment to human annotations across query popularity segments. Transitioning to VLM-based relevance assessment enables us to scale up the evaluation query set and redesign the sampling strategy, yielding a 6× reduction in MDEs and significantly improved detection of relevance shifts. We have successfully deployed this approach at Pinterest Search, significantly reducing manual annotation costs and achieving over 20× improvement in labeling turnaround time while maintaining metric reliability. With the efficiency boost, the relevance measurement job volume has grown more than 4× since launch, reflecting both the efficiency of the VLM-based pipeline and the growing demand for automated relevance measurement. Future work will extend VLM-based relevance measurement to broader multimodal search settings beyond text-to-image search, and improve multilingual capabilities to close the observed performance gap on non-English queries.

Acknowledgments We would like to thank Kofi Boakye, Sheng-Min Shih, Bonnie Liu, Manas Pathak for their collaboration in developing the VLM-based relevance model; and Miguel Madera, Pedro Sanchez, Jorge Amigon, Francisco Navarrete for their contributions to the automated labeling integration.

References [1] Zahra Abbasiantaeb, Chuan Meng, Leif Azzopardi, and Mohammad Aliannejadi. 2024. Can We Use Large Language Models to Fill Relevance Judgment Holes? arXiv preprint arXiv:2405.05600 (2024). [2] Negar Arabzadeh and Charles LA Clarke. 2025. Benchmarking LLM-based Relevance Judgment Methods. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval. 3194–3204. [3] Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. 2025. Qwen3-VL Technical Report. arXiv preprint arXiv:2511.21631 (2025). [4] Yoav Benjamini and Yosef Hochberg. 1995. Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal statistical society: series B (Methodological) 57, 1 (1995), 289–300. [5] Howard S Bloom. 1995. Minimum Detectable Effects: A Simple Way to Report the Statistical Power of Experimental Designs. Evaluation Review 19, 5 (1995), 547–556. [6] Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin 70 (1968), 213–220. [7] Alex Deng, Ya Xu, Ron Kohavi, and Toby Walker. 2013. Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data. Proceedings of the Sixth ACM International Conference on Web Search and Data Mining (2013). [8] Guglielmo Faggioli, Laura Dietz, Charles LA Clarke, Gianluca Demartini, Matthias Hagen, Claudia Hauff, Noriko Kando, Evangelos Kanoulas, Martin Potthast, Benno Stein, et al. 2023. Perspectives on Large Language Models for Relevance Judgment. In Proceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval. 39–50. [9] Ramin Giahi, Kehui Yao, Sriram Kollipara, Kai Zhao, Vahid Mirjalili, Jianpeng Xu, Topojoy Biswas, Evren Korpeoglu, and Kannan Achan. 2025. VL-CLIP: Enhancing Multimodal Recommendations via Visual Grounding and LLM-Augmented CLIP Embeddings. In Proceedings of the Nineteenth ACM Conference on Recommender Systems. 482–491. [10] Maarten Grootendorst. 2022. BERTopic: Neural topic modeling with a class-based TF-IDF procedure. arXiv preprint arXiv:2203.05794 (2022).

Wang et al.

[11] Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. 2024. A Survey on LLM-as-a-Judge. arXiv preprint arXiv:2411.15594 (2024). [12] Alpa Jain and Gilad Mishne. 2010. Organizing Query Completions for Web Search. In Proceedings of the 19th ACM International Conference on Information and Knowledge Management. 1169–1178. [13] Sean MacAvaney and Luca Soldaini. 2023. One-Shot Labeling for Automatic Relevance Estimation. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2230–2235. [14] Chuan Meng, Negar Arabzadeh, Arian Askari, Mohammad Aliannejadi, and Maarten de Rijke. 2025. Query Performance Prediction Using Relevance Judgments Generated by Large Language Models. ACM Transactions on Information Systems 43, 4 (2025), 1–35. [15] Luke W Miratrix, Jasjeet S Sekhon, and Bin Yu. 2013. Adjusting Treatment Effect Estimates by Post-Stratification in Randomized Experiments. Journal of the Royal Statistical Society Series B: Statistical Methodology 75, 2 (2013), 369–396. [16] Jerzy Neyman. 1934. On the Two Different Aspects of the Representative Method: The Method of Stratified Sampling and the Method of Purposive Selection. Journal of the Royal Statistical Society 97, 4 (1934), 158–606. [17] Catarina Pires, Sérgio Nunes, and Luís Filipe Teixeira. 2025. Expanding Relevance Judgments for Medical Case-based Retrieval Task with Multimodal LLMs. arXiv preprint arXiv:2506.17782 (2025). [18] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 (2019). [19] Shreya Shankar, JD Zamfirescu-Pereira, Björn Hartmann, Aditya Parameswaran, and Ian Arawjo. 2024. Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology. 1–14. [20] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 (2024). [21] Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. 2024. Large Language Models can Accurately Predict Searcher Preferences. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1930–1940. [22] Steven K Thompson. 2012. Sampling. John Wiley & Sons, Chapter 11, 141–170. [23] Shivani Upadhyay, Ronak Pradeep, Nandan Thakur, Daniel Campos, Nick Craswell, Ian Soboroff, Hoa Trang Dang, and Jimmy Lin. 2024. A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Look. arXiv preprint arXiv:2411.08275 (2024). [24] Shivani Upadhyay, Ronak Pradeep, Nandan Thakur, Nick Craswell, and Jimmy Lin. 2024. UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor. arXiv preprint arXiv:2406.06519 (2024). [25] Han Wang, Mukuntha Narayanan Sundararaman, Onur Gungor, Yu Xu, Krishna Kamath, Rakesh Chalasani, Kurchi Subhra Hazra, and Jinfeng Rao. 2024. Improving Pinterest Search Relevance Using Large Language Models. arXiv preprint arXiv:2410.17152 (2024). [26] Han Wang, Alex Whitworth, Pak Ming Cheung, Zhenjie Zhang, and Krishna Kamath. 2025. LLM-based Relevance Assessment for Web-Scale Search Evaluation at Pinterest. arXiv preprint arXiv:2509.03764 (2025). [27] Huizhi Xie and Juliette Aurisset. 2016. Improving the Sensitivity of Online Controlled Experiments: Case Studies at Netflix. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 645– 654. [28] Jheng-Hong Yang and Jimmy Lin. 2024. Toward Automatic Relevance Judgment using Vision–Language Models for Image–Text Retrieval Evaluation. arXiv preprint arXiv:2408.01363 (2024). [29] Chengcan Ye, Ting Peng, Tim Chang, Zhiyi Zhou, and Feng Wang. 2023. Queryaware Multi-modal based Ranking Relevance in Video Search. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track. 322–330. [30] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Haotong Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. ArXiv abs/2306.05685 (2023).

Related documents

• Focus Where It Counts: A Salience-Driven Vision-Language Model for Low Vision Assistance#784969
• Focus Where It Counts: A Salience-Driven Vision-Language Model for Low Vision Assistance#783379
• From Knowing to Abstaining: Bridging the Representation-Action Gap in Vision-Language Models#1116984
• Deterministic retrieval recovers biomedical associations lost by language models#234301
• Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering#996779