OXY E COM B ENCH: Benchmarking Multimodal Foundation Models across E-Commerce Ecosystems
arXiv:2605.13173v1 [cs.DB] 13 May 2026
Yong Liu1∗, Ximan Liu1∗ † , Guoqing Yang1 , Bing Bai1 , Xiaoqiang Xu1 Zhen Chen1 , Ke Zhang1 , Yan Li1 1 JD.COM [email protected]
Abstract Large language models (LLMs) and multimodal large language models (MLLMs) have become indispensable tools across a wide range of applications. Ecommerce, however, poses distinctive challenges—including intricate domain knowledge, long-tail product evidence, heterogeneous visual data, and the interplay among multiple stakeholder roles—that diverge substantially from the general world knowledge these models are primarily trained on, often causing a notable gap between their open-domain and e-commerce performance. To systematically quantify this gap, we introduce OxyEcomBench, a unified multimodal benchmark comprising approximately 6,300 high-quality instances for real-world bilingual Chinese–English e-commerce. Although several e-commerce benchmarks have been proposed, they typically adopt a single stakeholder perspective, target a narrow set of tasks, or address isolated challenges (e.g., visual salience or multi-turn dialogues independently), making it difficult to holistically assess models’ understanding of the full e-commerce pipeline. OxyEcomBench addresses these limitations by jointly covering platform operators, merchants, and customers across 6 capability aspects and 29 tasks, supporting text-only and mixed-modality inputs with single-image, multi-image, single-turn, and multi-turn configurations. All data is sourced from authentic e-commerce platforms and verified by domain experts. The benchmark further adopts a difficulty-aware design with a four-level P0–P3 rubric applied to all 29 tasks whose difficulty admits stable expert consensus, and rigorously prioritizes visually salient multimodal cases in which key evidence resides in images rather than text alone. Evaluations on 20 mainstream LLMs and MLLMs show that even the leading models attain modest performance and that performance gaps narrow on OxyEcomBench, suggesting that insufficient e-commerce-specific knowledge infusion mutes the advantages of advanced general-purpose models in this domain.
1
Introduction
Large language models (LLMs) [1] and multimodal large language models (MLLMs) [2, 3] have demonstrated remarkable capabilities across a wide range of tasks. However, deploying them in highly specialized and complex real-world scenarios, such as e-commerce, presents distinct challenges. As a data-rich and economically significant domain, e-commerce is characterized by platform-specific policies, long-tail product knowledge, and heterogeneous visual evidence. Critically, the e-commerce ecosystem involves three primary stakeholder roles with fundamentally different needs: platform operators who manage marketplace infrastructure, policies, and user experience; * Equal contribution. † Project lead.
Preprint.
Figure 1: Taxonomy of OXY E COM B ENCH. The inner ring represents the three stakeholder roles— platform, merchant, and customer—as independent evaluation perspectives. The middle ring groups the benchmark into 6 capability aspects, and the outer ring enumerates 29 distinct e-commerce tasks, with each task inheriting the color of its corresponding capability aspect. merchants who list products, handle orders, and manage post-sale operations; and customers who search, evaluate, and purchase products. This intricate multi-role interplay, combined with the requirement for deep domain expertise, diverges substantially from the general world knowledge these models are primarily trained on, often causing a notable performance gap in real business workflows. Recent efforts have made commendable progress in benchmarking LLMs and MLLMs for ecommerce, yet they typically focus on isolated dimensions of the domain. For instance, several benchmarks predominantly adopt a single stakeholder perspective, such as consumer-oriented tasks [4] or customer support scenarios [5, 6]. Concurrently, other pioneering works have begun to explore specific technical challenges, such as multi-turn conversational interactions [7] or the necessity of visual salience in multimodal tasks. While these targeted evaluations provide valuable insights, they remain largely fragmented. Real-world e-commerce workflows rarely occur in isolation; they frequently involve complex, multi-turn dialogues coupled with visually salient evidence, spanning the diverse and intertwined needs of platform operators, merchants, and customers. Consequently, the community still lacks a holistic evaluation framework that captures the full complexity of the e-commerce ecosystem. To the best of our knowledge, existing benchmarks largely fall short of simultaneously addressing the distinct evaluation requirements of multiple stakeholder roles within a unified framework. Furthermore, many existing datasets lack fine-grained, expertcalibrated difficulty annotations, making it challenging to determine whether a model’s success stems from genuine multi-step reasoning and cross-modal integration, or merely from solving simplistic, surface-level queries. To systematically quantify the gap between open-domain capabilities and specialized e-commerce requirements, we present OXY E COM B ENCH. Unlike previous datasets that target isolated challenges, OXY E COM B ENCH is designed to holistically assess models’ understanding of the full e-commerce 2
pipeline. As summarized in Figure 1, the benchmark organizes 3 stakeholder roles, 6 capability aspects, and 29 tasks into a unified taxonomy for real-world e-commerce evaluation. It comprises approximately 6,300 high-quality, bilingual Chinese–English instances spanning these capability aspects and tasks, with each instance manually inspected by e-commerce experts. The dataset rigorously prioritizes visually salient multimodal cases and authentic multi-turn interactions, with all data sourced from real-world e-commerce platforms to ensure ecological validity. Our main contributions are as follows: • Comprehensive Multi-Role Evaluation: We introduce OXY E COM B ENCH, the first benchmark to holistically integrate platform, merchant, and customer perspectives, enabling a complete assessment of foundation models across the entire e-commerce ecosystem. • Rigorous Multimodal & Interactive Assessment: By curating visually salient tasks and authentic multi-turn dialogues, the benchmark effectively prevents text-only shortcuts and evaluates models on complex, real-world multimodal reasoning. In addition, we establish a reliable fourlevel (P0–P3) difficulty rubric based on expert consensus, providing a nuanced tool to differentiate between surface-level pattern matching and deep cognitive reasoning. • Domain-Specific Empirical Insights: Extensive evaluations on mainstream LLMs and MLLMs show that even leading models attain modest performance and that performance gaps narrow on OXY E COM B ENCH, indicating that insufficient e-commerce-specific knowledge infusion can mute the advantages of advanced general-purpose models.
2
Related Work
2.1
General Multimodal Benchmarks
The rapid progress of multimodal large language models (MLLMs) has been mirrored by a rich landscape of evaluation suites, ranging from early perception and single-hop reasoning benchmarks (VQA [8], COCO Captions [9], OK-VQA [10], TextVQA [11]) and comprehensive multi-task suites (MME [12], MMBench [13], MM-Vet [14], TouchStone [15], LVLM-eHub [16], LAMM [17]), to expert-level reasoning tests (ScienceQA [18], MathVista [19], MMMU [20], CMMU [21], ChartQA [22]), multi-image and interleaved suites (BLINK [23], Mantis [24], MUIRBench [25]), and reliability-oriented diagnostics targeting hallucination and visual salience (POPE [26], HallusionBench [27], SEED-Bench [28], MMStar [29], MMHal-Bench [30], AMBER [31]). While these general-domain benchmarks offer broad coverage of world knowledge, multi-image reasoning, and visual dependency, they lack the specialized domain expertise—platform-specific policies, long-tail product attributes, and multi-role interactions—required for industrial e-commerce, so strong general-benchmark performance does not necessarily translate into reliability on real-world e-commerce scenarios. 2.2
E-commerce Specific Benchmarks
Recognizing the limits of general-domain evaluations, the community has introduced benchmarks tailored to the e-commerce domain. However, these pioneering efforts typically focus on isolated dimensions of the ecosystem, leaving significant gaps in holistic evaluation. Concept Understanding and Text-centric Interactions. Early e-commerce benchmarks predominantly focused on text-based concept comprehension and single-turn interactions. For instance, Shopping MMLU [4] introduced a comprehensive suite for evaluating consumer-oriented tasks, such as concept understanding and purchase behavior alignment. Similarly, ChineseEcomQA [32] focused on foundational e-commerce concept QA. To evaluate interactive capabilities, EComBench [5] assessed LLM agents within customer support scenarios, while Mix-Ecom [33] explored mixed-type dialogues and complex domain rules, revealing that current agents struggle with hallucination when navigating intricate e-commerce policies. While valuable, these benchmarks are largely text-centric and overlook the visually rich reality of online shopping. Multimodal Explorations in E-commerce. Recent works have begun incorporating visual modalities into e-commerce evaluation. EcomEval [6] integrates a multimodal subset to evaluate basic vision-language alignment. EcomMMMU [34] highlights the necessity of visually salient subsets, showing that product images can sometimes introduce redundancy rather than useful signals. Other targeted benchmarks evaluate multimodal representation learning (MBE [35]), missing modality 3
Table 1: Comparison of OXY E COM B ENCH with representative e-commerce benchmarks. Roles: P=Platform operator, M=Merchant, C=Customer. Mod.: T=Text-only, T+V=both text and visual variants. M-Turn: native multi-turn dialogue support. M-Img: multi-image inputs. V.Sal.: deliberate inclusion and emphasis of visually salient tasks to evaluate genuine visual understanding. Diff.: explicit difficulty annotation. Reliability characterizes data trustworthiness: Real indicates task instances are directly sourced from authentic real-world e-commerce platforms (rather than crowd-sourced, reformulated, or LLM-generated); Verif. indicates rigorous human verification by e-commerce domain experts. “−” denotes that the property is not reported by the original paper. Benchmark
# Tasks # Inst.
Roles
Capabilities
Reliability
Lang.
P M C Mod. M-Turn M-Img V.Sal. Diff. Real Verif. WebShop [39] Shopping MMLU [4] ChineseEcomQA [32] ECom-Bench [5] EcomMMMU [34] EcomEval [6] Mix-ECom [33] EcomBench [37] RAIR [7] EComStage [38]
1 12K 7 7 3 T+V 57 20.8K 7 7 3 T 10 1.8K 7 7 3 T 53 - 7 7 3 T+V 8 406K 7 7 3 T+V 37 3.1K 7 3 3 T+V 4 4.8K 7 7 3 T+V 7 − 7 3 3 T+V 14 49k 7 7 3 T+V 7 4.8K 7 3 3 T+V
3 7 7 3 7 7 3 3 7 3
7 7 7 7 3 7 3 7 7 7
7 7 7 7 3 7 7 7 3 7
7 7 7 7 7 3 7 3 3 7
3 3 7 3 3 3 3 3 3 3
7 7 3 3 7 3 3 3 3 3
EN Multi ZH ZH EN 7 langs ZH EN ZH ZH+EN
OXY E COM B ENCH (Ours)
28
6.3K 3 3 3 T+V
3
3
3
3
3
3
ZH+EN
completion (MMPCBench [36]), tool-use in multimodal environments (EcomBench [37]), visually salient relevance assessment (RAIR [7]), and stage-wise evaluation protocols (EComStage [38]). Despite these valuable targeted explorations, existing benchmarks remain fragmented, which motivates OXY E COM B ENCH to provide a unified, full-role, and visually salient evaluation suite. Despite these commendable advancements, the existing landscape remains fragmented. As summarized in Table 1, most benchmarks adopt a single stakeholder perspective (e.g., consumer-oriented or customer-support) and fail to integrate the distinct requirements of platform operators, merchants, and customers; few simultaneously enforce visual salience, multi-turn interactions, and expertcalibrated difficulty within a bilingual setting. OXY E COM B ENCH bridges this critical gap. By combining multi-role coverage, emphasis on visually salient tasks (preventing text-only shortcuts from dominating), authentic multi-turn interactions, and expert-calibrated difficulty levels within a bilingual framework, OXY E COM B ENCH provides the first holistic testbed for evaluating foundation models across the entire e-commerce ecosystem.
3
OXY E COM B ENCH
OXY E COM B ENCH is a unified bilingual multimodal benchmark for evaluating foundation models in real-world e-commerce. It comprises approximately 6,300 high-quality Chinese–English instances across 29 tasks, 6 capability aspects, and 3 stakeholder roles—platform operators, merchants, and customers—covering text-only, single-image, multi-image, single-turn, and multi-turn settings. All instances are sourced from authentic e-commerce platforms and assigned deterministic task-level P0–P3 difficulty labels. Their reference answers, derived from a hybrid of authentic human feedback and high-quality model generations, undergo systematic verification by powerful models and senior domain experts to ensure accuracy, reliability, and ecological validity. 3.1
Design Principles
The design of OXY E COM B ENCH is guided by three core principles: Full-role coverage. E-commerce ecosystems involve three distinct stakeholder roles—platform operators (governing marketplace policy and content quality), merchants (handling product listing, quality inspection, and operational decisions), and customers (searching, consulting, and purchasing). OXY E COM B ENCH jointly covers all three roles within a unified evaluation suite, enabling holistic assessment across the full e-commerce workflow rather than optimization for a single perspective as some previous works. 4
Text-Only Tasks Question： Determine whether this user review is malicious or unhelpful. Answer directly with 'Yes' or 'No'. [User Review]: Wow, I thought there was only one flavor, I didn't expect there to be a passion fruit flavor too. It's a pity that they no longer offer free shipping and the price has increased so much.
Visually Salient Tasks User-provided images of Product 1
Detailed platform sales image for Product 2
Answer：No Question： Please determine the user's sentiment toward the product based on the user review. A: Positive, B: Negative, C: Neutral. [User Review]: The flavors are quite diverse and good, and my cat loves it. However, the portion size of the packaging is too small. It would be best if they released a bottled version. Answer：A
Question： Based on the image information above, determine whether Product 1 and Product 2 are the same product. A. Not the same product B. The same product Answer：B
Text-Driven Tasks
产品名称：太阳镜盒便携男 抗压户外运动拉链新款⼥学 ⽣墨镜包 (Product Name: Unisex Portable Sunglasses Case, Crush-Resistant Outdoor Sports Zipper New Style, New Student-Style Sunglasses Pouch)
产品标题：润培roopy润培⾝体乳便携式⼥补⽔保湿通⽤⼩⽀秋冬季不油腻 滋润保湿 融雪之时套盒装
Question: 这个商品属于何种品类? 请找 出正确的答案。 A: 收纳⽤品 > 收纳盒 > ⾸饰盒 B: 收纳⽤品 > 收纳盒 > 眼镜 收纳盒 C: 男⼥鞋包 > ⼥包 > ⼿提包 D: 收纳⽤品 > 收纳柜 > ⾐物收纳柜 (Question: Which category does this product belong to? Please find the correct answer. A: Storage Supplies > Storage Boxes > Jewelry Box B: Storage Supplies > Storage Boxes > Glasses Storage Box C: Men's & Women's Shoes and Bags > Women's Bags > Handbag D: Storage Supplies > Storage Cabinets > Clothing Storage Cabinet)
Question：请结合图⽚与标题信息，判断标题与商品图⽚之间是否正确匹配。从下⾯三个选项中选择 最合理的判断结果： A. 正确 B. 错误。产品类型描述不⼀致：标题中描述为“⾝体乳”，但图⽚中商 品实际为“护⼿霜”。C. 错误。品牌描述不⼀致：标题中品牌为"润培"，但图⽚中商品品牌实际为"安 达"。 请直接输出选项字⺟。 (Question: Based on the image and title information, please determine whether the title correctly matches the product image. Select the most appropriate judgment from the following three options: A. Correct B. Incorrect. Inconsistent product type description: the title describes it as 'body lotion', but the actual product in the image is 'hand cream'. C. Incorrect. Inconsistent brand description: the brand in the title is ‘Roopy', but the actual brand of the product in the image is 'Anda'. Please directly output the option letter. )
Answer：B
Answer：B
(Product Name: Roopy Body Lotion Portable Women's Hydrating Moisturizing Universal Small Tube Autumn Winter Non-greasy Nourishing Moisturizing "Time of Snow Melting" Box Set)
Figure 2: Illustration of three types of tasks in OXY E COM B ENCH. Text-Only Tasks: instances that rely purely on textual information, such as review classification or sentiment analysis. Text-Driven Tasks: multimodal tasks where the answer can be derived primarily from textual cues. Visually Salient Tasks: cases where fine-grained visual evidence (e.g., matching product details across different environments) is indispensable for the correct answer.
Visual salience. Images serve as one of the most critical information carriers in e-commerce scenarios. To rigorously evaluate models’ understanding of visual content, we treat a multimodal task as visually salient when correct responses depend on visual attributes (appearance, fine-grained defects, or cross-image correspondence) rather than text alone. By curating a rich collection of visually salient instances, OXY E COM B ENCH ensures that models must genuinely process and integrate visual evidence to succeed, preventing text-only shortcuts from dominating and inflating multimodal benchmark scores [34, 7]. Figure 2 illustrates our three task modalities, highlighting our focus on visually salient tasks where fine-grained visual evidence is indispensable for correct reasoning. Difficulty-aware categorization. A four-level rubric (P0–P3, from lowest to highest difficulty) classifies tasks across three dimensions: cognitive complexity, required information integration, and domain knowledge depth. P0 (Fundamental Understanding & Explicit Extraction) requires basic single-modality comprehension and explicit information extraction; P1 (Intermediate Analysis & Routine Reasoning) demands context integration and routine domain reasoning; P2 (Advanced Reasoning & Fine-grained Comparison) necessitates fine-grained cross-modal comparison, multi-image reasoning, implicit intention mining, and sustained context tracking in multi-turn dialogues; and P3 (Expert-Level Judgment & Comprehensive Decision-Making) involves deep domain knowledge, complex policy application, and comprehensive multi-step decisions. To maintain consistency and rigorous evaluation, all 29 tasks receive deterministic task-level difficulty labels, which were determined by multiple e-commerce domain experts through comprehensive analysis of actual instances and consensus voting. This rigorous process preserves cross-task comparability and label reliability. The specific difficulty level assigned to each task is detailed in Table 2. 3.2
Task Taxonomy
OXY E COM B ENCH organizes 29 tasks across 6 capability aspects, reflecting the intertwined needs of the three stakeholder roles (Table 2). In task names, UGC and MGC denote user- and merchantgenerated content, respectively. The tasks test core abilities: Compliance & Quality Control examines rule following, inspection, and image-text consistency; Review Analysis evaluates opinion understanding, summarization, and response generation; Product Understanding covers product identity, attributes, relevance, and selling-point extraction; Domain-Specific Knowledge tests e-commerce Q&A and product classification; User Behavior Insight focuses on real-world recogni5
Table 2: Task taxonomy of OXY E COM B ENCH. The Role column lists the three stakeholder perspectives without horizontal subdivision, while horizontal rules start from the Aspect column to separate capability aspects. Mod.: T = Text-only, T+I = Text + Single Image, T+MI = Text + Multiple Images. The multimodal (T+I or T+MI) tasks emphasize visually salient data to evaluate genuine visual understanding. Role
Platform
Aspect
Tasks
# Samples
Mod.
Difficulty
Compliance & Quality Control
Product Compliance Verification Ad Creative Fidelity Diagnosis Product Damage Detection Address Verification Creative Content Quality Assessment MGC-to-Product Consistency Verification
199 160 177 100 160 200
T+I T+MI T+I T T+MI T+MI
P2 P2 P1 P1 P2 P3
Review Analysis
Review Sentiment Analysis Review Satisfaction Rating SKU-Level Summarization Malicious & Invalid Review Detection Review Response Generation
429 430 200 245 59
T T T T T+MI
P1 P1 P1 P0 P2
Product Understanding
Same-Item Matching Product Entity Captioning Product Attribute Q&A Highlights Understanding UGC-to-Product Relevance Assessment Selling Point Extraction
238 220 150 168 296 214
T+MI T+MI T+MI T+I T+MI T+MI
P2 P2 P2 P2 P1 P1
Domain-Specific Knowledge
E-commerce Knowledge Q&A (Single-turn) E-commerce Knowledge Q&A (Multi-turn) Product Category Classification
589 233 112
T T T+MI
P0 P2 P1
User Behavior Insight
UGC Visual Entity Captioning UGC-to-Product Matching Query Rewrite Quality Assessment Claim Intent Detection
212 287 173 200
T+I T+MI T T
P2 P2 P1 P1
Personalization Analysis
Target Audience Analysis Purchase Intention Prediction (Text) Purchase Intention Prediction (Multi-modal) Complementary Product Recommendation Price Range Prediction
200 176 139 155 193
T+MI T T+MI T+MI T+MI
P1 P3 P3 P1 P2
Merchant
Customer
tion, matching, query rewriting, and intent detection; and Personalization Analysis evaluates audience analysis, purchase prediction, recommendation, and pricing assessment. To comprehensively probe these capabilities, OXY E COM B ENCH accommodates diverse task formats. Question types include single-choice, multiple-choice, true/false, and open-ended Q&A. Dialogue formats include single-turn tasks and multi-turn tasks that simulate conversational interactions (e.g., customer service), evaluating the model’s ability to maintain context across exchanges. Modality configurations span text-only, single-image, and multi-image inputs. Our multimodal tasks emphasize visual salience as described in Section 3.1. Furthermore, as illustrated in Figure 3, the per-task average input-token length spans nearly two orders of magnitude—from short, query-style inputs of around 50 tokens to context-heavy inputs that approach 8,500 tokens. While modern foundation models can process significantly longer sequences in general domains (e.g., summarizing 100K-token books), 8,500 tokens in e-commerce represents a highly information-dense, longcontext scenario. It typically encompasses extensive multi-turn customer service histories with complex coreference, or intricate platform policy documents requiring strict logical adherence. Such breadth lets OXY E COM B ENCH jointly probe short-context understanding, mid-range comprehension, and e-commerce-specific long-context reasoning within a single benchmark. 3.3
Data Construction
To guarantee high data quality and ecological validity, the construction of OXY E COM B ENCH follows a systematic three-stage pipeline designed to source authentic data, challenge models via hard negative mining, and ensure reliability. Stage 1: Real-World Data Collection. We collect raw data from diverse business scenarios within a large-scale e-commerce platform, including advertising, transactions, search, recommendation, 6
Figure 3: Per-task average input-token length of OXY E COM B ENCH (log scale). Bars are grouped by capability aspect (top band); task names use the abbreviations defined in Table 2. Average inputs span nearly two orders of magnitude—from ≈50 tokens for short query-style tasks (e.g., QRQA) to ≈8,500 tokens for reasoning- and dialogue-intensive tasks (e.g., EKMQA)—enabling joint evaluation of short-, mid-, and e-commerce-specific long-context capabilities.
and compliance. The corpus encompasses heterogeneous modalities: product listings with single/multiple images, customer reviews, multi-turn dialogue logs, and complex policy documents. All data undergoes strict anonymization to remove PII and sensitive business metrics. Stage 2: Data Processing & Annotation. After filtering noisy or incomplete records, we design task-specific questions and interaction formats. For instances with natural ground truth, we construct hard negative samples to mimic realistic pitfalls. For example, in product understanding, we subtly alter textual attributes to misalign with the product; in matching tasks, we substitute images with highly similar but distinct products. For instances lacking explicit ground truth, we employ a collaborative paradigm where initial model-generated annotations are refined by human annotators following detailed guidelines. Stage 3: Quality Assurance. We implement a dual-verification mechanism. First, state-of-the-art foundation models (e.g., GPT-5.4 [40], Gemini 3.1 Pro [41]) automatically inspect the rationality of questions and answers against predefined rules. Finally, senior e-commerce domain experts conduct a comprehensive manual review to verify factual correctness, policy alignment, and absence of ambiguity, thereby guaranteeing the ultimate accuracy of the evaluation instances.
4
Experiments
4.1
Experimental Setup
Evaluated models. We benchmark 20 representative LLMs and MLLMs that span a diverse landscape of architectures, scales, and openness. Proprietary models include Gemini 3 Pro and Gemini 3 Flash, GPT-5.2 and GPT-4o, and Seed 2.0 Pro. Open-source / open-weight models include Kimi 2.5; Qwen3.5-397B-A17B, Qwen3.5-35B, Qwen3.5-27B, Qwen3.5-9B, and Qwen3.5-2B; Qwen3-VL-32B and Qwen3-VL-8B; InternVL3.5-8B; GLM-5.1 and GLM-5.0; DeepSeek-V4-Flash; Llama3.3-70B, Llama3.1-8B, and Llama3.2-11B-Vision. This suite jointly covers multimodal models (Gemini 3 series, GPT-4o/5.2, Seed 2.0 Pro, Kimi 2.5, Qwen3.5 series, Qwen3-VL series, InternVL3.5, and Llama3.2-Vision) and text-only models (GLM-5.1, GLM-5.0, Llama3.3, Llama3.1, and DeepSeek-V4-Flash), enabling controlled analysis of multimodal capability gaps. For text-only models on multimodal tasks, cells are left blank when no caption-fallback result is provided in the source spreadsheet; aggregate scores are computed over the available task results. Evaluation metrics. We adopt task-specific metrics following standard practices. Objective formats (true/false, single-/multiple-choice) are evaluated using Accuracy. For open-ended Q&A, we employ an LLM-as-judge protocol, utilizing a strong third-party model to assess responses based 7
Table 3: Main results on OXY E COM B ENCH across the six capability aspects. C&QC: Compliance & Quality Control; RA: Review Analysis; PU: Product Understanding; DK: Domain-Specific Knowledge; UBI: User Behavior Insight; PA: Personalization Analysis. Per-capability and overall scores are macro averages over the corresponding non-missing task scores in Table 4. Scores for text-only Large Language Models (marked with ∗ ) are computed only on text-only tasks and are therefore not directly comparable to Vision-Language Models with full multimodal task coverage. Bold denotes the best score and underlined denotes the second-best score within each column. Model
C&QC
RA
PU
DK
UBI
PA
Overall
69.7 69.5 73.7 67.4 68.9
67.8 71.9 69.3 68.7 71.7
48.6 47.8 47.3 47.3 45.9
69.1 68.9 65.3 62.0 65.6
Proprietary Models Vision-Language Models (VLM) Gemini 3 Pro Gemini 3 Flash GPT-5.2 GPT-4o Seed 2.0 Pro
74.5 74.3 66.6 56.9 64.8
79.4 76.2 70.3 69.4 71.0
72.7 72.9 68.0 66.0 72.8
Open-source / Open-weight Models Vision-Language Models (VLM) Kimi 2.5 Qwen3.5-397B-A17B Qwen3.5-35B Qwen3.5-27B Qwen3.5-9B Qwen3.5-2B Qwen3-VL-32B Qwen3-VL-8B InternVL3.5-8B Llama3.2-11B-V
70.6 67.7 59.3 62.4 60.6 41.2 68.8 62.3 47.2 45.6
68.9 73.5 66.1 70.6 66.7 57.9 69.5 68.7 60.0 56.7
74.8 73.7 67.2 68.8 64.9 58.3 69.1 64.9 55.5 48.8
69.1 71.5 69.6 70.9 69.4 66.4 85.7 70.5 66.9 63.0
74.4 74.0 68.6 70.5 64.4 61.5 75.2 70.1 60.5 45.5
49.5 49.5 47.4 47.2 46.5 40.2 45.7 44.9 43.8 38.8
67.9 68.0 62.4 64.5 61.5 52.9 67.6 62.9 54.4 48.8
73.5 67.6 74.0 77.2 61.3
– – – – –
59.9 60.3 58.1 56.1 55.6
63.5 57.6 61.0 62.8 58.4
1.1 3.8 1.7 5.1 2.8
63.4 60.2 63.5 64.0 53.4
Large Language Models (LLM) GLM-5.1∗ DeepSeek-V4-Flash∗ GLM-5.0∗ Llama3.3-70B∗ Llama3.1-8B∗
92.0 92.3 99.0 88.0 58.0
on a carefully designed grading rubric. All per-task scores are normalized to [0, 100]. In Table 3, per-capability scores and the Overall score are macro averages over the available task scores within each aspect and across all tasks, respectively. All models are evaluated zero-shot with manually designed task-specific prompts. Open-weight models are run locally or via compatible hosted endpoints; proprietary models are accessed via official APIs. 4.2
Main Results
Due to space constraints, Table 3 reports only the per-capability and overall scores of the 20 evaluated models; detailed per-task performance metrics are provided in Appendix A. A low ceiling despite strong general-purpose capabilities. The highest overall score is only 69.1 (Gemini 3 Pro), closely followed by Gemini 3 Flash (68.9) and the leading open-weight models, Qwen3.5-397B-A17B (68.0) and Kimi 2.5 (67.9). Unlike saturated leaderboards where a single frontier model dominates, models with vastly different architectures, scales, and open-domain capabilities are compressed into a narrow performance band on real e-commerce workflows. This suggests that general multimodal competence does not transfer cleanly to domain-specific knowledge, long-tail product evidence, and multi-role operational judgments. This validates the core motivation for OXY E COM B ENCH: broad evaluations obscure capability gaps that only emerge under realistic e-commerce constraints. Leaderboard compression highlights the need for domain-specific knowledge injection. Openweight systems rival proprietary models, with Qwen3.5-397B-A17B and Kimi 2.5 ranking third and fourth overall. While the Qwen3.5 family exhibits expected scaling benefits, these gains are modest compared to the remaining gap to reliable e-commerce performance. Thus, OXY E COM B ENCH diverges from the familiar ordering of general-purpose benchmarks. It highlights that in vertical do8
mains, the marginal gains from purely scaling up parameters diminish. Generic instruction following is insufficient without e-commerce-specific knowledge, visual grounding, and multi-role reasoning, underscoring the necessity of domain-specific post-training. (Note that text-only LLM scores reflect a limited language-only subset rather than full multimodal performance.) No model exhibits a complete e-commerce capability profile. Among VLMs, no single model dominates all six capability aspects: Gemini 3 Pro leads Review Analysis; Kimi 2.5 leads Product Understanding and Personalization Analysis; and Qwen3-VL-32B leads Domain-Specific Knowledge and User Behavior Insight. These crossovers indicate that e-commerce competence is not a single latent ability. By disentangling distinct capabilities, the benchmark reveals diverse failure modes for otherwise strong models (e.g., misjudging exaggerated merchant claims as factual, or missing fine-grained visual discrepancies across images). While text-only LLMs (e.g., GLM-5.0) score exceptionally well on the text-only subset of Compliance & Quality Control, they lack the visual capabilities required for the full multimodal quality control workflow, rendering their aspectlevel scores incomparable to VLMs. 4.3
Analysis
The hardest failures align with real business decisions. Task-level averages span a wide range, from 3.1 on Purchase Intention Prediction (Text) to 99.0 on Highlights Understanding. This spread confirms that low-scoring tasks are not artificial corner cases but authentic business bottlenecks. For instance, purchase-intention prediction requires synthesizing noisy user profiles and long-term behaviors. Averaging task means by stakeholder role yields 67.5 for Merchant-side, 64.3 for Platformside, and 55.4 for Customer-side tasks. Qualitative analysis reveals that while models readily adopt a "merchant persona" to extract explicit attributes, they struggle to infer latent "customer intent," frequently hallucinating preferences unsupported by the context. Per-task leadership reveals fragmented strengths. Rankings are distributed across model families rather than dominated by one system: Qwen3.5-397B-A17B ranks in the top two on 11 tasks, while Gemini 3 Pro and Kimi 2.5 do so on 9 tasks each. This fragmentation stems from divergent training paradigms: open-weight models often excel in Chinese e-commerce contexts and product understanding, whereas proprietary models (e.g., Gemini) demonstrate stronger complex reasoning. High-scoring tasks (e.g., Highlights Understanding at 99.0, Product Category Classification at 89.3) typically involve explicit extraction or isolated knowledge lookup. Conversely, operational tasks like Review Response Generation (49.7) demand nuanced response planning and tone control that models currently lack. Fine-grained visual recognition and complex reasoning expose critical failure modes. OXY ECOM B ENCH rigorously tests capabilities beyond general priors. In visually salient tasks like MGCto-Product Consistency Verification (averaging 20.8), error analysis indicates that models frequently fail to align subtle visual discrepancies—such as minor color variations or missing accessories— with product descriptions, often defaulting to a "consistent" guess. Similarly, in complex reasoning scenarios like E-commerce Knowledge Q&A (Multi-turn) (averaging 23.3), models exhibit severe context confusion. Rather than accurately synthesizing information across multiple dialogue turns to address intricate user queries, they often hallucinate non-existent product features or contradict their own earlier statements. These qualitative failures highlight the urgent need for enhanced finegrained visual-text alignment and robust multi-step reasoning in e-commerce applications.
5
Conclusion
We presented OXY E COM B ENCH, a multimodal benchmark for evaluating foundation models in real-world bilingual (Chinese–English) e-commerce, organizing 29 expert-verified tasks across 6 capability aspects for platform operators, merchants, and customers with visually salient and single/multi-turn formats. Evaluating 20 representative models, we find that even frontier general-purpose models struggle, with the highest overall score below 70, while compressed rankings and failure modes expose shared weaknesses in fine-grained visual-text alignment, domain knowledge integration, and multi-step reasoning. OXY E COM B ENCH thus serves as a diagnostic tool charting the path toward models that master the full e-commerce workflow rather than isolated recognition tasks. 9
References [1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023. [2] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems, 2023. [3] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023. [4] Yilun Jin, Zheng Li, Chenwei Zhang, Tianyu Cao, Yifan Gao, Pratik Jayarao, Mao Li, Xin Liu, Ritesh Sarkhel, Xianfeng Tang, Haodong Wang, Zhengyang Wang, Wenju Xu, Jingfeng Yang, Qingyu Yin, Xian Li, Priyanka Nigam, Yi Xu, Kai Chen, Qiang Yang, Meng Jiang, and Bing Yin. Shopping MMLU: A massive multi-task online shopping benchmark for large language models. In Advances in Neural Information Processing Systems, 2024. [5] Haoxin Wang, Xianhan Peng, Huang Cheng, Yizhe Huang, Ming Gong, Chenghan Yang, Yang Liu, and Jiang Lin. ECom-Bench: Can LLM agent resolve real-world e-commerce customer support issues? In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, 2025. [6] Shuyi Xie, Ziqin Liew, Hailing Zhang, Haibo Zhang, Ling Hu, Zhiqiang Zhou, Shuman Liu, and Anxiang Zeng. EcomEval: Towards reliable evaluation of large language models for multilingual and multimodal e-commerce applications. arXiv preprint arXiv:2510.20632, 2025. [7] Chenji Lu, Zhuo Chen, Hui Zhao, Zhenyi Wang, Pengjie Wang, Chuan Yu, and Jian Xu. RAIR: A rule-aware multimodal benchmark for challenging e-commerce relevance assessment. arXiv preprint arXiv:2512.24943, 2025. [8] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. VQA: Visual question answering. In Proceedings of the IEEE international conference on computer vision, 2015. [9] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015. [10] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. OK-VQA: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019. [11] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA models that can read. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019. [12] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, Rongrong Ji, Caifeng Shan, and Ran He. MME: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023. [13] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. MMBench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, 2024. [14] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. MM-Vet: Evaluating large multimodal models for integrated capabilities. In International Conference on Machine Learning, 2024. [15] Shuai Bai, Shusheng Yang, Jinze Bai, Peng Wang, Xingxuan Zhang, Junyang Lin, Xinggang Wang, Chang Zhou, and Jingren Zhou. TouchStone: Evaluating vision-language models by language models. arXiv preprint arXiv:2308.16890, 2023. 10
[16] Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. LVLM-eHub: A comprehensive evaluation benchmark for large vision-language models. arXiv preprint arXiv:2306.09265, 2023. [17] Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingning Liu, Mukai Li, Xiaoshui Huang, Zhiyong Wang, Lu Sheng, Lei Bai, Jing Shao, and Wanli Ouyang. LAMM: Language-assisted multi-modal instruction-tuning dataset, framework, and benchmark. In Advances in Neural Information Processing Systems, 2023. [18] Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Advances in Neural Information Processing Systems, 2022. [19] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. MathVista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations, 2024. [20] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. [21] Zheqi He, Xinya Wu, Pengfei Zhou, Richeng Xuan, Guang Liu, Xi Yang, Qiannan Zhu, and Hua Huang. CMMU: A benchmark for chinese multi-modal multi-type question understanding and reasoning. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, 2024. [22] Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, 2022. [23] Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. BLINK: Multimodal large language models can see but not perceive. In European Conference on Computer Vision, 2024. [24] Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. MANTIS: Interleaved multi-image instruction tuning. Transactions on Machine Learning Research, 2024. [25] Fei Wang, Xingyu Fu, James Y Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, et al. MuirBench: A comprehensive benchmark for robust multi-image understanding. In International Conference on Learning Representations, 2025. [26] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023. [27] Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. HallusionBench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. [28] Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. SEED-Bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. [29] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024. 11
[30] Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, and Trevor Darrell. Aligning large multimodal models with factually augmented RLHF. arXiv preprint arXiv:2309.14525, 2023. [31] Junyang Wang, Yuhang Wang, Jing Zhang, Yukai Gu, Haitao Jia, Jiaqi Wang, Jitao Sang, Guohai Xu, Haiyang Xu, Ming Yan, and Ji Zhang. AMBER: An LLM-free multi-dimensional benchmark for MLLMs hallucination evaluation. arXiv preprint arXiv:2311.07397, 2023. [32] Haibin Chen, Kangtao Lv, Chengwei Hu, Yanshi Li, Yujin Yuan, Yancheng He, Xingyao Zhang, Langming Liu, Shilei Liu, Wenbo Su, and Bo Zheng. ChineseEcomQA: A scalable e-commerce concept evaluation benchmark for large language models. arXiv preprint arXiv:2502.20196, 2025. [33] Chenyu Zhou, Xiaoming Shi, Hui Qiu, Yankai Jiang, Shaoguo Liu, Tingting Gao, Haitao Leng, Xiawu Zheng, and Rongrong Ji. Mix-ECom: Towards mixed-type e-commerce dialogues with complex domain rules. In International Conference on Learning Representations, 2026. [34] Xinyi Ling, Hanwen Du, Zhihui Zhu, and Xia Ning. EcomMMMU: Strategic utilization of visuals for robust multimodal e-commerce models. In Proceedings of the International Joint Conference on Natural Language Processing and the Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, 2025. [35] Daoze Zhang, Zhanheng Nie, Jianyu Liu, Chenghan Fu, Wanxian Guan, Yuan Gao, Jun Song, Pengjie Wang, Jian Xu, and Bo Zheng. MOON: Generative MLLM-based multimodal representation learning for e-commerce product understanding. In Proceedings of the 19th ACM International Conference on Web Search and Data Mining, 2026. [36] Junchen Fu, Wenhao Deng, Kaiwen Zheng, Yu Ye, Ioannis Arapakis, Yongxin Ni, Xuri Ge, and Joemon M. Jose. Benchmarking multimodal large language models for missing modality completion in product catalogues. arXiv preprint arXiv:2601.19750, 2026. [37] Rui Min, Zile Qiao, Ze Xu, Jiawen Zhai, Wenyu Gao, Xuanzhong Chen, Haozhen Sun, Zhen Zhang, Xinyu Wang, Hong Zhou, et al. EcomBench: Towards holistic evaluation of foundation agents in e-commerce. arXiv preprint arXiv:2512.08868, 2025. [38] Kaiyan Zhao, Zijie Meng, Zheyong Xie, Jin Duan, Yao Hu, Zuozhu Liu, and Shaosheng Cao. EComStage: Stage-wise and orientation-specific benchmarking for large language models in e-commerce. arXiv preprint arXiv:2601.02752, 2026. [39] Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. WebShop: Towards scalable real-world web interaction with grounded language agents. In Advances in Neural Information Processing Systems, 2022. [40] OpenAI. 2025.
GPT-5 system card.
https://cdn.openai.com/gpt-5-system-card.pdf,
[41] Google DeepMind. Gemini 3.1 Pro model card. https://deepmind.google/models/ model-cards/gemini-3-1-pro/, 2026.
12
A
Per-task Evaluation Results
To complement the aggregated results in Section 4.2, Table 4 reports the per-task scores of all evaluated models on the 29 tasks in OXY E COM B ENCH. The final Average row uses the same non-missing task-level macro average as the Overall column in Table 3. This convention avoids treating unavailable multimodal results for text-only LLMs as zero, but it also means that models with many missing multimodal entries should be compared with that coverage difference in mind. A.1
Per-task Patterns
The per-task view provides the evidence behind the compressed aggregate leaderboard in Table 3. Qwen3.5-397B-A17B appears in the top two on 11 tasks, Gemini 3 Pro and Kimi 2.5 on 9 tasks each, Gemini 3 Flash on 6 tasks, and Qwen3-VL-32B on 5 tasks. The leadership pattern is distributed rather than hierarchical: a model that is strong on review analysis or product understanding may still fail on personalization, multi-turn reasoning, or fine-grained image–text alignment. This highlights why an e-commerce benchmark must report task-level behavior rather than only an overall score. Task difficulty varies substantially across the benchmark. Highlights Understanding has the highest task-average score (99.0), followed by E-commerce Knowledge Q&A (Single-turn) (96.3), Product Category Classification (89.3), and UGC-to-Product Matching (88.2). These tasks mostly involve explicit extraction, recognition, or localized domain lookup. In contrast, Purchase Intention Prediction (Text) (3.1), Purchase Intention Prediction (Multi-modal) (9.0), MGC-to-Product Consistency Verification (20.8), and E-commerce Knowledge Q&A (Multi-turn) (23.3) are the lowest-average tasks. The hard cases correspond to real commercial decisions where the model must synthesize user preferences, fine-grained visual evidence, and complex multi-step logic. Notably, while models achieve high scores on single-turn tasks like EKSQA (96.3 average) that require isolated knowledge recall, they struggle severely on multi-turn tasks like EKMQA (23.3 average). Although these two tasks differ in construction and evaluation metrics and cannot be directly compared, their respective performance levels illustrate a broader trend: current models are proficient at retrieving static e-commerce facts but fail to maintain context, resolve coreferences, and synthesize information across dynamic, multi-turn interactions. Across capability aspects, Kimi 2.5 is strongest on Product Understanding and Personalization Analysis, Qwen3-VL-32B leads Domain-Specific Knowledge and User Behavior Insight, and Gemini 3 Pro leads Review Analysis and the overall average. These aspect-level crossovers support the main-text claim that e-commerce capability is not reducible to a single general intelligence score. Compliance & Quality Control requires more cautious interpretation because the newly added textonly LLMs only have the Address Verification score in this aspect; their high C&QC averages therefore reflect the available text-only compliance result rather than full coverage of all six C&QC tasks.
B
Difficulty-wise VLM Performance
Figure 4 groups the per-task VLM results by the deterministic P0–P3 task labels introduced in Section 3.1. The aggregate trend is monotonic: the all-VLM macro average decreases from 86.6 on P0 and 74.5 on P1 to 59.9 on P2, then drops sharply to 11.6 on P3. This pattern indicates that the difficulty rubric effectively separates tasks that current VLMs can solve reliably from those that remain largely unsolved under zero-shot evaluation. The curve also explains the compressed overall leaderboard: models perform similarly well on fundamental extraction tasks, but complex e-commerce decisions expose shared bottlenecks.
13
14
87.6 62.0 85.5 83.8 78.0
RSA RSR SLS MIRD RRG
97.7 19.7 91.8
EKSQA EKMQA PCC
PU
DK
PA
69.1
83.8 1.7 6.5 80.1 71.1
TAA PIPT PIPM CPR PRP
Average
52.2 95.7 69.7 53.5
UVEC UPM QRQA CID
UBI
Customer
68.2 53.3 42.4 98.8 88.7 84.9
SIM PEC PAQA HU UPRA SPE
Merchant
RA
84.4 46.5 92.5 97.9 57.5 67.9
81.0 2.8 12.9 79.1 60.6 65.3
68.9
55.0 95.5 71.5 55.0
97.4 31.8 92.0
76.9 61.0 40.9 98.8 47.3 83.2
82.3 60.7 74.5 84.8 49.0
79.9 71.5 96.0 92.9 47.5 12.0
83.8 2.3 2.9 80.9 68.9
59.1 96.9 76.4 55.0
97.6 19.0 91.9
80.0 60.0 37.9 99.4 75.3 84.9
82.3 61.9 82.0 79.6 75.0
84.9 76.0 98.3 94.0 47.7 45.0
62.0
82.5 3.4 12.2 80.4 57.8
46.3 92.6 70.7 65.0
95.7 18.1 88.4
70.8 54.5 34.0 99.4 56.1 81.3
83.0 58.8 83.0 77.2 45.0
70.2 66.0 88.7 86.9 21.2 8.5
65.6
78.5 0.6 5.0 78.4 66.8
63.7 95.8 74.8 52.5
98.0 19.4 89.3
78.2 66.7 33.8 98.2 79.1 80.8
82.5 57.0 79.0 81.5 55.0
69.8 71.5 88.1 93.9 55.6 9.5
67.9
83.2 1.5 10.8 84.9 66.8
70.3 95.1 76.7 55.5
92.3 20.6 93.7
79.5 79.8 37.0 99.4 68.2 85.1
83.8 58.7 88.0 66.1 48.0
79.7 82.4 83.5 96.6 31.4 50.0
68.0
81.8 3.8 6.8 86.5 68.4
65.2 97.2 81.5 52.0
97.4 24.1 92.9
83.0 64.4 37.0 99.4 72.3 86.0
82.4 62.6 89.9 78.5 54.1
84.5 83.5 84.2 93.0 33.9 27.0
62.4
80.0 4.0 8.6 80.0 64.2
56.2 88.5 65.9 64.0
98.0 22.3 88.4
80.7 42.9 39.1 98.8 54.7 86.9
77.6 56.3 69.5 72.5 54.5
77.4 77.0 65.5 89.0 35.0 12.0
64.5
83.1 3.4 8.0 80.8 60.9
54.5 92.7 67.2 67.5
96.7 24.0 92.0
83.7 50.3 35.6 99.4 59.1 84.6
78.0 61.4 84.5 72.9 56.0
81.9 78.5 66.1 85.0 47.7 15.0
61.5
84.5 2.8 10.1 75.5 59.4
52.6 87.1 54.5 63.5
96.7 22.2 89.3
75.2 49.5 33.5 99.4 47.0 85.0
76.2 56.7 80.9 77.4 42.0
71.9 71.0 75.7 77.0 45.6 22.5
52.9
77.5 5.1 8.6 66.5 43.5
48.3 79.8 48.8 69.0
94.6 18.9 85.7
66.0 42.2 32.6 99.4 33.1 76.6
74.4 51.6 66.5 70.0 27.0
59.8 65.0 50.8 46.0 24.4 1.0
67.6
84.5 3.4 10.1 82.6 48.2
61.3 93.4 69.1 77.0
98.5 68.4 90.2
80.3 62.9 31.7 98.8 55.4 85.5
85.3 60.2 74.5 85.4 42.0
74.4 81.5 93.2 95.0 53.8 15.0
62.9
84.5 4.0 11.5 78.1 46.6
53.3 87.1 62.6 77.5
97.4 22.9 91.1
76.1 54.6 30.9 99.4 43.6 85.0
76.7 57.9 78.0 82.0 49.0
79.9 70.0 71.8 81.0 48.8 22.5
54.4
82.5 3.4 13.7 70.6 48.7
35.3 80.5 63.4 63.0
95.0 16.5 89.3
62.7 32.4 32.4 98.2 28.0 79.0
76.9 44.9 76.5 64.6 37.0
59.3 45.5 64.4 78.8 27.5 8.0
48.8
68.0 4.5 7.2 67.7 46.6
19.4 45.6 52.8 64.0
96.3 14.9 77.7
50.8 18.9 28.1 98.8 26.4 69.6
75.1 40.9 60.0 73.3 34.0
54.8 62.0 48.0 59.0 24.4 25.5
63.4
– 1.1 – – –
– – 72.6 54.5
97.6 22.1 –
– – – – – –
75.5 61.2 82.0 75.5 –
– – – 92.0 – –
60.2
– 3.8 – – –
– – 50.0 65.2
94.4 26.2 –
– – – – – –
79.5 58.5 65.0 67.4 –
– – – 92.3 – –
63.5
– 1.7 – – –
– – 69.9 52.0
97.6 18.6 –
– – – – – –
87.2 60.5 81.9 66.5 –
– – – 99.0 – –
64.0
– 5.1 – – –
– – 62.6 63.0
92.2 20.0 –
– – – – – –
77.2 56.3 86.0 89.5 –
– – – 88.0 – –
53.4
– 2.8 – – –
– – 52.8 64.0
95.9 15.4 –
– – – – – –
75.8 41.2 54.5 73.8 –
– – – 58.0 – –
Gem-3P Gem-3F GPT-5.2 GPT-4o Seed-P Kimi Q3.5-397 Q3.5-35 Q3.5-27 Q3.5-9 Q3.5-2 Q3VL-32 Q3VL-8 InVL-8 Lm3.2V GLM5.1-L DS-V4F GLM5.0-L Lm3.3-70 Lm3.1-8
PCV ACFD PDD C&QC AV CCQA MPCV
Platform
Aspect Task
Table 4: Per-task results on OXY E COM B ENCH for all 29 tasks and the macro-average over available task scores (score in [0, 100], higher is better). Task abbreviations follow Table 2; model abbreviations follow Table 3 (Gem=Gemini, Seed-P=Seed 2.0 Pro, DS-V4F=DeepSeek-V4-Flash, Q3.5=Qwen3.5, Q3VL=Qwen3-VL, InVL=InternVL, Lm=Llama). Bold denotes the best score and underlined the second-best within each row; – denotes a missing result in the source spreadsheet.
VLM performance across task difficulty levels
Macro-average score
Proprietary
Open-source / Open-weight
100 80 60 40 20 0
P0
P1
P2
P3
P0
Task difficulty
Gemini 3 Pro Gemini 3 Flash GPT-5.2
P1
P2
P3
Task difficulty
GPT-4o Seed 2.0 Pro Group mean
Kimi 2.5 Qwen3.5-397B-A17B Qwen3.5-35B Qwen3.5-27B Qwen3.5-9B Qwen3.5-2B
Qwen3-VL-32B Qwen3-VL-8B InternVL3.5-8B Llama3.2-11B-Vision Group mean
Figure 4: Difficulty-wise performance trends of all evaluated VLMs on OXY E COM B ENCH. Each point is the non-missing macro average over tasks assigned to the corresponding P0–P3 difficulty tier. The dashed black curve reports the group mean within proprietary or open-weight VLMs. The most severe degradation occurs at P3. Even the strongest P3 model, Gemini 3 Pro, reaches only 25.4, while most VLMs remain near or below the low teens. The P3 tier contains tasks such as MGCto-Product Consistency Verification and purchase-intention prediction, where success requires the joint synthesis of visual evidence, domain-specific rules, and user- or merchant-specific context. By contrast, Qwen3-VL-32B leads P0 with 92.0, Gemini 3 Pro leads P1 with 81.5, and Gemini 3 Flash leads P2 with 67.1. This gap suggests that present VLMs handle explicit extraction and routine reasoning robustly, but fail at expert-level, multi-step decision-making. The proprietary and open-weight curves exhibit similar trajectories, with no group immune to the P3 collapse. While open-weight VLMs are competitive on P0–P2, their performance also plummets at P3. This reinforces the main-text finding that aggregate leaderboard differences are overshadowed by the sheer difficulty of advanced tasks. Progress on OXY E COM B ENCH will require models to acquire deep e-commerce grounding and robust multi-step reasoning abilities, rather than merely scaling up generic multimodal recognition.
C
Task Examples by Difficulty Level
To complement the difficulty-aware categorization presented in Section 3.1, we provide concrete examples of representative tasks in OXY E COM B ENCH across different difficulty levels (Figure 5). The four examples jointly cover the P0 to P3 difficulty tiers, illustrating the progression from fundamental understanding to comprehensive decision-making.
D
Error Analysis of SOTA Multimodal Models
To further understand the gap between open-domain capabilities and specialized e-commerce requirements, we conduct an error analysis on the failure cases of state-of-the-art (SOTA) multimodal large language models, taking Gemini 3 Pro as a representative example. As illustrated in Figure 6, we identify three primary challenges that general-purpose MLLMs currently face in e-commerce scenarios: 1. Visual Hallucination and Fine-Grained Recognition Difficulties. E-commerce applications necessitate highly precise visual inspection. In tasks like Product Damage Detection or MGC-toProduct Consistency Verification, models must detect subtle defects, texture differences, or minor missing accessories. General MLLMs frequently suffer from visual hallucinations, overlooking these fine-grained details and defaulting to a "normal" or "consistent" judgment, which is fatal for e-commerce quality control. 2. Lack of Domain-Specific Professional Knowledge. The e-commerce ecosystem operates on specialized terminology and distinct operational concepts. General MLLMs lack sufficient expo15
User Comment: It works great. USB power supply is very convenient for plugging into the TV. The converted sound quality is perfectly fine. It also comes with an instruction manual, which is quite thoughtful. The delivery is smooth with door-to-door service and intact packaging. Question: Determine whether this user comment is malicious or useless. Please answer only yes or no. Answer No Which of the following aspects mainly reflects the impact of seasonal elements on the design and sales of sports and outdoor products?A. Improve the durability and functionality of products B. Cater to consumers' seasonal needs and enhance fashion sense C. Reduce production costs and increase profit margins D. Cut product categories and simplify the supply chain Answer B
P0
You are a professional e-commerce visual quality control expert. Your task is to check whether the product display image for advertising (the right image) faithfully restores the main features of the original object (the left image). Please carefully compare the **core product main body** in the left and right images, and determine whether the product in the right image has undergone unacceptable **structural changes** or **loss of details**. Ignore the background and lightand-shadow rendering style, and only focus on the physical attributes and details of the product itself. Check item by item as follows: 1. **Shape and Structure**: Is the product’s outline, proportion, and aspect ratio distorted? (e.g., a round bottle turned into a square bottle, the sole becoming thicker) 2. **Key Details**: Are any product components missing or unnecessarily added? (e.g., consistent position of handles, buttons, zippers and interfaces) 3. **Texture and Material**: Has the inherent texture of the material changed substantially? (e.g., leather turned into plastic, metal turned into fabric) 4. **Logo and Text**: If there is a logo or text on the product, is it clear and correctly spelled in the right image? (This is the most error-prone part of generated images) 5. **Color**: Excluding the influence of light and shadow, has the inherent color of the product changed? 6. **Fine Details**: Zoom in on the main body of the product, and carefully check for severe structural missing or inconsistent tiny details (e.g., inconsistent trouser legs, missing blades, etc.) Judgment Standards: **1 (Qualified)**: The main structure of the product is complete, key details are accurately restored, with no visible logical errors to the naked eye. **0 (Unqualified)**: Any of the above defects are found, with structural differences or obvious unrealistic detail artifacts. Only output one number in your reply: `1` or `0`.
P1
Please observe this fruit image. Check whether the fruits in the image have any damage (such as mold, rot, appearance damage, etc.). Output "1" if there is damage, and output "0" if there is no damage. Answer 0
Answer 1
Please read the following user profile summary and graphic-text clues, and complete the multiple-choice questions. ## User Profile Summary - Basic Profile: Female aged 36–45; usual delivery address: Changping District, Beijing; identity labels: bachelor‘s degree, office staff, unmarried; parenting-related demand intensity: relatively high - Purchasing Power Tier: Rated 1 to 5 from high to low – Tier 1 (highest); purchasing power level: extremely high; top target crowd: urban family; user life cycle: mature stage; promotion sensitivity: extremely sensitive; review sensitivity: extremely sensitive. Short-cycle order preferences: root vegetables, blood glucose meters, cleaning & deodorizing products. Medium-cycle order preferences: blood glucose meters, root vegetables, medical & beauty care products. Long-cycle order preferences: leafy vegetables, root vegetables, blood glucose meters. Brand preferences: generic brands, other brands, McDonald’s. Browsing preference categories: eggs, beverages & instant drinks, nutrition & health care. Order preference categories: fresh food, gardening fertilizers, latte coffee. Earlier browsing preference categories: blood glucose meters, leafy vegetables, pork. Earlier representative browsed products: Vigour Fresh Ginger Lemon HPP ColdPressed Juice Ginger Shot 120ml×6 bottles; Xiaotangshan Golden Sweet Potato 500g, directly supplied from the base, fresh traceable vegetables; Gushengtang Unsweetened Lotus Root Starch, original handmade organic lotus root powder for stomach care, suitable for post-surgery recovery, children and puerperae, breakfast meal replacement 200g Earlier cart-addition. Preference categories: other types of congee, tableware, leafy vegetables. Earlier representative cart-added products: Xiaotangshan Organic Potato approx. 400g, organically certified vegetables, delivered by SF Express from Beijing farm; Seasonal Fresh Garlic Scapes from Yunnan 250g; Pork & Egg McMuffin Set Meal Earlier transaction. Preference categories: other types of congee, tableware, pies. Earlier representative purchased products: Pickled Mustard & Minced Pork Pie (1 piece); Vegetable Bun (1 piece); Pork & Egg McMuffin Set Meal
Recently Browsed Product Images:
**Question**: Combined with the above user profile and recent behavioral data, predict the products the user has actually purchased in their **recent real consumption** (multiple choice). A. Lion Pet Clean Antibacterial Wet Wipes 80 Sheets Per Pack, Antibacterial & Deodorizing for Cats and Dogs. Main Image:
Answer 0
P2
Answer 1
B. Egg Tart Set 1.1kg (500g Egg Tart Liquid + 30 Tart Shells), Air Fryer & Oven Baking Semifinished Product. Main Image:
C. Xiaotangshan Golden D. Qin Hua 【Direct Shipment Sweet Potato 500g, Direct from Source】HPP Celery Supply from Base, Fresh Juice, Kale Juice, 100% Freshly Squeezed Fruit & Vegetables Traceable & Vegetable Juice, 296ml*6 Safe. Main Image: Bottles. Main Image:
P3
Answer CD
Figure 5: Representative task examples in OXY E COM B ENCH across different difficulty levels. Topleft (P0): Malicious & Invalid Review Detection requires fundamental understanding to explicitly identify spam or invalid content in user reviews. Top-right (P1): Product Damage Detection demands intermediate analysis and routine reasoning to identify specific defects or damage in product images. Bottom-left (P2): Ad Creative Fidelity Diagnosis necessitates advanced reasoning and fine-grained visual inspection to detect unacceptable structural changes or detail losses between the original product and the generated advertisement image. Bottom-right (P3): Purchase Intention Prediction (Multi-modal) involves expert-level judgment and comprehensive decision-making, requiring the model to synthesize detailed user profiles and long-term behavioral history to predict actual purchasing choices. sure to this deep professional knowledge. Consequently, in tasks like Target Audience Analysis or Selling Point Extraction, models frequently confuse similar semantic concepts or generate generic marketing copy that fails to capture the specific commercial intent or target demographic of the product. 3. Complex Visual Distributions in E-commerce Scenarios. Unlike standard natural images, ecommerce visuals (e.g., product detail pages, promotional posters, and user-generated reviews) are highly complex. They typically feature intricate layouts, dense text overlays, multiple sub-images, and varied professional photography styles. This significant distribution shift from natural images introduces substantial difficulties for general MLLMs, often causing them to miss key textual information embedded within promotional posters or fail to align multiple sub-images correctly.
16
Question:
Question:
Question:
You are an e-commerce material quality control reviewer. Please select the most reasonable review conclusion (result + reason) based on the given information. [Review Guidelines] - Review Object: Generated image (i.e., the material to be reviewed). - Reference Object: Product image (only for comparison, not the object of review). - Final Conclusion: Determine whether the generated image can be used for material making. [Input Information] - Secondary Category: Glasses - Material Type: Transparent background image. (Image order: 1st image = generated image, 2nd image = product image) [Review Rules] 1. Product cutout issues (incomplete, jagged edges, background residue, etc.) 2. Model cutout issues (incomplete limbs) 3. Product or model floating (incoherent with the scene) 4. Image PS traces (abnormal text, LOGO, watermark, arrows, effects) 5. Incomplete product display (main body or key components missing) 6. Model limbs not aligned with the edge 7. Background unrelated to the product 8. Product style/model inconsistency 9. Product color inconsistency 10. Product quantity inconsistency Based on the images and product information, select the most reasonable review conclusion (result + reason). The review object is the generated image, and the product image is only for reference. A. Review Result: Fail; Review Reason: Incomplete product display, missing core functional components B. Review Result: Pass; Review Reason: Image quality check passed, no obvious illegal issues found C. Review Result: Fail; Review Reason: Product cutout has jagged edges and background residue D. Review Result: Fail; Review Reason: Product color inconsistent with the product image
The first image is the picture of Product 1 uploaded by the user, and the other images are the display pictures of Product 2. Based on the image information above, determine whether Product 1 and Product 2 are the same product. A. Same product B. Not the same product
产品名称：焕采红唇乳纹绣专⽤漂唇后的红唇素纹 唇润唇修护帮助反⾊悄悄红 ⻘春橘 展示标题：焕采 红唇乳纹绣专⽤漂唇后的红唇素纹唇润唇修护帮助 反⾊悄悄红 ⻘春橘 参数：产品（注册/备案）名称 ：幻彩红唇乳；总净含量：15ml；保质期：3年；产 品产地：中国⼤陆；批准⽂号/备案编号：粤G妆⽹ 备字2022167055；是否特殊化妆品：⾮特殊化妆品 ；功效：修护、淡化唇纹、提亮唇⾊；类别：乳霜 状 商品主图： 商品详情图： 平台价：79.0元 参考价：79.0元 这款15ml乳霜状的唇部产品是什么 品类? 请找出正确的答案。 A: 钟表眼镜 > 眼镜 > 眼 镜配件/护理剂 B: 美妆护肤 > ⾯部护肤 > 唇膜/唇部 精华 C: 美妆护肤 > 美妆⼯具 > 化妆棉 D: 美妆护肤 > ⾯部护肤 > 润唇膏
Labels: A
Labels: B Model Response: Based on the review guidelines and a careful comparison of the images, here is the analysis: 1. **Analyze the Generated Image (Image 1):** - Observing the left side of the glasses (viewer's left), the temple (arm) extends backward naturally and ends with a curved ear piece. - However, observing the right side of the glasses (viewer's right), the temple extends horizontally backward (with text printed on it) but abruptly stops. The curved ear piece (temple tip) is completely detached from the arm and is bizarrely attached to the bottom right edge of the lens frame. 2. **Evaluate against Review Rules:** - This represents a severe AI generation error, leading to a structural deformity where a core functional component (the right temple/ear piece) is broken and incorrectly placed. This falls directly under the rule regarding **"Incomplete product display (main body or key components missing)"**. 3. **Compare with Options:** - A perfectly describes this structural failure. - B is incorrect because there is a blatant generation issue. - C is incorrect as the primary issue is not cutout edges but a deformed/broken product structure. - D is incorrect because the color (black) remains consistent. Therefore, the generated image cannot be used for material placement. **Correct Option: A** Review Result: Fail; Review Reason: Incomplete product display, missing core functional components**
Model Response: Based on a detailed comparative analysis of the provided images: Product 1: The matching lid is clearly shown in the bottom right corner of the image. The lid handle (knob) is black and flat in shape. Product 2: The main image is clearly marked with the words "Free wooden lid included". In the subsequent size chart, lifestyle images, and white background product shots, the lid handle is consistently shown as a woodcolored cylindrical design made of solid wood. Although both are Japanese-style snow pans with a black body and maifanite speckled texture, there is a clear difference in the material and color of the included lid handles — Product 1 has a black knob, while Product 2 features a highlighted solid wood knob. This means they belong to different specifications and product variants. Therefore, they are not identical products. Final Answer: B. Different products
Labels: B Model Response: 根据您提供的商品信息，该产品名称为“幻彩红唇乳”， 其主要功效为“修护、淡化唇纹、提亮唇⾊”，并在标题 中明确提到了“润唇修护”以及“会变⾊的红唇乳”，具有 根据体温变⾊的特点（类似变⾊润唇膏）。它主要⽤于 唇部的保湿滋润、修护及⽇常提⽓⾊。 因此，该产品 属于美妆护肤类⽬下的⾯部护肤（唇部护理）产品，最 符合的品类是润唇膏。 正确答案是：D: 美妆护肤 > ⾯ 部护肤 > 润唇膏
Figure 6: Representative failure cases of SOTA multimodal models (e.g., Gemini 3 Pro) in OXY ECOM B ENCH . The errors highlight three main deficiencies: severe visual hallucination in finegrained recognition, confusion of domain-specific semantic concepts, and struggles with the dense and complex visual distribution typical of e-commerce images.
17