CodeMMR: Bridging Natural Language, Code, and Image for Unified Retrieval
arXiv:2604.15663v1 [cs.SE] 17 Apr 2026
Jiahui Geng1,2 Qing Li1,3 * Fengyu Cai4 Fakhri Karray1 1 MBZUAI, UAE 2 Linköping University, Sweden 3 University of Groningen, Netherlands 4 TU Darmstadt, Germany
Abstract Code search, framed as information retrieval (IR), underpins modern software engineering and increasingly powers retrieval-augmented generation (RAG), improving code discovery, reuse, and the reliability of LLM-based coding. Yet existing code IR models remain largely textcentric and often overlook the visual and structural aspects inherent in programming artifacts such as web interfaces, data visualizations, SVGs, schematic diagrams, and UML. To bridge this gap, we introduce MMCoIR, the first comprehensive benchmark for evaluating multimodal code IR across five visual domains, eight programing languages, eleven libraries, and show the challenge of the task through extensive evaluation. Therefore, we then propose CodeMMR, a unified retrieval model that jointly embeds natural language, code, and images into a shared semantic space through instruction-based multimodal alignment. CodeMMR achieves strong generalization across modalities and languages, outperforming competitive baselines (e.g., UniIR, GME, VLM2Vec) by an average of 10 points on nDCG@10. Moreover, integrating CodeMMR into RAG enhances code generation fidelity and visual grounding on unseen code generation tasks, underscoring the potential of multimodal retrieval as a core enabler for next-generation intelligent programming systems. Datasets are available at HuggingFace1,2 .
1. Introduction Code search, typically implemented via information retrieval (IR), is a cornerstone of modern software engineering, enabling developers to effectively locate, understand, and reuse relevant code snippets [6, 9, 17, 25]. Code IR can further improves software quality by surfacing related explanations, bug analyses, and design patterns,as ev* Corresponding author: [email protected] 1 https://huggingface.co/datasets/JiahuiGengNLP/
MMCoIR-train 2 https://huggingface.co/datasets/JiahuiGengNLP/ MMCoIR-test
idenced by its integration into commercial products such as Commercial products such as GitHub Code Search and VS Code’s intelligent lookup tools [5]. Recently, code retrievalaugmented generation (RAG) systems [26, 35, 37] have leveraged retrieval to mitigate hallucinations in large language models (LLMs), improving reliability and grounding in programming-related tasks, such as code generation. Despite advances in code IR, existing systems remain predominantly text-based, focusing on semantic similarity between code and natural language [6, 9, 17]. In practice, however, modern software artifacts are deeply multimodal: code often embodies visual functionality, such as defining a web layout [7, 34], rendering a chart [13, 31, 40], or generating a UML diagram [1], etc. Developers need to understand what a piece of code looks like when executed, or, conversely, refer to the code that can generate a given visual result from an image or design. This mismatch highlights an urgent need for multimodality in code IR. To bridge this gap, we introduce MMCoIR, a comprehensive benchmark for systematically evaluating multimodal multilingual code retrieval models across diverse domains. As shown in Table 1, MMCoIR unifies datasets from WebUI, data charts, SVGs, schematic diagrams, and UML diagrams, each containing paired samples of images and codes in eight programming languages (e.g., HTML, CSS, JS, Python, XML, LaTeX, PlantUML, etc) and eleven libraries. This design supports a wide range of retrieval tasks, including text-to-code, image-to-code, and text+image-tocode, as well as their reverse directions for visual retrieval. The textual queries include both natural image descriptions and instruction-style prompts derived from code editing and repair datasets. As the first multimodal and multilingual code retrieval testbed, MMCoIR enables research for more realistic and diverse retrieval scenarios. Multimodal code retrieval is challenging; we comprehensively evaluating the existing multimodal embedding models [11, 20, 38] spanning different architectures and scales on MMCoIR and finding their performance lacking despite strong model initialization and large-scale training. Therefore, we introduce CodeMMR, a unified multimodal, multilingual code retrieval model that jointly encodes nat-
Instruction: Please retrieve the image that matches the description.
Instruction: Please retrieve the code that matches this image.
Instruction: Please retrieve the image that matches this code.
Query Image
Query Code x = [11.6, 11.9, 12.25, 12.6, 12.8] y = [0.011, 0.02, 0.016, 0.037, 0.048] , … ax.errorbar(x, y, yerr=yerr, fmt=‘o’, color=‘royalblue’, capsize=4, …)
Query Text The X-axis is labeled "Month" and displays the twelve months of the year. The Y-axis is labeled "Temperature Change (°C)" …
Instruction: Please retrieve the code that matches the input.
Query Text + Image Change the color of the nodes labeled 𝑘0 and 𝑘𝑡+1 = 𝑓(𝑘𝑡 ) to purple, and adjust the font to be bold for better visibility.
CodeMMR MMCoIR Image Candidate
Code Candidate Code Candidate
Image Candidate
[LaTeX] \begin{tikzpicture}[scale=0.7] \draw[->] (0,0)--(0,4); \node [right, text=purple, font=\bfseries] at (0,4) {$D_{\min}$}; …
[PlantUML] @startuml |1| exodus containing |2| stickler sexuality :foxy mope; <<continuous>> :jamboree gyms; -> yawn newest;
Figure 1. Illustration of CodeMMR and MMCoIR: Unified multimodal CodeMMR for text, image, and code, evaluated on MMCoIR.
ural language, code, and visual modalities into a shared embedding space. Trained through instruction-based multimodal alignment, CodeMMR delivers strong in- and outof-distribution perforamnce across all MMCoIR subtasks, surpassing VLM2Vec-v2 [21] by an average of 10 absolute points on nDCG@10, which demonstrates its accurate and robust retrieval across code, image, and text. On two unseen image-to-code generation tasks, ChartMimic Direct and WebCode2M-Mid, RAG with CodeMMR outperforms the non-retrieval baseline, by 10.0 points in Execution Rate and 9.4 points in Visual Accuracy, respectively, and also beats RAG with baseline retrievers, demonstrating its substantial practical values. Moreover, we outline promising directions for multimodal code retrieval, including longcontext retrieval (e.g., SVG) and fine-grained retrieval along specific dimensions, e.g., text, layout, and color. Overall, our contributions are threefold:
• We introduce MMCoIR, the first large-scale benchmark for embedding model evaluation on multimodal multilingual code retrieval across domains and modalities. • We further train CodeMMR, a unified embedding model that aligns text, code, and visual representations through instruction-based multimodal training. • Extensive experiments show that CodeMMR outperforms existing baselines and improves RAG-based code generation, showcasing significant practical value.
2. Related Work Multimodal Embedding and Universal Representation Learning Early vision–language models (VLMs) such as CLIP [23], ALIGN [10], and BLIP [15] demonstrated that large-scale contrastive learning between image–text pairs can produce general-purpose multimodal representations. Subsequent research extended this idea toward universal multimodal embeddings capable of handling diverse downstream tasks within a shared semantic space [11, 21, 29, 36]. Frameworks like UniIR [29] and MagicLens [36] unified multiple retrieval tasks through shared encoders and contrastive objectives, showing improved task generalization. MM-EMBED [18] first unified text, image, and video representations via multimodal large language models (MLLMs) under an instruction-driven retrieval objective. Building on this, VLM2Vec [11] advanced vision–language models through instruction-conditioned contrastive learning, enabling scalable multitask multimodal embedding within a shared representation space. Further advances such as Think-then-Embed [4] incorporate generative reasoning contexts prior to embedding, enhancing compositional understanding and semantic coherence. LamRA [20] leveraged large multimodal models as retrieval assistants, bridging encoder-based embedding and cross-modal reasoning via adaptive attention integration. Overall, recent advances point toward unifying multimodal retrieval with MLLMbased embeddings; here we take the next step by incorpo-
rating code, a critical yet long-overlooked modality. Code IR and RAG Early datasets such as CodeSearchNet [9] and CoSQA [8] enabled text–code semantic alignment evaluation but lacked domain and task diversity. CoIR [17] addressed this by unifying ten datasets and eight retrieval types across fourteen programming languages; CoQuIR [6] further enriched it with quality-aware annotations, and XCodeEval [12] extended evaluation to multilingual, multi-task settings. On the methodological side, CodeXEmbed [19] applies a two-stage training paradigm to adapt general-purpose retrievers to code domains, while Revela [3] achieves competitive CoIR performance through unsupervised retriever learning framed as language modeling. In generation, code RAG [28] reduces hallucinations by grounding code generation with retrieved reference code, documentation, and tutorials. Yet existing work treats code purely as text, overlooking visual modalities—such as web interfaces and visualizations—that are integral to real-world human–machine interaction. We address this gap by extending code retrieval and RAG to incorporate these coderelated visual modalities.
3. Task Overview 3.1. Problem: Code Multimodal Retrieval In a unified multimodal code retrieval framework, users issue queries originating from diverse modalities, including natural language, source code, and visual content. Formally, a query is denoted as q and may take the form of text qt , image qi , or code qc . Beyond unimodal cases, certain retrieval scenarios involve composed queries that combine multiple modalities (e.g., qt,i or qt,c ), which are prevalent in tasks such as code editing, code repair, and visualization refinement [30, 39]. Likewise, a retrieval target r can correspond to any single or composite modality, such as ri , rc , or ri,c . This formulation allows a single retriever to accommodate heterogeneous search paradigms, such as text-tocode, image-to-code, and text+image-to-code, under a unified representation framework, as summarized in Table 1. To ensure flexibility of retrieval intent across heterogeneous tasks, each query is accompanied by a naturallanguage instruction qinst , which explicitly specifies the retrieval goal and domain context (e.g., “please retrieve the code that matches this image.”). This instruction-based design clarifies task semantics and improves compatibility across datasets with varying modality support. For example, datasets containing only image–code pairs (qi → rc or qc → ri ) are augmented with standardized prompts to indicate the intended retrieval direction. In more complex composed scenarios, such as qt,i → rc , the textual component qt is typically drawn from dataset-provided prompts (e.g., “change the color of the nodes to purple.”).
The retrieval process is then defined as follows: given a query q and its corresponding instruction qinst , the goal is to identify the most relevant candidate r∗ from a heterogeneous candidate pool R by maximizing the similarity in a shared multimodal embedding space: r∗ = arg max fθ (q, qinst )⊤ fθ (r) , r∈R
where fθ (·) denotes a multimodal encoder parameterized by θ. This unified formulation enables instruction-conditioned retrieval across natural language, code, and visual domains. A detailed description of the datasets and retrieval configurations is provided in Section 4, where we introduce the proposed benchmark, MMCoIR.
3.2. Solution: Code Multimodal Retriever Training To enable effective cross-modal retrieval between natural language, code, and images, we train CodeMMR to project heterogeneous modalities into a shared embedding space. Given a pretrained vision–language model (VLM) as the initialization, CodeMMR extends its capability to understand structured code representations and to align them with both textual and visual semantics. Training Objective. We employ a contrastive learning objective based on the InfoNCE loss [22]. For a batch of B B query–target pairs {(qi , r+ i )}i=1 , CodeMMR encodes each instruction-conditioned query and target as: hqi = fθ (qi , qinst ),
hr+ = fθ (r+ i ), i
where fθ is the multimodal encoder parameterized by θ. The model maximizes the similarity between each query and its positive target while contrasting against all negatives r− i ∈ N . The loss is defined as: B ϕ(hqi , hr+ ) 1 X i P log , Lret = − B i=1 ϕ(hqi , hr+ ) + r− ∈N ϕ(hqi , hr− ) i i i (1) where ϕ(hq , hr ) = exp τ1 h⊤ q hr is a temperature-scaled similarity and τ is a hyper-parameter. We incorporate both in-batch negatives and hard negatives drawn from semantically close examples to improve discriminative alignment.
4. MMCoIR Benchmark 4.1. Overview The MMCoIR benchmark establishes a unified multimodal code retrieval framework for comprehensive evaluation across five representative visual domains: (i) web development and UI design, (ii) data charts, (iii) scalable vector graphics, (iv) schematic diagrams, and (v) software engineering (UML). It unifies heterogeneous datasets across
Domains WebUI
Datasets
Images
Languages
WebSight [14]
Screenshots
HTML+CSS+JS
Web2Code [33] Screenshots HTML+CSS+JS Sketch2Code [16] Hand-drawn sketches Chart2Code [40] Chart images
Python (2 libraries)
ChartGen [13]
Chart images
Python (11 libraries)
ChartEdit [39]
Chart images
Python (2 libraries)
SVGStack [24]
SVG images
XML for SVG
MMSVG [32]
SVG images
XML for SVG
DiagramGen Schematic Benchmark [30] Diagrams
Diagrams
LaTeX + TikZ, Graphviz DOT
DATIKZv3 [2]
Diagrams
LaTeX + TikZ
PlantUML [1]
UML diagrams
PlantUML scripts
Data Charts
SVG
SE UML
#Training qi → rc (100k) qc → ri (100k) qi → rc (100k) qi → rc (100k) qi → rc (100k) qc → ri (100k) qt → rc (100k) qt → ri (100k)
qi → rc (100k) qc → ri (100k) qi → rc (200k) qc → ri (200k) qt → rc (200k) qt → ri (200k)
qi → rc (100k) qc → ri (100k) qi → rc (200k) qc → ri (200k)
#Test qi → rc (100k) qc → ri (100k) qi → rc (2k) qi → rc (2k) qi → rc (2k) qi → rc (2k) qc → ri (2k) qt → rc (2k) qt → ri (2k) qi → rc (2k) qc → ri (2k) qt,c → ri (2k) qt,c → ri,c (2k) qi → rc (2k) qc → ri (12k) qi → rc (4k) qc → ri (4k) qt → rc (4k) qt → ri (4k) qi → rc (270) qc → ri (270) qt → rc (267) qt,i → rc (200) qi → rc (2k) qc → ri (2k) qi → rc (4k) qc → ri (4k)
Table 1. Overview of MMCoIR, which covers various domains, languages, libraries, and format combinations of queries and targets.
multiple programming languages and visual modalities, enabling systematic evaluation of cross-domain and crosslingual understanding. Specifically, MMCoIR incorporates datasets from several key areas: • Web Development: WebSight [14], Web2Code [33], and Sketch2Code [16] translate realistic webpage screenshots into HTML/CSS code, emphasizing layout understanding and visual fidelity. • Data Visualization: Chart2Code [40], ChartGen [13], and ChartEdit [39] capture the correspondence between data and visual encodings. • Vector Graphics: SVGStack [24] and MMSVG [32] involve structured SVG representations for scalable vector graphic synthesis and manipulation. • Schematic Diagrams: DiagramGenBenchmark [30] and DATIKZv3 [2] focus on schematic or symbolic diagrams with LaTeX TikZ and Graphviz-style languages. • Software Engineering: PlantUML [1] targets software diagrams such as class and sequence diagrams, highlighting semantic precision and logical relationships. Together, they cover a linguistically diverse spectrum—from markup and scripting languages (HTML, CSS, JavaScript, XML, Python) to domain-specific formats such as LaTeX TikZ and PlantUML. Overall, MMCoIR pro-
vides the first comprehensive foundation for evaluating multimodal and multilingual code retrieval systems, fostering research toward more generalizable and semantically grounded cross-modal embeddings. Please refer to supplmentary documents for more details.
4.2. Dataset Structure MMCoIR employs a unified data schema to support a wide range of multimodal retrieval scenarios across vision–language–code modalities. To ensure consistent query semantics across these heterogeneous datasets, we standardize how textual instructions are represented and interpreted during retrieval. Instruction formulation. When datasets provide predefined textual instructions, we directly use them as queries. For datasets without explicit instructions, we introduce standardized prompts (e.g., Please retrieve the image that matches the description.”) to clarify retrieval intent and explicitly define the input and output modalities. The interpretation of textual fields varies across datasets. In ChartGen, the text corresponds to chart descriptions, while in ChartEdit and DiagramGenBenchmark, it represents editing or repair instructions (e.g., Change the line color to red.”). Notably, ChartEdit and DiagramGenBenchmark also introduce
WebUI Methods
WebSight qc → ri qi → rc
Data Charts
Web2Code Chart2Code qi → rc
qi → rc
SVG
ChartGen
SVGStack
MMSVG
Schematic
UML
DATIKZv3
PlantUML
Avg
qc → ri qi → rc qc → ri qi → rc qc → ri qi → rc qt → rc qt → ri qc → ri qi → rc qc → ri qi → rc Metric: Hit@1
UniIR (CLIP SF) UniIR (BLIP SF) VLM2Vec (2B) VLM2Vec (7B) VLM2Vec-v2 (2B) LamRA (7B) GME (2B) GME (7B)
9.6 10.8 15.7 35.1 72.9 17.2 67.1 79.0
9.8 11.5 16.2 38.0 74.6 16.7 66.2 77.8
10.5 12.2 16.1 42.5 85.0 17.4 67.9 89.3
8.2 7.2 10.7 5.4 91.9 22.1 80.3 87.2
11.5 12.7 9.9 17.3 55.5 26.0 27.7 37.0
11.0 11.1 0.6 10.0 62.1 15.2 52.7 52.7
0.1 0.2 0.3 0.4 7.4 9.9 2.3 3.5
0.1 0.1 0.1 0.4 7.8 9.9 3.7 4.3
0.1 0.0 0.1 0.1 0.1 0.2 0.1 0.0
0.0 0.1 0.1 0.1 0.1 6.8 0.0 1.5
0.1 0.0 0.0 0.1 0.0 3.3 0.1 3.3
23.7 19.5 40.0 33.5 40.0 42.2 47.6 51.8
9.2 8.3 9.8 16.8 85.2 44.6 67.5 65.1
8.5 7.9 9.4 16.4 81.7 51.7 63.7 71.4
20.5 21.7 29.8 44.6 67.7 45.4 89.9 67.7
21.3 20.0 24.7 48.2 100.0 62.0 99.7 100.0
9.0 8.9 11.5 19.3 53.3 24.4 46.0 49.5
UniIR-FT (CLIP SF) UniIR-FT (BLIP SF)
55.8 57.6
56.1 58.4
58.9 59.7
47.3 51.2
50.2 46.7
56.8 53.1
6.4 5.6
4.7 5.2
3.3 3.5
3.6 3.3
2.5 2.7
42.9 43.3
37.3 40.1
45.6 43.9
61.8 58.0
53.3 50.7
36.6 36.4
CodeMMR (2B) CodeMMR (2B)-Mix
96.4 97.7
97.4 97.2
97.9 97.8
99.5 99.4
83.3 81.8
84.9 84.1
14.9 12.9
16.7 14.7
7.9 6.1
7.4 6.2
5.4 5.8
52.6 52.8
93.5 93.2
94.3 94.5
100.0 100.0
100.0 100.0
65.4 65.2
UniIR (CLIP SF) UniIR (BLIP SF) VLM2Vec (2B) VLM2Vec (7B) VLM2Vec-v2 (2B) LamRA (7B) GME (2B) GME (7B)
18.6 23.4 31.0 50.3 83.7 27.6 73.1 82.3
19.1 22.5 30.7 50.0 82.2 28.4 70.8 81.6
20.2 23.2 30.4 56.4 90.8 28.0 78.0 93.0
15.3 14.7 16.0 11.2 95.2 28.3 87.9 92.1
25.3 26.3 22.1 30.0 69.3 42.3 44.9 54.5
22.9 20.0 2.3 19.1 75.5 26.5 68.7 69.3
0.2 0.3 0.6 2.0 8.8 9.9 3.2 5.0
0.3 0.3 0.4 2.5 9.3 9.9 4.3 5.9
0.2 0.1 0.2 0.3 0.3 2.1 0.3 3.0
0.1 0.1 0.2 0.4 0.2 1.8 0.6 8.5
0.2 0.1 0.2 0.2 0.3 2.5 0.8 3.7
44.2 38.2 49.3 53.1 40.9 53.1 58.1 43.0
14.7 13.8 14.1 32.3 88.4 57.5 72.4 71.1
15.3 14.2 12.2 35.7 87.4 62.2 68.8 73.6
30.1 27.9 37.9 52.0 99.7 100.0 93.0 74.1
27.6 26.9 34.9 55.0 95.2 100.0 99.8 100.0
15.9 15.8 17.7 28.2 58.0 36.3 51.5 53.8
UniIR-FT (CLIP SF) UniIR-FT (BLIP SF)
72.2 69.9
68.7 70.2
67.8 68.2
57.4 63.5
59.7 55.6
67.0 64.1
8.8 7.4
7.9 8.0
5.4 4.9
5.2 5.1
3.0 3.2
46.9 48.1
40.7 44.2
51.2 50.0
63.9 60.7
61.2 59.3
42.9 42.7
CodeMMR (2B) CodeMMR (2B)-Mix
92.8 91.9
90.5 89.8
98.6 98.4
99.8 99.7
90.4 88.8
91.5 91.2
18.6 12.3
19.7 15.5
11.2 9.7
12.3 11.3
9.4 8.2
58.2 59.5
97.8 93.2
97.0 94.5
100.0 100.0
100.0 100.0
68.0 66.5
Metric: nDCG@10
Table 2. Performance on MMCoIR (Hit@1 and nDCG@10, %): Across model scales, CodeMMR substantially outperforms prior multimodal embedding models, highlighting the task’s difficulty and the need for specialized retriever learning. Blue and light blue indicates the best and second best performance across the models.
several novel retrieval settings, such as text + code → image, text + code → image + code, and text + image → code, which are absent from the training data and therefore serve as unseen tasks for evaluating model generalization. Sampling and balance. To ensure balanced representation across domains, we subsample instances from larger datasets to maintain comparable scales. In most cases, the training set contains 100k instances and the test set contains 2k instances. Some datasets include multiple subsets for finer granularity. For example, MMSVG is divided into MMSVG-Icon and MMSVG-Illustration, each containing 100k training and 2k test instances, while PlantUML is partitioned into PlantUML-Act and PlantUML-Seq, focusing on activity and sequence diagrams, respectively. Datasets, such as Sketch2Code, ChartEdit, and DiagramGenBenchmark, only provide test data and are therefore used to assess the model’s generalization ability on unseen domains.
5. Experiments Leveraging MMCoIR, we evaluate existing vision–language embedding models on the benchmark and train a unified code–image–text embedding model.
Baselines. We evaluate our method against a diverse set of state-of-the-art vision–language representation and retrieval models to ensure a fair and comprehensive comparison across domains. The baselines include two UniIR [29] variants, UniIR (CLIP SF) and UniIR (BLIP SF), which are unified multimodal retrievers trained with instruction tuning to support diverse retrieval formats. We also compare with several VLM-derived embedding models: GME [38] (2B and 7B), VLM2Vec [11] (2B and 7B), VLMVec-v2 [21], and LamRA [20]—which are primarily fine-tuned from the Qwen2VL [27] backbone on large-scale image–text data. These models extend Qwen2VL’s multimodal capabilities by incorporating task-specific objectives and instructionaware embeddings. Finally, we fine-tune the UniIR models on the MMCoIR training set (denoted as UniIR-FT) to better adapt them to our tasks. For more details about the baselines, please refer to the supplementary material. Implementation Details. CodeMMR is implemented using the PyTorch framework and built upon the Qwen2VL-2B-Instruct backbone [27]. We employ a contrastive learning objective for multimodal alignment following the VLM2Vec setup [11]. The model is fine-tuned with LoRA (r=8) in bf16 precision, using EOS pooling with ℓ2 normalization and a contrastive loss temperature of 0.02. All ex-
Methods
WebUI
Data Charts
Schematic Diagram
Sketch2Code
ChartEdit
DiagramGenBenchmark
qi → rc