arXiv CSopen access

Learning Generalizable Multimodal Representations for Software Vulnerability Detection

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

artificial intelligence, reasoning, knowledge representation

Learning Generalizable Multimodal Representations for Software Vulnerability Detection Zeming Dong1 , Yuejun Guo2 , Qiang Hu*3 , Yao Zhang3 , Maxime Cordy1 , Hao Liu3 , Mike Papadakis1 , and Yongqiang Lyu3 1 University of Luxembourg, 2 Luxembourg Institute of Science and Technology, 3 Tianjin University

50000

Source code and its accompanying comments are complementary yet naturally aligned modalities — code encodes structural logic while comments capture developer intent. However, existing vulnerability detection methods mostly rely on single-modality code representations, overlooking the complementary semantic information embedded in comments and thus limiting their generalization across complex code structures and logical relationships. To address this, we propose M ULTI V UL, a multimodal contrastive framework that aligns code and comment representations through dual similarity learning and consistency regularization, augmented with diverse code-text pairs to improve robustness. Experiments on widely adopted DiverseVul and Devign datasets across four large language models (LLMs) (i.e., DeepSeek-Coder-6.7B, Qwen2.5-Coder-7B, StarCoder2-7B, and CodeLlama-7B) show that M ULTI V UL achieves up to 27.07% F1 improvement over prompting-based methods and 13.37% over code-only Fine-Tuning, while maintaining comparable inference efficiency.

CCS Concepts • Security and privacy → Software security engineering.

Keywords Software vulnerability detection, Multimodal learning, Cybersecurity

Introduction

Software vulnerability detection is a critical task for ensuring the security and reliability of modern software systems [8]. The need for effective detection has become more urgent as reported vulnerabilities continue to increase in both volume and diversity. As shown in Figure 1, the number of reported vulnerabilities in the first four months of 2026 is already close to the full-year total in 2021, and this growth spans multiple vulnerability types, including memory corruption, SQL injection, and information leakage. This trend indicates that vulnerability detection is becoming increasingly challenging and places growing pressure on existing detection methods. Undetected vulnerabilities may lead to severe consequences, such as unauthorized access and system compromise, causing harm to both users and organizations. Traditional vulnerability detection methods, including static analysis and dynamic testing, remain indispensable in practice because they provide rule-based and interpretable analyses, and can be highly effective for identifying known vulnerability types in established software security workflows. However, they often fail to capture the complex semantics underlying security issues in source code and face trade-offs in scalability, coverage, and false positives, especially

Number of vulnerabilities

arXiv:2604.25711v1 [cs.SE] 28 Apr 2026

Abstract

40000

SQL injection XSS Execute code

Information leak Memory corruption Other vulnerabilities

48448 40313 60.1%

30000

25084

20000 10000 0

16510 55.3%

6449

53.5% 17.0% 19.2%

2016

10.4% 8.4% 10.5% 12.4% 3.0%

2018

18323 60.3% 10.0% 5.9% 9.2% 12.0%

2020

57.0%

20571

20153 55.7% 55.6% 12.4% 4.5% 10.4% 13.5% 3.7%

2021

11.2% 4.4% 8.2% 13.4% 7.0%

Year

2022

5.8% 9.8% 18.4% 6.6%

2024

6.0% 6.3% 18.0% 8.1%

2025

68.4% 4.6%

6.9% 12.2% 6.3%

2026

Figure 1: Vulnerabilities by type and year. The statistics are collected from CVEdetails [1].

when source code is large, complex, or obfuscated [33, 48, 52, 58]. To improve automation and scalability, recent work has increasingly turned to learning-based vulnerability detection, leveraging deep neural models (e.g., graph neural networks [11]), pre-trained code models, and large language models (LLMs) to identify vulnerable code types across a wide variety of coding environments, including diverse programming languages and software libraries [8, 52, 55, 66]. Among these, pre-trained code models have become an important foundation for modern code intelligence. Earlier encoder-based models, such as CodeBERT [20] and GraphCodeBERT [23], established strong baselines for code understanding and downstream software engineering tasks, while recent surveys and empirical studies [13, 50, 66] suggest that code LLMs (e.g., CodeLlama [46]) have emerged as increasingly strong and representative models for a wide range of software engineering tasks. Despite this progress, most learning-based vulnerability detection methods still rely primarily on a single code modality, such as token sequences or graph-derived code representations [8, 10, 55]. This design still has two important limitations. First, it can restrict detection accuracy, because vulnerability-relevant features are often subtle, distributed across execution paths, and closely tied to program semantics. Sequence-only representations may overlook structural dependencies, while even structure-enhanced representations can remain insufficient for capturing the program behaviors most relevant to vulnerabilities [10, 23]. Second, single-modality vulnerability detection methods often generalize poorly. Existing studies show that vulnerability models can learn superficial mappings from source code to labels, overfit to dataset-specific artifacts, and degrade substantially on unseen projects and out-of-distribution (OOD) data [18, 47]. These observations suggest that improving vulnerability detection requires not only stronger code encoders but

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

also richer supervision signals that expose models to higher-level semantics and reduce overfitting to surface features of source code. Recent studies have explored contrastive and multimodal learning as promising directions for improving semantic modeling in vulnerability detection and related software engineering tasks [10, 35]. These efforts suggest that incorporating information beyond a single code modality can help models learn more informative representations. However, important challenges remain. Existing methods often rely on only a single original code–text view of the input, and lack explicit mechanisms for maintaining robustness under lightweight perturbations of code and text. Moreover, in some cases, they still depend on natural language descriptions or prompts during inference [35]. Consequently, they may struggle to learn code representations that are simultaneously semantically aligned, robust to input variation, and practical for deployment where only code is available. These limitations are particularly important in real-world vulnerability detection, where comments may be missing, noisy, or unavailable at test time, and where models must generalize beyond dataset-specific coding conventions. To address these limitations, we propose M ULTI V UL, a multimodal vulnerability detection framework that enriches code supervision with automatically generated natural language comments and learns more generalizable code representations through dualview multimodal training. Specifically, M ULTI V UL combines a dualencoder architecture, dual-CLIP alignment over original and augmented code–text pairs, and consistency regularization across views. In this way, the model leverages richer semantic supervision during training while retaining a code-only inference pipeline at test time. Rather than introducing yet another multimodal contrastive architecture, we focus on a more practical and still underexplored question in vulnerability detection: how can multimodal supervision be used to improve generalization without assuming the availability of natural language context during deployment? Our key insight is that automatically generated natural language comments can provide complementary semantic information about code functionality and intent, which may be difficult to infer from source code alone. This is particularly useful for vulnerability detection, where security-relevant information can be subtle and depend on API usage patterns, control-flow conditions, and function-level behavior. Prior work has shown that comments can express program behavior in natural language, and that comment augmentation can improve downstream code understanding in code LLMs [7, 53]. We therefore use generated comments to connect low-level implementation details with higher-level functional behavior, providing richer supervision for vulnerability detection [45]. However, simply appending text to code is often insufficient [61]. To better use textual supervision, the model should align code and text representations and keep them stable under lightweight perturbations. Existing studies show that lightweight code transformations, such as GenCode [16], can improve downstream code modeling, while simple textual perturbations such as random swap and random deletion are also effective data augmentation strategies [15, 60]. Motivated by these findings, we design a dual-view multimodal training strategy that aligns original and augmented code–text pairs with two CLIP losses and applies consistency regularization to keep nearby views close in the shared embedding space. Together, these mechanisms

Dong et al.

provide richer training supervision and reduce reliance on shallow code features, thereby improving generalization [30, 67]. We evaluate M ULTI V UL on two widely used vulnerability detection benchmarks, DiverseVul [9] and Devign [68], across four representative code LLMs, including DeepSeek-Coder-6.7B [24], Qwen2.5-Coder-7B [3], StarCoder2-7B [36], and CodeLlama-7B [46]. Experimental results show that M ULTI V UL consistently outperforms prompting-based and code-only Fine-Tuning baselines, improving F1 by up to 27.07% and 13.37%, respectively. Ablations confirm the contribution of augmented alignment and consistency regularization, while OOD and latency results show that M ULTI V UL improves cross-dataset generalization and preserves efficient code-only inference. Our contributions are summarized as follows: • We propose M ULTI V UL, a multimodal vulnerability detection framework that leverages automatically generated comments during training without requiring text input during inference. • We design a dual-view training strategy that aligns original and augmented code–text pairs and stabilizes representations with cross-view consistency regularization. • We conduct extensive experiments on two benchmarks and four code LLMs, demonstrating consistent improvements in effectiveness, OOD generalization, and inference efficiency.

Background and Related Work

This section reviews three areas related to M ULTI V UL, including software vulnerability detection, multimodal contrastive learning, and data augmentation for code and text. We highligh how M ULTI V UL differs from existing vulnerability detection methods.

2.1

Software Vulnerability Detection

Software vulnerability detection is a long-standing and practically important problem in software security. In real-world software systems, vulnerabilities rarely appear as explicit syntactic errors. Instead, they often arise from semantic errors, implicit developer assumptions, or unsafe interactions among APIs and data manipulations. This makes vulnerability detection particularly challenging in large and evolving projects, where software must interoperate with third-party components, span diverse coding conventions, and satisfy security requirements with incomplete context [25]. Traditional vulnerability detection methods, such as static analysis and dynamic testing, remain fundamental in security practice because they rely on symbolic execution, formal verification, or fuzzing [4], which provide interpretable analyses and can be highly effective for identifying known types of software weaknesses [49]. However, they often face well-known limitations in scalability, coverage, and false positives, especially when applied to large and complex source code projects. These challenges have motivated learning-based methods that derive representations directly from source code and use data-driven models for vulnerability detection. Prior work has explored diverse learning-based architectures for vulnerability detection, including task-specific neural models and pre-trained code models. Representative ones include LineVul [21] and VulBERTa [27]. More recently, code LLMs (e.g., CodeLlama and StarCoder) have emerged as increasingly strong baselines for vulnerability detection and related software engineering tasks, because

Learning Generalizable Multimodal Representations for Software Vulnerability Detection

they support strong code modeling capacity and adaptability under both fine-tuning and instruction-based prompting [13, 29, 50, 66]. At the same time, recent studies have begun to investigate additional modalities (e.g., textual data) beyond a single code modality for improving semantic modeling through contrastive and multimodal learning [10, 35]. These directions are closely related to our work, but differ from it in several important respects. For example, CLeVeR [35] relies on vulnerability descriptions during inference, whereas other representative vulnerability detection methods are built on older pre-trained code models or different training formulations, such as CodeBERT-based line-level detection, RoBERTa-based vulnerability pre-training, or multi-task instruction tuning [18, 21, 27, 35]. Directly comparing M ULTI V UL with such methods would make it hard to tell whether performance differences come from the proposed training strategy or simply from differences in inference inputs and base model setups. Therefore, in the main experiments, we compare M ULTI V UL with strong code-only fine-tuning and prompting-based baselines using the same code LLM, so that the comparison focuses on the learning framework itself rather than differences in inference input or the underlying code LLM. In addition, we include a standard code–text CLIP baseline in the ablation study to isolate the contribution of basic code–text contrastive alignment relative to M ULTI V UL.

2.2

Multimodal Contrastive Learning

Contrastive learning [26] has become a widely used paradigm for representation learning in both computer vision (CV) and natural language processing (NLP). As one of the representative methods, CLIP [43] aligns paired image and text representations in a shared embedding space and has shown strong transferability across downstream tasks. Multimodal contrastive learning [64] extends this idea to settings with multiple data modalities. Its core goal is to bring semantically matched inputs from different modalities closer while pushing mismatched pairs apart. In this way, it provides an effective mechanism for aligning representations across modalities, such as image data and text data, and has been shown to improve generalization by encouraging shared semantic structure in the learned embedding space [30, 67]. Inspired by these advances, researchers in the software engineering community have recently begun to explore multimodal and contrastive learning for code-related tasks, including vulnerability detection [31, 35]. The main intuition is that additional natural language information (e.g., code comments) can complement source code and help models learn richer semantic representations than code alone. However, applying multimodal contrastive learning to vulnerability detection remains nontrivial. Code and text differ substantially in granularity and information density, making precise semantic alignment difficult in this task [35]. Moreover, recent multimodal vulnerability detection studies [61] suggest that simple fusion strategies may be insufficient to preserve discriminative security signals under practical conditions. These challenges become especially important when comments are noisy or when deployment must remain code-only rather than rely on description-side inputs. These observations motivate our design of M ULTI V UL. Instead of relying on a single original code–text pair, M ULTI V UL constructs original and augmented views of both code and text, optimizes their

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

alignment with two CLIP losses, and further stabilizes the shared space with consistency regularization. Unlike previous methods that require textual inputs during inference, M ULTI V UL uses automatically generated and critique-refined code comments only during training and operates on code-only inputs at inference.

2.3

Data Augmentation

Data augmentation is a widely used technique for improving model generalization by providing diverse yet related input data variants during training. In NLP, data augmentation methods include lightweight token-level perturbations such as synonym replacement, random swap, and random deletion [19, 60]. These transformation methods are useful because they are simple, label-preserving, and easy to apply at scale [19]. Data augmentation has also been actively studied in software engineering. Prior work has explored a variety of code transformations, including identifier renaming, refactoring, adversarial training, and interpolation-based synthesis [2, 6, 22, 57, 62]. This design principle for strategy is to create alternative views of the same source code while preserving as much of the original semantics as possible [34]. These studies suggest that data augmentation can be useful for code representation learning, but they also show that not all data augmentation strategies are equally effective or equally practical [15]. In particular, existing empirical work reports that traditional code refactoring-based data augmentation often yields limited and inconsistent gains for code modeling [5, 15, 17, 63]. Although semanticspreserving code transformations are intuitively appealing, they are often tied to language-specific transformation rules and nontrivial rewriting pipelines [34]. Moreover, several data augmentation methods are constrained by the task formulation. For example, MixCode [14] is designed for classification and relies on one-hot labels during interpolation. These dependencies make such methods less flexible across languages, tools, and downstream tasks. More recent studies revisit simpler perturbation-based data augmentation strategies and show that lightweight transformations (e.g., randomly swapping two statements) can still provide useful training signals for code understanding and robustness, even when they introduce only small local changes rather than fully semantics-preserving rewrites [15, 16]. This observation is especially relevant for M UL TI V UL , where the goal is not to generate fully independent new programs, but to construct nearby views that encourage stable representation learning. Motivated by these findings, our work adopts augmented views for both code and text. We focus on lightweight perturbations, in particular random swap (RS) and random deletion (RD), because they are simple, model-agnostic, and do not rely on language-specific refactoring rules or external program analyses. Moreover, they are also well-suited to contrastive and consistency-based training, as they create nearby but non-identical views of the same sample. In our design, this property matters more than strict semantic preservation, since the augmented views are used to regularize representation learning rather than replace the original data. As a result, RS and RD provide a practical way to construct dual views in M ULTI V UL.

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Dong et al.

4.1

Problem Formulation

We study function-level vulnerability detection. Given a source code function 𝑐, the detector predicts a binary label 𝑦 ∈ {0, 1}, where 𝑦 = 1 denotes a vulnerable function and 𝑦 = 0 denotes a nonvulnerable function. During training, each function can be paired with an automatically generated natural language comment 𝑡, forming a multimodal training set: 𝑁 D = {(𝑐𝑖 , 𝑡𝑖 , 𝑦𝑖 )}𝑖=1 ,

(1)

where 𝑐𝑖 is the source code function, 𝑡𝑖 is its generated code comment, 𝑦𝑖 is the vulnerability label, and 𝑁 is the dataset size. At inference time, the detector receives only the source code function 𝑐𝑖 . It first obtains a code representation and outputs a vulnerability probability: 𝑝𝑖 = 𝑓 (𝑐𝑖 ),

(2)

where 𝑝𝑖 ∈ (0, 1) denotes the predicted probability that 𝑐𝑖 is vulnerable. The final prediction is obtained by thresholding: 𝑦ˆ𝑖 = I[𝑝𝑖 > 𝛿] ,

(3)

where I[·] is the indicator function and 𝛿 is the decision threshold, set to 0.5 by default. This formulation reflects practical secure code review and repositoryscale scanning, where source code is available but high-quality textual context may be missing, noisy, or inconsistent. M ULTI V UL therefore uses multimodal supervision only during training, while preserving code-only inference at deployment time. It does not require comments, prompts, commit messages, vulnerability reports, or other external textual artifacts during inference.

M ULTI V UL

M ULTI V UL is a multimodal contrastive learning framework for software vulnerability detection. Its core idea is to exploit LLMgenerated code comments as auxiliary supervision during training. In this way, the model learns vulnerability-relevant semantics from both code and text, while retaining a lightweight code-only inference pipeline for practical deployment. As illustrated in Figure 2, M UL TI V UL operates in two phases. During training, each source code function and its generated comment are treated as paired multimodal inputs. Both code and text are further augmented to construct an additional view, resulting in original and augmented code–text pairs. These inputs are encoded by a dual-encoder architecture and projected into a shared embedding space. The model is optimized with dual-CLIP alignment, namely original code–text alignment and augmented code–text alignment, together with cross-view consistency that stabilizes learning across different views. Although the models in M ULTI V UL are decoder-only code LLMs (e.g., Qwen2.5-Coder), we use the term encoder to describe their functional role in representation learning, that is, each code LLM maps an input function or comment into a contextual hidden representation, which is then pooled and projected into the shared embedding space. Accordingly, dual-encoder in our framework refers to two modality-specific representation producers for code and text, rather than to the original Transformer encoder architecture [56]. A vulnerability classifier is trained on top of the learned code representation and is applied to code representations during inference.

Multimodal Data Construction

In real-world software repositories, source code often lacks highquality natural language comments, which limits the semantic information available to learning-based vulnerability detection [51]. To enrich training-time supervision, M ULTI V UL augments each code with an automatically generated comment and further constructs augmented code–text views for multi-view learning. For each code 𝑐𝑖 , we apply a strong instruction-tuned code LLM, Qwen2.5-Coder-32B-Instruct, to generate a concise one-sentence comment 𝑡𝑖 that summarizes the functionality expressed by the code: 𝑡𝑖 = LLM_gen(𝑐𝑖 ).

(4)

To improve faithfulness and reduce speculative descriptions, comment generation follows a critique prompting procedure rather than direct single-pass generation [54]. Specifically, the model first produces a draft summary of the code, then reviews its own output to identify unsupported or overly speculative claims, and finally generates a revised one-sentence comment that describes only the functionality explicitly shown in the code. In the default setting used in this work, the prompt explicitly instructs the model not to mention security or vulnerabilities, and not to infer properties such as validation, error handling, permission checks, or safety guarantees unless they are explicitly present in the code. This procedure is fully automatic. More details are provided in Appendix A. These automatically generated and critique-refined comments provide semantically enriched supervisory information that links code tokens with higher-level program semantics, thereby supporting cross-modal alignment. We additionally generate perturbed code and text views using a combination of random swap (RS) and random deletion (RD), as lightweight perturbations have been shown in existing empirical studies [15, 16] to provide useful training information for source code representation learning. Compared with more aggressive transformation methods such as back-translation and MixCode [14], RS and RD are simple, model-agnostic perturbations that introduce limited local changes, making them suitable for constructing nearby but non-identical views for contrastive alignment and consistency regularization. Let 𝑐˜𝑖 and 𝑡˜𝑖 denote the augmented code and text, respectively: 𝑐˜𝑖 = Aug𝛼 (𝑐𝑖 ),

𝑡˜𝑖 = Aug𝛼 (𝑡𝑖 ),

(5)

where Aug𝛼 (·) applies RS and RD with augmentation strength 𝛼. Here, 𝛼 controls how far the augmented view deviates from the original input. We study the effect of different 𝛼 values in Section 6.4. This stage produces paired original and augmented code–text views for each training instance. The generated comments provide semantic supervision for code understanding, while RS and RD provide controlled view diversity for subsequent dual-encoder learning and dual-CLIP alignment.

4.2

Dual-Encoder and Dual-CLIP Alignment

Given the training set D defined in Section 3 and the augmented views (𝑐˜𝑖 , 𝑡˜𝑖 ) introduced in Section 4.1, M ULTI V UL adopts dualencoder learning and optimizes the model with dual-CLIP alignment and cross-view consistency regularization. During training, both the original and augmented code–text pairs are used to improve multimodal representation learning.

Learning Generalizable Multimodal Representations for Software Vulnerability Detection

Input Code Repository

I. Code & Label Collection

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

III. Data Augmentation

II. Comment Generation

IV. Dual-Encoder

V. Training & Alignment

Source Code

Source Code Source Code Source Code Source Code void copy_input(char Source Code *src) { Source Code voidbuf[16]; copy_input(char char Source Code *src) { Source Code char buf[16]; strcpy(buf, src); void copy_input(char *src) { strcpy(buf, src); } char buf[16]; }

Vulnerability Label Collect }

strcpy(buf, src);

SourceCode Code Source Augmented Source Code code Source Code Source Code Source Code *src) { void copy_input(char

GeneratedComment Comment Generated

Generated code Comment

Code Encoder

void copy_input(char copy_input(char *src)*src) { void { char buf[16]; char buf[16]; strcpy(buf, src); strcpy(buf, src); strcpy(buf, src); } } char buf[16]; }

Thefunction functioncopies copiesinput input data datainto intoaa The fixed-size buffer without checking thea The function copies input data into fixed-size buffer without checking the lengthofofthe the input, whichcan cancause causethe fixed-size buffer without checking length input, which aa bbuuf ff ef er r oovveer fr lfol oww vvuul nl neer raabbi liil ti yt y. .

Generate length of the input, which can cause

Label: 1 (Vulnerable)

Augment

Shared Embedding Space

GeneratedComment Comment Generated

Augmented Text

Text Encoder

Thefunction function copies copies input input data data intoaa The The function copies inputchecking data into intothe a fixed-size buffer without fixed-size buffer without checking the buffer checking lengthofwithout ofthe the input, input, whichthe canlength causeof length which can cause aa buuf ff ef erwhich r oovveercause ul nl neer aroverflow abbi li il ti yt y. . binput, fr fl ol oww a vvubuffer

Augment

Original Code--Text Alignment

Encode

Label: 1 (Vulnerable) Label: 1 (Vulnerable)

a buffer overflow vulnerability.

Label: 1 (Vulnerable) Label: 11(Vulnerable) Label: (Vulnerable)

Code Projection

Optimize

Text Projection

Cross-View Consistency Regularization

vulnerability.

Shared Parameters

Output Prediction

Augmented Code--Text Alignment

IV. Vulnerability Classification

III. Code Representation

Classifier

Code Representation Represent

Predict

Vulnerability (0 / 1)

II. Shared Code Encoder

I. Inference Input Source Code Source Code Source Code Source Code Raw Source Code Source Source Code Code Source Code Source Code Source Source Code Code Raw Raw Source Source Code Code Source Code

SourceCode Code Source Code Source Source Code Source Code Source Source Code Code Code Raw Source Source Code Source SourceCode Code

Source Code Source SourceCode Code Source Code Source Code

Code Encoder

Encode

Input

(from Shared Code Encoder )

def safe_copy(buf,len) { { def safe_copy(buf,len) def >= safe_copy(buf,len) { if len MAX_SIZE: if len >= MAX_SIZE: len >= MAX_SIZE: return error returnifreturn error error …} …}…}

Figure 2: Architecture overview of M ULTI V UL. Dual-encoder learning. For each mini-batch of size 𝐵 that is 𝐵 , we use a code encoder 𝑓 and a text denoted as {(𝑐𝑖 , 𝑡𝑖 , 𝑐˜𝑖 , 𝑡˜𝑖 , 𝑦𝑖 )}𝑖=1 𝜃 encoder 𝑔𝜙 to encode the original and augmented inputs: h𝑐𝑖 = 𝑓𝜃 (𝑐𝑖 ),

h𝑖𝑡 = 𝑔𝜙 (𝑡𝑖 ),

h̃𝑐𝑖 = 𝑓𝜃 (𝑐˜𝑖 ),

h̃𝑖𝑡 = 𝑔𝜙 (𝑡˜𝑖 ),

(6)

where h𝑐𝑖 and h𝑖𝑡 denote the hidden representations of the original code and text, and h̃𝑐𝑖 and h̃𝑖𝑡 denote those of the augmented code and text, respectively. To perform multimodal contrastive alignment, we project code and text representations into a shared embedding space using modalityspecific projection heads, followed by ℓ2 normalization: z𝑐𝑖 = norm(𝑊𝑐 h𝑐𝑖 ), z̃𝑐𝑖 = norm(𝑊𝑐 h̃𝑐𝑖 ),

z𝑖𝑡 = norm(𝑊𝑡 h𝑖𝑡 ), z̃𝑖𝑡 = norm(𝑊𝑡 h̃𝑖𝑡 ),

Given a similarity matrix 𝑆 ∈ R𝐵×𝐵 , we optimize it with a standard symmetric InfoNCE objective [42]: " # 𝐵 exp(𝑆𝑖𝑖 ) exp(𝑆𝑖𝑖 ) 1 ∑︁ − log Í𝐵 Lclip (𝑆) = − log Í𝐵 . 2𝐵 𝑖=1 𝑗=1 exp(𝑆𝑖 𝑗 ) 𝑗=1 exp(𝑆 𝑗𝑖 ) (9) The first term performs code-to-text matching by treating the 𝑖-th code embedding as the query and its paired text embedding as the positive target, while the second term performs the symmetric textto-code matching. The dual-CLIP alignment objective is then defined as: orig

orig

(7)

where 𝑊𝑐 and 𝑊𝑡 are the projection heads shared across views for code data and text data, respectively. Dual-CLIP alignment. Within each mini-batch of size 𝐵, M ULTI V UL computes two batch-wise code–text similarity matrices, 𝑆 orig ∈ R𝐵×𝐵 and 𝑆 aug ∈ R𝐵×𝐵 , for the original and augmented code–text pairs, respectively. Each entry measures the similarity between the code embedding of the 𝑖-th sample and the text embedding of the 𝑗-th sample in the mini-batch:

aug

Ldual-clip = 𝜆clip Lclip (𝑆 orig ) + 𝜆clip Lclip (𝑆 aug ),

(10)

aug

where 𝜆clip and 𝜆clip control the contributions of the original-view and augmented-view alignment losses, respectively. This dual-CLIP design enforces cross-modal alignment for both the original code–text pair (𝑐𝑖 , 𝑡𝑖 ) and the augmented pair (𝑐˜𝑖 , 𝑡˜𝑖 ), thereby improving the generalization of the shared embedding space to lightweight perturbations in both code and text.

(8)

Cross-view consistency regularization. To further stabilize representation learning across the original and augmented views, M UL TI V UL introduces a consistency regularization term in the shared embedding space presented in Figure 2: ! 𝐵 𝐵 1 1 ∑︁ 𝑐 1 ∑︁ 𝑡 𝑐 2 𝑡 2 (11) Lcons = z − z̃𝑖 2 + z − z̃𝑖 2 , 2 𝐵 𝑖=1 𝑖 𝐵 𝑖=1 𝑖

where 𝑖, 𝑗 ∈ {1, . . . , 𝐵} index samples within the mini-batch, 𝛼 = exp(𝑠) is a learnable logit scale, and (·) ⊤ denotes the inner product between two ℓ2 -normalized embeddings. The diagonal entries correspond to matched code–text pairs from the same training instance, whereas the off-diagonal entries correspond to mismatched pairs within the mini-batch.

where z𝑐𝑖 and z̃𝑐𝑖 denote the projected embeddings of the original and augmented code, respectively. z𝑖𝑡 and z̃𝑖𝑡 denote those of the original and augmented text. Here ∥ · ∥ 22 denotes the squared ℓ2 distance, and the factor averages the code-view and text-view consistency terms. This regularization encourages augmented views to remain close to their original counterparts in the shared embedding space, promoting local smoothness under lightweight perturbations.

orig

= 𝛼 (z𝑐𝑖 ) ⊤ z𝑡𝑗 , aug 𝑆𝑖 𝑗 = 𝛼 ( z̃𝑐𝑖 ) ⊤ z̃𝑡𝑗 ,

𝑆𝑖 𝑗

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Dong et al.

Vulnerability Classifier. In addition to contrastive alignment, M UL TI V UL trains a supervised binary classifier using the projected code

representation. For each original code input 𝑐𝑖 , the classifier takes the projected code embedding z𝑐𝑖 as input and outputs a scalar logit: 𝑠𝑖 = MLP(z𝑐𝑖 ),

𝑝𝑖 = 𝜎 (𝑠𝑖 ),

(12)

where MLP(·) denotes a multi-layer perceptron classification head, 𝜎 (·) denotes the sigmoid function, and 𝑝𝑖 ∈ (0, 1) is the predicted probability that 𝑐𝑖 is vulnerable. We optimize the binary cross-entropy loss: 𝐵

Lcls = −

1 ∑︁ [𝑦𝑖 log 𝑝𝑖 + (1 − 𝑦𝑖 ) log(1 − 𝑝𝑖 )] , 𝐵 𝑖=1

(13)

where 𝑦𝑖 ∈ {0, 1} is the ground-truth vulnerability label. Overall, M ULTI V UL uses dual encoders to learn code and text representations, employs dual-CLIP to align both original and augmented code–text pairs, and applies consistency regularization to stabilize representations across views.

Experimental Setup

We implement all methods in PyTorch and conduct experiments on a server equipped with 8× NVIDIA H200 GPUs (143,771 MiB memory per GPU), 2× Intel Xeon Platinum 8558 CPUs, and 1.8TB RAM. For a fair comparison, all baselines and M ULTI V UL use the same code LLM and data splits. Furthermore, detailed hyperparameters and training configurations are provided in Appendix B. Datasets. We conduct experiments on two widely used vulnerability detection benchmarks (i.e., Devign [68] and DiverseVul [9] ). Table 1 reports the statistics of the datasets used in our experiments across different splits. Specifically, the table includes the number of functions (#Funcs), average lines of code (Avg LOC), average non-empty lines of code (Avg nLOC), average number of tokens (Avg Tokens), and the number of non-vulnerable and vulnerable samples in each split. Models. We conduct experiments on four representative opensource code LLMs. Especially, we focus on 7B-scale code LLMs in our experiments because they offer a practical balance between effectiveness and deployment cost, which is particularly relevant in real-world industrial scenarios where compute and latency constraints matter. Prior work [37] has shown that reducing model size can substantially lower downstream fine-tuning and inference costs, while carefully designed 7B-scale models can still achieve strong performance with efficient inference. We adopted CodeLlama-7B [46], DeepSeek-Coder-6.7B [24], Qwen2.5-Coder-7B [3], and StarCoder2-7B [36]. These models are all strong code-oriented LLMs, but differ in their pretraining corpora, architectural design, and data coverage. CodeLlama is built on the Llama 2 family and is widely used for code generation and understanding. DeepSeek-Coder is trained on a large multilingual code corpus covering many programming languages. Qwen2.5-Coder is a code-specialized continuation-pretrained model over large-scale GitHub repositories, and StarCoder2 is developed by the BigCode project and trained on the Software Heritage archive, offering broad coverage of real-world code. Baselines. We compare M ULTI V UL with representative prompting-based and training-based baselines. For prompting-based baselines, we evaluate Zero-shot, One-shot, and Three-shot inference,

as well as Chain-of-Thought (CoT) prompting variants, including Zero-shot-CoT, One-shot-CoT, and Three-shot-CoT, using the same code LLM. For training-based baselines, we report Fine-Tuning, which performs supervised fine-tuning on code inputs only using the same code LLM, without multimodal alignment. All methods are evaluated on the same data splits and with the same metrics. At test time, M ULTI V UL performs code-only inference, as text views are used exclusively during training for multimodal alignment and consistency regularization. Evaluation Metrics. We formulate software vulnerability detection as a binary classification task, treating vulnerable as the positive class. Accordingly, we report Accuracy, Precision, Recall, and F1, where Precision, Recall, and F1 are computed with respect to the positive class. All metrics are computed on the testing dataset using a fixed decision threshold of 0.5.

Results

To evaluate the effectiveness and practicality of M ULTI V UL, we conduct comprehensive experiments and aim to answer the following research questions (RQs): • RQ1: How effective is M ULTI V UL in detecting software vulnerabilities? • RQ2: How does each key component contribute to M ULTI V UL? • RQ3: How well does M ULTI V UL generalize under cross-dataset distribution shift? • RQ4: How sensitive is M ULTI V UL to the scale of the code comment generation LLM and the data augmentation strength? • RQ5: How efficient is M ULTI V UL during inference?

6.1

RQ1: Effectiveness of M ULTI V UL

Table 2 summarizes the effectiveness of M ULTI V UL and all baselines on two datasets under four code LLMs. Overall analysis. Across all code LLMs and both datasets, M UL TI V UL consistently achieves the best Accuracy and F1, demonstrating its effectiveness for software vulnerability detection. Specifically, standard prompting-based methods (e.g., Zero-shot) are unstable and generally underperform, whereas CoT prompting substantially improves Recall, often exceeding 90% on DiverseVul and reaching competitive Recall on Devign (e.g., 65.10% with Three-shot-CoT on StarCoder2-7B and 56.36% with One-shot-CoT on CodeLlama7B). However, these Recall gains usually come at the expense of lower Precision, leading to only moderate F1, with CoT promptingbased methods averaging 65.63% on DiverseVul and 42.04% on Devign across the four code LLMs. This suggests that promptingbased methods, especially CoT prompting, bias predictions toward the vulnerable class, yielding high Recall but relatively low Precision. This observation is consistent with findings from previous studies [32, 38, 41] that CoT prompting can strengthen the ability of vulnerability reasoning by eliciting more explicit intermediate reasoning, while prompt-based LLM detection still exhibits clear limitations in Precision and overall Accuracy. In contrast, M ULTI V UL substantially outperforms the strongest prompting-based baseline across all code LLMs and datasets. Concretely, on DiverseVul, M ULTI V UL improves F1 by 24.35%, 27.07%, 27.06%, and 25.02% over the best prompting-based baseline for DeepSeek-Coder-6.7B, Qwen2.5-Coder-7B, StarCoder2-7B, and

Learning Generalizable Multimodal Representations for Software Vulnerability Detection

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Table 1: Dataset statistics used in our experiments. LOC counts include blank lines. nLOC denotes non-empty lines of code. Ratio is reported as non-vulnerable: vulnerable. Dataset

Split

#Funcs

Avg LOC

Avg nLOC

Avg Tokens

#Non-Vulnerable

#Vulnerable

Ratio

Devign

Training Valitation Test

21,854 2,733 2,731

110.56 110.68 114.18

51.01 51.52 52.65

431.11 433.70 436.96

11,886 1,486 1,485

9,968 1,247 1,246

1.19:1 1.19:1 1.19:1

DiverseVul

Training Validation Test

9,984 1,248 1,250

18.60 18.82 18.71

16.43 16.67 16.49

117.26 118.73 117.38

4,992 624 625

1.00:1 1.00:1 1.00:1

Table 2: Comparison of baselines and M ULTI V UL on DiverseVul and Devign across different code LLMs. All metrics are reported in percentage (%). Fine-Tuning refers to standard supervised fine-tuning on code inputs only. Bold marks the best value per column within each code LLM. Blue shading highlights the highest Accuracy and F1 for each code LLM–dataset pair. DeepSeek-Coder-6.7B Methods

DiverseVul Accuracy Precision Recall

Zero-shot One-shot Three-shot Zero-shot-CoT One-shot-CoT Three-shot-CoT Fine-Tuning MultiVul

49.76 48.72 49.44 48.48 49.68 50.16 78.72 90.00

48.45 47.74 47.98 49.20 49.84 50.08 81.88 92.04

Qwen2.5-Coder-7B Devign

7.52 27.04 13.28 93.92 97.92 99.52 73.76 89.92

F1 13.02 34.53 20.80 64.58 66.06 66.63 77.61 90.98

DiverseVul

Accuracy Precision Recall 54.51 54.73 55.53 51.14 47.69 52.75 56.19 60.95

51.16 61.90 56.56 45.58 45.82 47.72 55.51 65.73

7.06 2.09 11.08 26.44 33.10 32.92 20.22 30.18

F1 12.41 4.04 18.52 33.47 38.43 38.96 29.64 41.36

Accuracy Precision Recall 50.40 50.88 55.12 49.92 49.84 50.08 88.72 93.60

63.16 52.00 58.21 49.96 49.92 50.04 92.61 93.74

1.92 22.88 36.32 98.72 98.72 99.20 84.16 93.44

StarCoder2-7B Methods

DiverseVul Accuracy Precision Recall

Zero-shot One-shot Three-shot Zero-shot-CoT One-shot-CoT Three-shot-CoT Fine-Tuning MultiVul

47.60 47.36 49.44 50.16 53.12 48.32 84.00 91.84

48.60 48.32 49.64 50.09 51.83 49.07 87.35 93.37

83.20 76.16 76.96 92.64 88.48 88.64 79.52 91.52

61.36 59.13 60.35 65.02 65.37 63.17 83.25 92.43

Accuracy Precision Recall 54.43 54.51 54.51 49.52 50.33 50.99 63.07 67.11

DiverseVul

Accuracy Precision Recall 50.62 51.28 54.73 53.92 48.28 46.37 62.20 64.69

F1 3.73 31.78 44.73 66.34 66.31 66.52 88.18 93.59

55.56 57.14 56.25 45.45 45.57 45.51 58.51 64.75

0.80 1.28 1.44 52.97 45.43 37.40 65.65 76.21

F1 1.58 2.51 2.82 48.93 45.50 41.06 61.88 70.01

CodeLlama-7B Devign

Devign

38.36 44.47 52.14 48.91 45.07 40.08 61.92 62.57

13.48 27.13 9.79 21.51 60.83 65.10 44.62 56.34

CodeLlama-7B, respectively. The corresponding Accuracy gains are also substantial, reaching up to 41.52%. On Devign, M ULTI V UL also consistently improves over the best prompting-based baseline, with the largest F1 and Accuracy gains reaching 21.08% and 12.60%, respectively. These results better reflect practical vulnerability detection performance under class imbalance. Although prompting-based methods (e.g., One-shot-CoT) can sometimes achieve high Recall, M ULTI V UL provides a substantially better trade-off between Precision and Recall, leading to stronger overall performance. We further compare M ULTI V UL with the training-based baseline, namely standard Fine-Tuning. As shown in Table 2, M ULTI V UL consistently outperforms Fine-Tuning on all four code LLMs and both datasets. For instance, the largest F1 gains over Fine-Tuning reach 13.37% on DiverseVul and 11.72% on Devign. Averaged across the four code LLMs, M ULTI V UL improves over Fine-Tuning by 8.15% and 7.97% in F1 on DiverseVul and Devign, respectively. The corresponding average gains in Accuracy are 7.10% and 3.77%, respectively. In summary, compared with prompting-based baselines, M ULTI V UL avoids the common CoT failure problem of very high Recall coupled with low Precision [39]. Relative to supervised fine-tuning

F1 19.95 33.70 16.49 29.88 51.78 49.61 51.87 59.29

Accuracy Precision Recall 43.76 50.16 48.16 48.24 49.36 50.08 87.28 91.68

46.31 50.08 49.05 49.10 49.68 50.04 89.49 93.06

78.24 96.96 94.56 95.52 98.72 99.20 84.48 90.08

Devign F1 58.18 66.05 64.59 64.86 66.10 66.52 86.91 91.54

Accuracy Precision Recall 49.96 49.82 49.45 53.04 46.45 56.41 60.51 64.30

47.12 45.37 45.01 47.70 45.44 53.85 59.86 63.62

48.65 48.80 48.48 30.02 56.36 31.46 40.93 45.75

F1 47.87 46.02 46.68 36.85 50.31 39.72 48.62 53.22

on code inputs only, it consistently improves both Accuracy and F1, demonstrating the value of multimodal alignment for software vulnerability detection. Visualizing principal component analysis. To further understand why M ULTI V UL achieves superior performance, we visualize the code embeddings after dimensionality reduction on the testing dataset of DiverseVul and Devign using Principal Component Analysis (PCA), as shown in Figure 3. Figures (a)-(c) show the results on DiverseVul using Qwen2.5-Coder-7B, while Figures (d)-(f) show the corresponding results on Devign using the same code LLM. In both datasets, the Zero-shot embeddings form highly overlapping clusters where vulnerable and non-vulnerable samples are mixed, indicating that the pretrained code LLM lacks task-specific discriminative capacity. Fine-tuning produces clearer separation but still exhibits substantial overlap along the decision boundary, suggesting that single-modal supervision (i.e., code only) learns partial semantic distinctions but fails to capture deeper vulnerability semantics. In contrast, M ULTI V UL demonstrates distinctly separated clusters between vulnerable and non-vulnerable samples, with compact intraclass distributions and clear inter-class margins. This well-structured

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

representation space implies that the multimodal alignment (i.e., code and its textual comments) effectively enhances the semantic grounding of LLMs [12]. By incorporating the paired textual descriptions, M ULTI V UL captures semantic intent that is difficult to infer from code tokens alone, e.g., error handling in memory operations. Overall, M ULTI V UL produces a more structured and separable embedding space, consistent with the Accuracy and F1 gains observed over both prompting-based and fine-tuning baselines. Answer to RQ1: M ULTI V UL consistently outperforms promptingbased and code-only Fine-Tuning baselines across four code LLMs on DiverseVul and Devign, with up to 27.07% and 13.37% F1 improvements, respectively. The PCA results further show that M ULTI V UL learns more separable vulnerability representations.

6.2

Dong et al.

signal by introducing stronger multi-view correspondence between code data and text data, while consistency regularization stabilizes learning across views. Their combination produces the most reliable improvements in both Accuracy and F1, especially on the more challenging Devign dataset, where partial multimodal designs are considerably less stable. Additionally, to keep the main text focused, we report only Accuracy and F1 in Table 3 as they most directly summarize overall effectiveness and the balance between Precision and Recall. The complete results, including Precision and Recall, are provided in Appendix C. Answer to RQ2: Both augmented code–text alignment and crossview consistency contribute to M ULTI V UL. For instance, on Devign, the largest F1 gains reach 18.31% over removing augmented alignment and 23.31% over removing consistency.

RQ2: Ablation Study

We further analyze the contributions of the two key components in M ULTI V UL, namely Augmented Code–Text Alignment and CrossView Consistency Regularization, as illustrated in Figure 2. Starting from standard Fine-Tuning, we first add the original code–text alignment (w/o Augmented Alignment), then further introduce the augmented alignment while removing consistency regularization (w/o Consistency), and finally obtain M ULTI V UL by enabling both components. Overall, the full training method, i.e., M ULTI V UL, delivers the strongest and most consistent performance across code LLMs and datasets. Table 3 shows that, on DiverseVul, M ULTI V UL consistently outperforms both ablated variants across all four code LLMs in terms of both Accuracy and F1. This shows that augmented alignment and cross-view consistency provide complementary benefits. Compared with w/o Augmented Alignment, M ULTI V UL improves F1 by up to 3.06%, and compared with w/o Consistency, the gain reaches up to 3.91%. A similar trend is observed for Accuracy, with gains reaching up to 2.40% over w/o Augmented Alignment and 3.52% over w/o Consistency. Overall, these results indicate that both augmented alignment and cross-view consistency contribute positively to DiverseVul, with the effect of cross-view consistency being especially evident on StarCoder2-7B, where removing consistency leads to a notable drop in both Accuracy and F1. On Devign, the effects of w/o Augmented Alignment and w/o Consistency become even more pronounced. M ULTI V UL again achieves the best Accuracy and F1 across all four code LLM, and the gaps relative to the ablated variants are substantially larger than on DiverseVul. For instance, for DeepSeek-Coder-6.7B, F1 increases from 23.05% with w/o Augmented Alignment and 18.05% with w/o Consistency to 41.36% with M ULTI V UL, corresponding to gains of 18.31% and 23.31%, respectively. This indicates that neither partial variant is sufficient on this more challenging dataset. A similar phenomenon is observed for StarCoder2-7B, where F1 rises from 41.98% and 55.73% to 59.29%. Qwen2.5-Coder-7B is comparatively more stable, but M ULTI V UL still achieves the best overall result, improving F1 from 66.18% and 65.95% to 70.01%. The Accuracy trends are consistent with the F1 results, with gains reaching up to 2.57% over w/o Augmented Alignment and 2.85% over w/o Consistency. In conclusion, these ablation results show that augmented code– text alignment and cross-view consistency are both important components of M ULTI V UL. Augmented alignment enriches the supervision

6.3

RQ3: Out-of-Distribution Generalization

Different from in-distribution (ID) evaluation in Section 6.2, where the training, validation, and testing datasets are drawn from the same benchmark split, out-of-distribution (OOD) evaluation in our study focuses on cross-dataset generalization. Specifically, we train on Devign and evaluate on the DiverseVul testing dataset, and vice versa. We adopt this protocol because cross-project and cross-dataset transfer provides a realistic and widely used setting for assessing the performance of OOD generalization in software vulnerability detection [18, 40, 65], where distribution shifts naturally arise from differences in coding standards, library usage, and project structure. This design is particularly meaningful because Devign is constructed from four diversified C projects [68], whereas DiverseVul spans a much broader range of projects and CWEs, thereby inducing a stronger cross-dataset shift [9]. Table 4 reports the OOD results under this cross-dataset protocol. The column labels indicate the source training dataset. For example, DiverseVul means training on DiverseVul and testing on Devign. Overall, M ULTI V UL consistently achieves the best performance in terms of Accuracy and F1 across all four code LLMs under both cross-dataset settings, showing that the proposed multimodal training strategy improves generalization under distribution shift. When training on DiverseVul and testing on Devign, the gains over Fine-Tuning are moderate but consistent. The largest F1 gain appears with StarCoder2-7B, where F1 improves from 59.88% to 63.40%, yielding a 3.52% increase. For Accuracy, CodeLlama-7B shows the largest gain, increasing from 47.55% to 49.95%, with a 2.40% improvement. When training on Devign and testing on DiverseVul, the gains are generally larger. DeepSeek-Coder-6.7B obtains the largest F1 improvement, increasing from 33.02% to 50.74%, with a 17.72% gain. CodeLlama-7B achieves the largest Accuracy gain, improving from 54.16% to 59.92%, producing a 5.76% increase. These results suggest that M ULTI V UL provides especially strong benefits under more challenging cross-dataset transfer. The ablation results further show that the two components play complementary roles under OOD shift. Specifically, removing augmented alignment generally weakens performance and makes the gains less stable. For example, when training on Devign and testing on DiverseVul, F1 on DeepSeek-Coder-6.7B decreases from 50.74% to 37.66%, a drop of 13.08%. Accuracy on CodeLlama-7B also falls

Learning Generalizable Multimodal Representations for Software Vulnerability Detection

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

(a) Zero-shot

(b) Fine-Tuning

(d) Zero-shot

(e) Fine-Tuning

(f) MultiVul

Figure 3: Visualization of code embeddings after dimension reduction using Principal Component Analysis (PCA). Figures (a)–(c) show embeddings from Zero-shot, Fine-Tuning, and M ULTI V UL on DiverseVul, while Figures (d)–(f) present the corresponding results on Devign. Model: Qwen2.5-Coder-7B. Table 3: Ablation study on DiverseVul and Devign across four code LLMs. All metrics are reported in percentage (%). Fine-Tuning refers to standard supervised fine-tuning on code inputs only. w/o Augmented Alignment removes the augmented code–text alignment. w/o Consistency removes the cross-view consistency regularization. Bold marks the best value per column within each code LLM. Blue shading highlights the highest Accuracy and F1 for each code LLM–dataset pair. DeepSeek-Coder-6.7B Methods

Fine-Tuning w/o Augmented Alignment w/o Consistency MultiVul

DiverseVul

Qwen2.5-Coder-7B

Devign

DiverseVul

StarCoder2-7B

Devign

DiverseVul

CodeLlama-7B

Devign

DiverseVul

Devign

Accuracy

78.72 89.60 89.84 90.00

77.61 89.29 89.55 90.98

56.19 58.90 58.10 60.95

29.64 23.05 18.05 41.36

88.72 92.64 92.96 93.60

88.18 92.73 92.70 93.59

63.07 65.86 65.20 67.11

61.88 66.18 65.95 70.01

84.00 89.44 88.32 91.84

83.25 89.37 88.52 92.43

62.20 62.12 62.64 64.69

51.87 41.98 55.73 59.29

87.28 91.28 91.52 91.68

86.91 91.31 91.52 91.54

60.51 64.03 63.74 64.30

48.62 46.69 45.78 53.22

Table 4: Ablation study of M ULTI V UL for OOD detection across four code LLMs. All metrics are reported in percentage (%). Fine-Tuning refers to standard supervised fine-tuning on code inputs only. w/o Augmented Alignment removes the augmented code–text alignment. w/o Consistency removes the cross-view consistency regularization. Bold marks the best value per column within each code LLM. Blue shading highlights the highest Accuracy and F1 for each code LLM–dataset pair. DeepSeek-Coder-6.7B Methods

Fine-Tuning w/o Augmented Alignment w/o Consistency MultiVul

DiverseVul

Qwen2.5-Coder-7B

Devign

DiverseVul

StarCoder2-7B

Devign

DiverseVul

CodeLlama-7B

Devign

DiverseVul

Devign

Accuracy

46.89 46.15 46.23 47.25

61.75 61.66 61.85 62.80

54.24 53.12 51.52 55.68

33.02 37.66 44.59 50.74

48.35 48.28 47.11 49.13

61.62 62.21 62.55 63.67

48.08 49.92 49.84 50.64

62.02 65.87 65.87 66.29

46.89 47.03 46.81 47.45

59.88 60.69 61.63 63.40

55.12 55.84 57.76 58.04

65.65 63.97 62.18 67.87

47.55 49.60 49.16 49.95

61.34 62.11 62.20 63.17

54.16 53.60 54.80 59.92

57.96 62.53 63.19 63.77

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

results on both the Effectiveness (ID detection) and Generalization (OOD detection). As 𝛼 increases, performance generally declines, suggesting that stronger perturbations produced by data augmentation introduce noise and weaken vulnerability-relevant code–text alignment. This trend is especially clear in the OOD detection. For example, compared with 𝛼 = 0.05, setting 𝛼 = 0.30 reduces the accuracy and F1 of generalization by 2.32% and 1.64%, respectively, while 𝛼 = 0.50 still leads to drops of 1.95 and 1.64%. A similar but milder trend is observed in the ID detection. For instance, when 𝛼 increases from 0.05 to 0.40, the accuracy and F1 of ID detection decrease by 0.80% and 0.82%, respectively. These results indicate that lightweight data augmentation provides useful regularization, whereas overly strong perturbations may distort the original semantics and reduce both effectiveness and generalization [14, 17, 44].

Accuracy Accuracy vs 3B (%)

from 59.92% to 53.60%, corresponding to a drop of 6.32%. These results suggest that the additional augmented code–text view provides useful supervision for learning more transferable representations. Removing consistency is often even more detrimental, especially in the more challenging transfer direction from Devign to DiverseVul. For example, F1 on DeepSeek-Coder-6.7B drops from 50.74% to 44.59%, decreasing by 6.15%. Similarly, F1 on StarCoder2-7B declines from 67.87% to 62.18%, a decrease of 5.69%. The largest drops in Accuracy follow the same trend, where Accuracy decreases from 59.92% to 54.80% on CodeLlama-7B, a drop of 5.12%. These results suggest that cross-view consistency plays an important role in stabilizing multimodal learning under distribution shift. Furthermore, we observe an interesting phenomenon. In several cases, training on DiverseVul and testing on Devign yields higher F1 than training and testing on Devign itself. For example, under M ULTI V UL, the F1 score of DeepSeek-Coder-6.7B increases from 41.36% in the ID setting to 62.80% under cross-dataset transfer, while StarCoder2-7B improves from 59.29% to 63.40%, and CodeLlama-7B from 53.22% to 63.17%. This is plausible because DiverseVul provides broader project coverage and richer vulnerability semantics, which may help the model learn more transferable representations than those obtained from the narrower training distribution of Devign [28]. In summary, M ULTI V UL improves not only ID effectiveness but also OOD generalization. By combining augmented code–text alignment with cross-view consistency, it yields the most reliable performance across diverse code LLMs. Detailed results are reported in Appendix D.

Dong et al.

For sensitivity analysis, we use Qwen2.5-Coder on DiverseVul as a representative configuration. Qwen2.5-Coder is one of the strongest and most stable code LLMs in the main results shown from Section 6.1, while DiverseVul serves as a primary benchmark and exhibits clearer and more consistent performance trends. We analyze the sensitivity of M ULTI V UL to two important factors, including the scale of the code LLM used for the task of code comment generation and the data augmentation strength 𝛼. Figure 4 shows that larger code LLMs used for code comment generation generally lead to better performance for M ULTI V UL. On the Effectiveness (ID detection), the gain is relatively modest. Compared with Qwen2.5-Coder-3B, Qwen2.5-Coder-32B improves Accuracy and F1 by only 0.64% and 0.65%, respectively. This suggests that M ULTI V UL does not rely heavily on the size of code LLMs, as smaller models already provide competitive supervision. In contrast, the improvement is more pronounced on the Generalization (OOD detection). Using Qwen2.5-Coder-32B increases Accuracy and F1 by 2.32% and 1.21% respectively, over Qwen2.5-Coder-3B. This indicates that higher-quality generated code comments are particularly beneficial for generalization under distribution shift. As shown in Figure 5, M ULTI V UL performs best under mild data augmentation, with 𝛼 = 0.05 achieving the strongest overall

1.5 1.0 0.5 3B

14B

Comment Generation LLM Scale

32B

(a) Δ Accuracy vs 3B (%).

F1 vs 3B (%)

RQ4: Sensitivity Analysis

2.0

0.0

Answer to RQ3: M ULTI V UL consistently improves cross-dataset generalization across four code LLMs. The ablation results further show that augmented alignment and consistency regularization are both important for stable performance under distribution shift.

6.4

ID OOD

1.2 1.0 0.8 0.6 0.4 0.2 0.0 0.2 0.4

ID OOD

14B

Comment Generation LLM Scale

32B

(b) Δ F1 vs 3B (%).

Figure 4: Sensitivity analysis of M ULTI V UL to the scale of the comment generation LLM on Qwen2.5-Coder over DiverseVul. In summary, these results show that M ULTI V UL remains relatively stable across different scales of code LLMs, while being more sensitive to the choice of data augmentation strength. In particular, mild perturbation yields the best overall performance, whereas stronger data augmentation tends to introduce noise and weaken both effectiveness and generalization. For completeness, the full ID and OOD results are provided in Appendix E.

Learning Generalizable Multimodal Representations for Software Vulnerability Detection

Accuracy

l l ul Vu Vu ul rse n gn rse seV ign eV e n g r e s v i v e g r v e i v i evi i e e iv v ·D ·D ·D ·D De Div ·D ·D 2 2 a a · · r r k k .5 .5 de de ee ee lam lam rCo rCo en2 en2 epS epS deL CodeL De Qw Sta De Qw Sta Co

ID OOD

0.5

Zero-Shot 1.661 3.221 1.384 4.457 1.459 2.735 1.759 3.317

1.0

One-Shot 2.265 5.778 1.732 4.127 1.961 5.996 2.393 6.149

1.5

Three-Shot 3.459 11.025 2.624 7.848 2.938 14.484 3.656 12.464

2.0

Zero-Shot-CoT 0.521 0.844 0.799 0.565 0.710 0.778 0.499 0.260

2.5 0.05

0.10

0.20

0.30

Augmentation Strength

0.40

0.50

(a) Δ Accuracy vs 𝛼 =0.05.

One-Shot-CoT 0.118 0.129 0.055 0.075 0.738 0.768 0.139 0.070 Three-Shot-CoT 0.046 0.115 0.093 0.115 0.659 0.790 0.093 0.251

0.00

ID OOD

0.25

100

10 1

Fine-Tuning 0.051 0.197 0.052 0.127 0.042 0.189 0.051 0.195 MultiVul 0.022 0.197 0.015 0.127 0.020 0.189 0.022 0.196

0.50 0.75

Figure 6: Inference latency across different methods.

1.00 1.25 1.50 0.05

0.10

0.20

0.30

Augmentation Strength

0.40

0.50

(b) Δ F1 vs 𝛼 =0.05.

Figure 5: Sensitivity analysis of M ULTI V UL to data augmentation strength 𝛼 on Qwen2.5-Coder over DiverseVul. Answer to RQ4: Compared with 3B, using 32B improves ID Accuracy by only 0.64%, but improves OOD Accuracy by 2.32%, suggesting that code comments generated from larger-scale LLMs are more useful under distribution shift. For data augmentation strength, 𝛼 = 0.05 performs best, while stronger perturbations reduce both effectiveness and generalization.

and alignment during the training phase, the inference phase is performed using code inputs only, with a single forward pass through the encoder and a lightweight classifier head. It therefore avoids the expensive autoregressive decoding required by prompting-based methods (e.g., three-shot), whose latency scales with prompt length and generated output length [59]. As a result, M ULTI V UL remains comparable to Fine-Tuning in inference cost while achieving roughly one to two orders of magnitude lower inference latency than standard prompting-based methods. This efficiency, together with its strong predictive performance, makes M ULTI V UL more attractive for practical vulnerability detection scenarios where both effectiveness and deployment are important. Answer to RQ5: M ULTI V UL preserves efficient code-only inference. Its latency is comparable to Fine-Tuning, and is roughly one to two orders of magnitude lower than prompting-based methods (e.g., one-shot), making it practical for deployment.

6.6 6.5

101

Inference Time (s, log scale)

Accuracy vs = 0.05 (%)

0.0

F1 vs = 0.05 (%)

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

RQ5: Inference Latency

Figure 6 presents the inference latency of different methods across four code LLMs 1 and two datasets. We can observe a clear trend that training-based methods, including Fine-Tuning and M ULTI V UL, are consistently much faster and more stable than prompting-based baselines. Prompting latency increases substantially as the number of in-context examples grows, with the highest cost appearing in the three-shot prompting. This effect is especially shown on Devign, where standard zero-shot, one-shot, and three-shot prompting ranges from 2.735 to 14.484 seconds per sample, whereas M ULTI V UL requires only 0.127 to 0.197 seconds. Importantly, M ULTI V UL keeps the same deployment as standard fine-tuning. Although M ULTI V UL leverages multimodal fusion 1 For simplicity, we refer to these models as DeepSeek, Qwen2.5, StarCoder2, and CodeLlama.

Case Study

To better understand the behavior of M ULTI V UL in vulnerability detection, we further conduct a qualitative analysis of false negative (FN) cases on DiverseVul using StarCoder2-7B. Figure 7 visualizes the overlap among the FN sets of Fine-Tuning, CLIP, and M ULTI V UL. Fine-Tuning, CLIP, and M ULTI V UL miss 151, 95, and 57 vulnerable functions, respectively. Among the cases missed by both baselines, M ULTI V UL correctly recovers 33. In addition, there is no vulnerable function missed only by M ULTI V UL. These results show that M ULTI V UL reduces FNs by correctly identifying cases that both baselines miss, without introducing additional cases that only M ULTI V UL fails to detect. Table 5 reports the error distribution by CWE category among FNs. Overall, M ULTI V UL shows clearer reductions on vulnerability categories that depend on contextual program behavior, such as exceptional-condition handling and memory-boundary reasoning. For example, on CWE-703, the number of missed cases decreases

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Dong et al.

Table 5: Error distribution by CWE category among false negatives (FN) on DiverseVul using StarCoder2-7B. CLIP refers to the standard code–text alignment, without augmented code–text alignment or consistency regularization.

CWE CWE-119 CWE-703 CWE-20 CWE-416 CWE-787 CWE-362 CWE-476 CWE-200 CWE-190 CWE-617 Others

Fine-Tuning

CLIP

M ULTI V UL

22 20 19 18 16 13 12 10 6 4 61

16 13 14 8 8 6 11 8 5 4 28

9 7 9 5 3 5 7 6 4 0 19

Fine-Tuning (FN=151)

Description Improper Restriction of Operations within the Bounds of a Memory Buffer Improper Check or Handling of Exceptional Conditions Improper Input Validation Use After Free Out-of-bounds Write Concurrent Execution using Shared Resource with Improper Synchronization NULL Pointer Dereference Exposure of Sensitive Information to an Unauthorized Actor Integer Overflow or Wraparound Reachable Assertion Others

CLIP (FN=95)

MultiVul (FN=57) Figure 7: Venn diagram showing the overlap among the falsenegative (FN) sets of Fine-Tuning, CLIP, and M ULTI V UL on DiverseVul using StarCoder2-7B. M ULTI V UL resolves 33 vulnerable functions that are missed by both Fine-Tuning and CLIP, while introducing no method-specific false negatives.

from 20 under Fine-Tuning and 13 under CLIP to 7 under M UL TI V UL . On CWE-119, they decrease from 22 and 16 to 9. Simi-

lar reductions also appear for CWE-787 and CWE-416, involving memory-boundary and memory-lifetime errors, respectively. These categories are difficult for code-only Fine-Tuning because the vulnerability information is often distributed across validation checks, error-handling paths, buffer sizes, and memory operations, rather than being identifiable from a single local token[10, 21, 68]. CLIP baseline improves over Fine-Tuning by introducing code–text alignment, but it still aligns only the original code–text pair. Therefore, it lacks the key components, including augmented-view alignment and cross-view consistency, used in M ULTI V UL. In contrast, M ULTI V UL combines original code–text alignment, augmented code–text alignment, and consistency regularization. This design

encourages the model to learn vulnerability-relevant semantics that remain stable across nearby code and text views, to help reduce the number of FNs in categories such as CWE-703 and CWE-119. However, the improvement is not uniform across all CWE categories. For example, for CWE-190, FNs decrease only from 6 and 5 under Fine-Tuning and CLIP to 4 under M ULTI V UL. This suggests that integer-overflow vulnerabilities remain challenging for M UL TI V UL . Unlike exceptional-condition or memory-boundary cases, CWE-190 often requires more precise numerical, range, or valueflow reasoning. The comments and lightweight data augmentations used by M ULTI V UL can provide functional context and improve representation stability, but they do not explicitly model arithmetic constraints or value propagation. As a result, M ULTI V UL shows more limited gains on this category than on categories that depend more on contextual program behavior.

Conclusion

This paper presented M ULTI V UL, a multimodal vulnerability detection framework that uses automatically generated code comments as training-time supervision while preserving code-only inference. M ULTI V UL aligns original and augmented code–text pairs using dual-CLIP losses and stabilizes the learned representation space through cross-view consistency regularization. Experiments on two benchmarks and four code LLMs show that M ULTI V UL improves detection effectiveness and OOD generalization while maintaining efficient code-only inference, and ablations further confirms the importance of augmented alignment and consistency regularization. These results indicate that multimodal supervision can strengthen vulnerability detection without requiring natural language input at inference time. Future work will explore richer training-time context, such as commit messages, issue reports, and vulnerability advisories, to further improve cross-project and cross-dataset generalization.

References [1] 2026. CVE Details: Vulnerability Statistics. https://www.cvedetails.com/. Accessed: 2026-04-24. [2] Miltiadis Allamanis, Henry Richard Jackson-Flux, and Marc Brockschmidt. 2021. Self-supervised bug detection and repair. In Advances in Neural Information Processing Systems. [3] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv

Learning Generalizable Multimodal Representations for Software Vulnerability Detection

preprint arXiv:2309.16609 (2023). [4] Sofia Bekrar, Chaouki Bekrar, Roland Groz, and Laurent Mounier. 2011. Finding software vulnerabilities by smart fuzzing. In 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation. IEEE, 427–430. [5] Pavol Bielik and Martin Vechev. 2020. Adversarial robustness for code. In Proceedings of the 37th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 119). PMLR, 896–907. [6] Nghi DQ Bui, Yijun Yu, and Lingxiao Jiang. [n. d.]. Self-supervised contrastive learning for code retrieval and summarization via semantic-preserving transformations. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (Virtual Event, Canada) (SIGIR ’21). Association for Computing Machinery, New York, NY, USA, 511–521. doi:10.1145/3404835.3462840 [7] Liuwen Cao, Hongkui He, Hailin Huang, Jiexin Wang, and Yi Cai. 2025. Rethinking-based code summarization with chain of comments. In Proceedings of the 31st International Conference on Computational Linguistics. 3043–3056. [8] Saikat Chakraborty, Rahul Krishna, Yangruibo Ding, and Baishakhi Ray. 2021. Deep learning based vulnerability detection: Are we there yet? IEEE Transactions on Software Engineering 48, 9 (2021), 3280–3296. [9] Yizheng Chen, Zhoujie Ding, Lamya Alowain, Xinyun Chen, and David Wagner. 2023. Diversevul: A new vulnerable source code dataset for deep learning based vulnerability detection. In Proceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses. 654–668. [10] Xiao Cheng, Guanqin Zhang, Haoyu Wang, and Yulei Sui. 2022. Path-sensitive code embedding via contrastive learning for software vulnerability detection. In Proceedings of the 31st ACM SIGSOFT international symposium on software testing and analysis. 519–531. [11] Zhaoyang Chu, Yao Wan, Qian Li, Yang Wu, Hongyu Zhang, Yulei Sui, Guandong Xu, and Hai Jin. 2024. Graph neural networks for vulnerability detection: A counterfactual explanation. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 389–401. [12] Giordano Cicchetti, Eleonora Grassucci, Luigi Sigillo, and Danilo Comminiello. 2025. Gramian Multimodal Representation Learning and Alignment. In The Thirteenth International Conference on Learning Representations. [13] Yangruibo Ding, Yanjun Fu, Omniyyah Ibrahim, Chawin Sitawarin, Xinyun Chen, Basel Alomair, David Wagner, Baishakhi Ray, and Yizheng Chen. 2024. Vulnerability detection with code language models: How far are we? arXiv preprint arXiv:2403.18624 (2024). [14] Zeming Dong, Qiang Hu, Yuejun Guo, Maxime Cordy, Mike Papadakis, Zhenya Zhang, Yves Le Traon, and Jianjun Zhao. 2023. Mixcode: enhancing code classification by mixup-based data augmentation. In 2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 379–390. [15] Zeming Dong, Qiang Hu, Yuejun Guo, Zhenya Zhang, Maxime Cordy, Mike Papadakis, Yves Le Traon, and Jianjun Zhao. 2025. Boosting source code learning with text-oriented data augmentation: an empirical study. Empirical Software Engineering 30, 3 (2025), 68. [16] Zeming Dong, Qiang Hu, Xiaofei Xie, Maxime Cordy, Mike Papadakis, Yves Le Traon, and Jianjun Zhao. 2026. GenCode: A generic data augmentation framework for boosting deep learning-based code understanding. Empirical Software Engineering 31, 3 (2026), 72. [17] Zeming Dong, Qiang Hu, Zhenya Zhang, Yuejun Guo, Maxime Cordy, Mike Papadakis, Yves Le Traon, and Jianjun Zhao. 2024. On the effectiveness of hybrid pooling in mixup-based graph learning for language processing. Journal of Systems and Software 216 (2024), 112139. [18] Xiaohu Du, Ming Wen, Jiahao Zhu, Zifan Xie, Bin Ji, Huijun Liu, Xuanhua Shi, and Hai Jin. 2024. Generalization-enhanced code vulnerability detection via multitask instruction fine-tuning. In Findings of the Association for Computational Linguistics: ACL 2024. 10507–10521. [19] Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. 2021. A survey of data augmentation approaches for NLP. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. Association for Computational Linguistics, 968–988. doi:10. 18653/v1/2021.findings-acl.84 [20] Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. 2020. Codebert: A pre-trained model for programming and natural languages. In Findings of the association for computational linguistics: EMNLP 2020. 1536–1547. [21] Michael Fu and Chakkrit Tantithamthavorn. 2022. Linevul: A transformer-based line-level vulnerability prediction. In Proceedings of the 19th International Conference on Mining Software Repositories. 608–620. [22] Xiang Gao, Ripon K. Saha, Mukul R. Prasad, and Abhik Roychoudhury. 2020. Fuzz testing based data augmentation to improve robustness of deep neural networks. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (ICSE ’20). Association for Computing Machinery, 1147–1158. doi:10.1145/3377811.3380415 [23] Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et al. 2020. Graphcodebert:

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Pre-training code representations with data flow. arXiv preprint arXiv:2009.08366 (2020). [24] Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming–The Rise of Code Intelligence. arXiv preprint arXiv:2401.14196 (2024). [25] Yuejun Guo, Seifeddine Bettaieb, and Fran Casino. 2024. A comprehensive analysis on software vulnerability detection datasets: trends, challenges, and road ahead. International Journal of Information Security 23, 5 (2024), 3311–3327. [26] Raia Hadsell, Sumit Chopra, and Yann LeCun. 2006. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR’06), Vol. 2. IEEE, 1735–1742. [27] Hazim Hanif and Sergio Maffeis. 2022. Vulberta: Simplified source code pretraining for vulnerability detection. In 2022 International joint conference on neural networks (IJCNN). IEEE, 1–8. [28] Steffen Herbold, Alexander Trautsch, and Jens Grabowski. 2018. A comparative study to benchmark cross-project defect prediction approaches. In Proceedings of the 40th international conference on software engineering. 1063–1063. [29] Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large language models for software engineering: A systematic literature review. ACM Transactions on Software Engineering and Methodology 33, 8 (2024), 1–79. [30] Wei Huang, Andi Han, Yongqiang Chen, Yuan Cao, Zhiqiang Xu, and Taiji Suzuki. 2024. On the comparison between multi-modal and single-modal contrastive learning. Advances in Neural Information Processing Systems 37 (2024), 81549– 81605. [31] Chen Ji, Su Yang, Hongyu Sun, and Yuqing Zhang. 2024. Applying Contrastive Learning to Code Vulnerability Type Classification. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 11942–11952. [32] Avishree Khare, Saikat Dutta, Ziyang Li, Alaia Solko-Breslin, Rajeev Alur, and Mayur Naik. 2025. Understanding the effectiveness of large language models in detecting security vulnerabilities. In 2025 IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE, 103–114. [33] Seulbae Kim, Seunghoon Woo, Heejo Lee, and Hakjoo Oh. 2017. Vuddy: A scalable approach for vulnerable code clone discovery. In 2017 IEEE symposium on security and privacy (SP). IEEE, 595–614. [34] Guilherme Lacerda, Fabio Petrillo, Marcelo Pimenta, and Yann Gaël Guéhéneuc. 2020. Code smells and refactoring: a tertiary systematic review of challenges and observations. Journal of Systems and Software 167 (2020), 110610. [35] Jiayuan Li, Lei Cui, Sen Zhao, Yun Yang, Lun Li, and Hongsong Zhu. 2025. CLeVeR: Multi-modal Contrastive Learning for Vulnerability Code Representation. In Findings of the Association for Computational Linguistics: ACL 2025. 7940–7951. [36] Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel LamyPoirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. 2024. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173 (2024). [37] Zhenyan Lu, Xiang Li, Dongqi Cai, Rongjie Yi, Fangming Liu, Wei Liu, Jian Luan, Xiwen Zhang, Nicholas D Lane, and Mengwei Xu. 2025. Demystifying small language models for edge deployment. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 14747–14764. [38] Qiheng Mao, Zhenhao Li, Xing Hu, Kui Liu, Xin Xia, and Jianling Sun. 2025. Towards explainable vulnerability detection with large language models. IEEE Transactions on Software Engineering (2025). [39] Hoang Nguyen, Ye Liu, Chenwei Zhang, Tao Zhang, and Philip S Yu. 2023. CoF-CoT: Enhancing large language models with coarse-to-fine chain-of-thought prompting for multi-domain NLU tasks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 12109–12119. [40] Van Nguyen, Trung Le, Chakkrit Tantithamthavorn, John Grundy, and Dinh Phung. 2024. Deep domain adaptation with max-margin principle for cross-project imbalanced software vulnerability detection. ACM Transactions on Software Engineering and Methodology 33, 6 (2024), 1–34. [41] Yu Nong, Mohammed Aldeen, Long Cheng, Hongxin Hu, Feng Chen, and Haipeng Cai. 2024. Chain-of-thought prompting of large language models for discovering and fixing software vulnerabilities. arXiv preprint arXiv:2402.17230 (2024). [42] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018). [43] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning. PmLR, 8748–8763. [44] Sylvestre-Alvise Rebuffi, Sven Gowal, Dan Andrei Calian, Florian Stimberg, Olivia Wiles, and Timothy A Mann. 2021. Data augmentation can improve robustness. Advances in neural information processing systems 34 (2021), 29935– 29948.

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

[45] Guoping Rong, Yongda Yu, Song Liu, Xin Tan, Tianyi Zhang, Haifeng Shen, and Jidong Hu. 2025. Code comment inconsistency detection and rectification using a large language model. In Proceedings of the IEEE/ACM 47th International Conference on Software Engineering. 1832–1843. [46] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023). [47] Rijha Safdar, Danyail Mateen, Syed Taha Ali, M Umer Ashfaq, and Wajahat Hussain. 2025. Data and Context Matter: Towards Generalizing AI-based Software Vulnerability Detection. arXiv preprint arXiv:2508.16625 (2025). [48] Hitesh Sajnani, Vaibhav Saini, Jeffrey Svajlenko, Chanchal K Roy, and Cristina V Lopes. 2016. Sourcerercc: Scaling code clone detection to big-code. In Proceedings of the 38th international conference on software engineering. 1157–1168. [49] Janaka Senanayake, Harsha Kalutarage, Mhd Omar Al-Kadri, Andrei Petrovski, and Luca Piras. 2023. Android source code vulnerability detection: a systematic literature review. Comput. Surveys 55, 9 (2023), 1–37. [50] Ze Sheng, Zhicheng Chen, Shuning Gu, Heqing Huang, Guofei Gu, and Jeff Huang. 2025. Llms in software security: A survey of vulnerability detection techniques and insights. Comput. Surveys 58, 5 (2025), 1–35. [51] Ensheng Shi, Yanlin Wang, Lun Du, Junjie Chen, Shi Han, Hongyu Zhang, Dongmei Zhang, and Hongbin Sun. 2022. On the evaluation of neural code summarization. In Proceedings of the 44th international conference on software engineering. 1597–1608. [52] Nima Shiri Harzevili, Alvine Boaye Belle, Junjie Wang, Song Wang, Zhen Ming Jiang, and Nachiappan Nagappan. 2024. A systematic literature review on automated software vulnerability detection using machine learning. Comput. Surveys 57, 3 (2024), 1–36. [53] Demin Song, Honglin Guo, Yunhua Zhou, Shuhao Xing, Yudong Wang, Zifan Song, Wenwei Zhang, Qipeng Guo, Hang Yan, Xipeng Qiu, et al. 2024. Code needs comments: Enhancing code llms with comment augmentation. In Findings of the Association for Computational Linguistics: ACL 2024. 13640–13656. [54] Weisong Sun, Yun Miao, Yuekang Li, Hongyu Zhang, Chunrong Fang, Yi Liu, Gelei Deng, Yang Liu, and Zhenyu Chen. 2024. Source code summarization in the era of large language models. arXiv preprint arXiv:2407.07959 (2024). [55] Wenxin Tao, Xiaohong Su, Jiayuan Wan, Hongwei Wei, and Weining Zheng. 2023. Vulnerability detection through cross-modal feature enhancement and fusion. Computers & Security 132 (2023), 103341. [56] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017). [57] Deze Wang, Zhouyang Jia, Shanshan Li, Yue Yu, Yun Xiong, Wei Dong, and Xiangke Liao. [n. d.]. Bridging pre-trained models and downstream tasks for source code understanding. In Proceedings of the 44th International Conference on Software Engineering (Pittsburgh, Pennsylvania) (ICSE ’22). Association for Computing Machinery, New York, NY, USA, 287–298. doi:10.1145/3510003. 3510062 [58] Pengcheng Wang, Jeffrey Svajlenko, Yanzhao Wu, Yun Xu, and Chanchal K Roy. 2018. CCAligner: a token based large-gap clone detector. In Proceedings of the 40th International Conference on Software Engineering. 1066–1077. [59] Xinfeng Wang, Jin Cui, Fumiyo Fukumoto, and Yoshimi Suzuki. 2025. AGRec: Adapting Autoregressive Decoders with Graph Reasoning for LLM-based Sequential Recommendation. In Findings of the Association for Computational Linguistics: ACL 2025. Association for Computational Linguistics, Vienna, Austria, 7076–7090. doi:10.18653/v1/2025.findings-acl.369 [60] Jason Wei and Kai Zou. 2019. EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLPIJCNLP). Association for Computational Linguistics, Hong Kong, China, 6382– 6388. doi:10.18653/v1/D19-1670 [61] Peng Xu, Xiatian Zhu, and David A Clifton. 2023. Multimodal learning with transformers: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 10 (2023), 12113–12132. [62] Noam Yefet, Uri Alon, and Eran Yahav. 2020. Adversarial examples for models of code. Proceedings of the ACM on Programming Languages 4, OOPSLA (2020), 1–30. doi:10.1145/3428230 [63] Shiwen Yu, Ting Wang, and Ji Wang. 2022. Data augmentation by program transformation. Journal of Systems and Software 190 (2022), 111304. doi:10. 1016/j.jss.2022.111304 [64] Xin Yuan, Zhe Lin, Jason Kuen, Jianming Zhang, Yilin Wang, Michael Maire, Ajinkya Kale, and Baldo Faieta. 2021. Multimodal contrastive training for visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 6995–7004. [65] Chunyong Zhang, Bin Liu, Yang Xin, and Liangwei Yao. 2023. CPVD: Cross project vulnerability detection based on graph attention network and domain adaptation. IEEE Transactions on Software Engineering 49, 8 (2023), 4152– 4168.

Dong et al.

[66] Quanjun Zhang, Chunrong Fang, Yang Xie, Yaxin Zhang, Shengcheng Yu, Weisong Sun, Yun Yang, and Zhenyu Chen. 2026. A survey on large language models for software engineering. Science China Information Sciences 69, 4 (2026), 141102. [67] Qi Zhang, Yifei Wang, and Yisen Wang. 2023. On the generalization of multimodal contrastive learning. In International Conference on Machine Learning. PMLR, 41677–41693. [68] Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. 2019. Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks. Advances in neural information processing systems 32 (2019).

Critique Prompting

For code comment generation, we use a critique prompting that improves factuality while avoiding unsupported or hallucinated descriptions. Specifically, we use a three-step critique prompting strategy to generate code comments in M ULTI V UL. Given a function, the code LLM first produces a one-sentence summary of its core functionality. It is then prompted to critically review this draft and identify unsupported claims, missing core behavior, or statements not explicitly grounded in the code, etc. Finally, the code LLM revises the summary according to its self-critique and outputs a single sentence that describes only the behavior evidenced by the function. The prompting process is defined as follows:

Hyperparameters

The hyperparameters used for the M ULTI V UL model training include various settings for model optimization and regularization, shown in Table 6.

Detailed Results for Ablation Study

We provide the full results, including Accuracy, Precision, Recall, and F1, for the ablation experiments discussed in Section 6.2. Specifically, Table 7 reports the detailed results on the in-distribution (ID) testing, respectively. The table complements the main-text discussion by showing the complete performance of each ablated variant, including the effects of removing augmented-view alignment and consistency regularization.

Detailed Results for Out-of-Distribution Generalization

We further report the full results for the out-of-distribution (OOD) generalization experiments discussed in Section 6.3. These results from Table 8 provide a more detailed view of how different methods behave under cross-dataset evaluation, including Accuracy, Precision, Recall, and F1 on the OOD detection. They supplement the main-text analysis by showing the extent to which M ULTI V UL maintains stronger generalization than prompting-based and code-only fine-tuning baselines under distribution shift.

Detailed Results for Sensitivity Analysis

We also provide the full results for the sensitivity analysis discussed in Section 6.4. Specifically, Tables 9 and 10 report the detailed results for varying the scale of the code comment generation LLM and the augmentation strength 𝛼, respectively. These tables include Accuracy, Precision, Recall, and F1, and complement the main-text figures by showing the complete numerical trends underlying the sensitivity analysis.

Learning Generalizable Multimodal Representations for Software Vulnerability Detection

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

System: "You are an expert code summarization assistant. Hard constraints: - Output exactly ONE sentence in English. - Describe ONLY what is explicitly shown in the code. - Do NOT claim input validation, error handling, bounds checks, permissions, or safety guarantees unless the code clearly shows them. - Avoid speculative words such as 'ensure/ensures/ensuring', 'handle(s) errors', 'validate(s)', 'sanitize (s)', 'filter(s)', 'guarantee(s)' unless explicitly present. - Do NOT mention security or vulnerabilities (this is the normal setting)." User: "Please generate a short one-sentence comment describing the core functionality of the following function: <code> def calculate_factorial(n): if n == 0 or n == 1: return 1 else: return n * calculate_factorial(n - 1) </code> Output ONLY the sentence." Assistant (Draft): "Computes the factorial of n recursively, returning 1 when n is 0 or 1." User: "Review your previous answer and list problems. Check specifically for: - Any speculation beyond the code (e.g., 'ensures', 'handles errors', 'validates', 'guarantees'). - Any claims of checks that are not explicitly shown (input validation, bounds checks, error handling, permissions). - Missing core behavior (main operations, key calls, main data flow). Output ONLY short bullet points. Do NOT revise yet." Assistant (Review): - No unsupported claims are made. - The recursive call and base cases are clearly described. User: "Based on the problems you found, improve your answer. Requirements: - Output exactly ONE sentence in English. - Describe ONLY what is explicitly shown in the code. - Remove any speculative or non-evidenced claims. - Do NOT mention security or vulnerabilities. Output ONLY the final sentence." Assistant (Final): "Computes the factorial of n recursively with base cases for 0 and 1."

Figure 8: Code comment generation with critique prompting.

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Dong et al.

Table 6: Hyperparameters of M ULTI V UL.

Hyperparameter

Value

Description

Maximum Input Length Projection Dimension Batch Size Training Epochs Original-view CLIP Loss Weight Augmented-view CLIP Loss Weight Consistency Loss Weight Learning Rate Weight Decay

4096 768 8 10 0.5 0.5 0.1 3e-05 0.0001

Maximum number of tokens per input sequence. Dimensionality of the shared space for code and text embeddings. Number of samples per optimization step. Number of full passes over the training dataset. Weight of the contrastive loss for the original code–text view. Weight of the contrastive loss for the augmented code–text view. Weight of the consistency between the original view and the augmented view. Optimizer step size. Regularization via penalizing large weights.

Table 7: Ablation study on DiverseVul and Devign across four code LLMs. All metrics are reported in percentage (%). Fine-Tuning refers to standard supervised fine-tuning on code inputs only. w/o Augmented Alignment removes the augmented code–text alignment. w/o Consistency removes the cross-view consistency regularization. Bold marks the best value per column within each code LLM. Blue shading highlights the highest Accuracy and F1 for each code LLM–dataset pair. DeepSeek-Coder-6.7B Methods

DiverseVul Accuracy Precision Recall

Fine-Tuning w/o Augmented Alignment w/o Consistency MultiVul

78.72 89.60 89.84 90.00

81.88 92.02 92.20 92.06

Qwen2.5-Coder-7B Devign

73.76 86.72 87.04 89.92

F1 77.61 89.29 89.55 90.98

DiverseVul

Accuracy Precision Recall 56.19 58.90 58.10 60.95

55.51 79.25 84.00 65.73

20.22 13.48 10.11 30.18

F1 29.64 23.05 18.05 41.36

Devign

Accuracy Precision Recall 88.72 92.64 92.96 93.60

92.61 91.58 96.21 93.74

84.16 93.92 89.44 93.44

StarCoder2-7B Methods

DiverseVul

Fine-Tuning w/o Augmented Alignment w/o Consistency MultiVul

84.00 89.44 88.32 91.84

87.35 89.95 87.02 93.37

79.52 88.80 90.08 91.52

F1 83.25 89.37 88.52 92.43

63.08 65.86 65.20 67.11

DiverseVul

Accuracy Precision Recall 62.20 62.12 62.64 64.69

88.18 92.73 92.70 93.59

Accuracy Precision Recall 58.51 60.40 59.59 64.75

65.65 73.19 73.84 76.21

F1 61.88 66.18 65.95 70.01

CodeLlama-7B Devign

Accuracy Precision Recall

61.92 69.78 60.68 62.57

44.62 30.02 51.52 56.34

F1 51.87 41.98 55.73 59.29

Devign

Accuracy Precision Recall 87.28 91.28 91.52 91.68

89.49 90.95 91.52 93.06

84.48 91.68 91.52 90.08

F1 86.91 91.31 91.52 91.54

Accuracy Precision Recall 60.51 64.03 63.74 64.30

59.86 72.15 72.07 63.62

40.93 34.51 33.55 45.75

F1 48.62 46.69 45.78 53.22

Table 8: Ablation study of M ULTI V UL for OOD detection across four code LLMs. All metrics are reported in percentage (%). Fine-Tuning refers to standard supervised fine-tuning on code inputs only. w/o Augmented Alignment removes the augmented code–text alignment. w/o Consistency removes the cross-view consistency regularization. Bold marks the best value per column within each code LLM. Blue shading highlights the highest Accuracy and F1 for each code LLM–dataset pair. DeepSeek-Coder-6.7B Methods

Fine-Tuning w/o Augmented Alignment w/o Consistency MultiVul

DiverseVul Accuracy Precision Recall

46.15 45.67 45.73 46.15

61.75 61.66 61.85 62.80

46.89 46.15 46.23 47.25

Qwen2.5-Coder-7B Devign

93.26 94.86 95.51 98.23

DiverseVul

Accuracy Precision Recall

61.57 56.19 51.69 60.06

33.02 37.66 44.59 50.74

54.24 53.12 51.52 55.68

22.56 28.32 39.20 43.92

48.35 48.28 47.11 49.13

46.62 46.67 46.21 47.17

90.85 93.26 96.79 97.93

StarCoder2-7B Methods

DiverseVul Accuracy Precision Recall

Fine-Tuning w/o Augmented Alignment w/o Consistency MultiVul

46.89 47.03 46.81 47.45

45.69 45.89 45.94 46.74

86.84 89.57 93.58 98.52

59.88 60.69 61.63 63.40

61.62 62.21 62.55 63.67

Accuracy Precision Recall 48.08 49.92 49.84 50.64

DiverseVul

Accuracy Precision Recall

85.76 78.40 69.44 83.21

65.65 63.97 62.18 67.87

55.12 55.84 57.76 58.04

48.89 49.96 49.92 50.33

84.80 96.64 96.80 97.04

F1 62.02 65.87 65.87 66.29

CodeLlama-7B Devign

Devign

Accuracy Precision Recall

53.17 54.02 56.29 57.31

Accuracy Precision Recall 47.55 49.60 49.16 49.95

46.35 47.28 47.07 47.59

90.70 90.53 91.65 93.89

Devign F1 61.34 62.11 62.20 63.17

Accuracy Precision Recall 54.16 53.60 54.80 59.92

53.52 52.44 53.30 58.18

63.20 77.44 77.60 70.56

F1 57.96 62.53 63.19 63.77

Learning Generalizable Multimodal Representations for Software Vulnerability Detection

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Table 9: Sensitivity analysis of M ULTI V UL to the scale of the code comment generation LLM on Qwen2.5-Coder over DiverseVul. All metrics are reported in percentage (%). Bold marks the best value in each column. Blue shading highlights the best Accuracy and F1 within each split.

3B 7B 14B 32B

OOD (Generalization)

ID (Effectiveness)

LLM Scale Accuracy

Precision

Recall

Accuracy

Precision

Recall

92.96 92.96 93.44 93.60

93.24 94.68 94.15 93.74

92.64 91.04 92.64 93.44

92.94 92.82 93.39 93.59

46.81 47.18 46.96 49.13

46.07 46.20 46.08 47.17

96.95 95.51 95.18 97.93

62.46 62.27 62.09 63.67

Table 10: Sensitivity analysis of M ULTI V UL to augmentation strength 𝛼 on Qwen2.5-Coder over DiverseVul. All metrics are reported in percentage (%). Bold marks the best value in each column. Blue shading highlights the best Accuracy and F1 within each split. ID (Effectiveness)

Augmentation Strength 𝛼 0.05 0.10 0.20 0.30 0.40 0.50

OOD (Generalization)

Accuracy

Precision

Recall

Accuracy

Precision

Recall

93.60 93.52 93.20 93.20 92.80 92.96

93.74 94.16 93.13 93.41 93.21 94.09

93.44 92.80 93.28 92.96 92.32 91.68

93.59 93.47 93.21 93.18 92.77 92.87

49.13 46.59 46.81 46.81 47.18 47.18

47.17 45.93 46.07 46.00 46.27 46.16

97.93 95.99 96.95 95.18 97.59 94.54

63.67 62.13 62.46 62.03 62.78 62.03

Record · ID 141535 · SHA-256 331625f0dc05e5fb
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.