Detecting Vulnerability-Inducing Commits via Multi-Stage Reasoning with LLM-Based Agents Liyou Chen1 , Hailong Sun1,2,⋆ , Xiang Gao1,2,⋆ , and Yue Pan3 State Key Laboratory of Complex & Critical Software Environment (CCSE), Beihang University, Beijing, China. [email protected] 2 Hangzhou Innovation Institute of Beihang University, Hangzhou, Zhejiang, China. [email protected], [email protected] 3 North China Municipal Engineering Design & Research Institute Co., Ltd., Beijing, China. [email protected]
arXiv:2607.05772v1 [cs.SE] 7 Jul 2026
1
Abstract. Detecting vulnerability-inducing commits (VICs) at submission time is critical for improving the security and reliability of software systems. However, this task is highly challenging because it requires reasoning about the semantic impact of code changes from heterogeneous information sources, including code diffs, commit messages, and the surrounding contextual code. Existing approaches often struggle to fully capture these complex interactions, resulting in limited detection performance. In this paper, we propose VIC-RAGENT, an LLM-based multiagent framework for effective and explainable vulnerability detection. VIC-RAGENT leverages multiple specialized agents to provide complementary perspectives, including structural analysis, intent understanding, and vulnerability inspection. To further improve detection reliability, the framework employs a multi-stage reasoning process that progressively refines candidate vulnerabilities through preliminary inspection, reanalysis, and a final decision stage. Experimental results on a real-world dataset across multiple LLMs demonstrate that VIC-RAGENT consistently outperforms baselines, including Direct, CoT, and CodeAgent. Compared to the strongest baseline, VIC-RAGENT achieves 1.2–1.7× higher F1-scores across different models. Overall, VIC-RAGENT offers a robust, explainable, and practical solution for detecting VICs in modern software development workflows. Keywords: vulnerability-inducing commits · just-in-time vulnerability detection · large language models · software security.
1
Introduction
Open source software (OSS) has become fundamental infrastructure for modern software systems [25, 5], but its collaborative development model enlarges the software supply chain attack surface [29]. In large-scale projects with continuous integration, even minor code changes may unintentionally introduce exploitable ⋆
Corresponding authors
2
Chen et al.
weaknesses [12]. Prior studies [28, 12] show that many vulnerabilities originate from vulnerability-inducing commits (VICs) [2], which introduce security flaws during routine development activities such as feature additions and refactoring [10, 35]. Detecting whether a newly submitted commit introduces a vulnerability— known as Just-In-Time Vulnerability Detection (JIT-VD) [21]—remains a challenging task. JIT-VD aims to detect whether a commit introduces vulnerabilities based on code changes and contextual information. Existing approaches include feature-based methods and deep learning models [26, 36], but they struggle to capture commit semantics and developer intent. To address these limitations, deep learning approaches have been proposed to learn representations from code and diffs, including neural vulnerability detection systems [16, 15], graph-based models [40, 3], and transformer-based models [8]. More recently, JIT-specific models [21, 31] attempt to capture the semantics of code changes. Recent advances in LLMs have demonstrated strong capabilities in code understanding and reasoning [34, 23, 37]. LLM-based approaches have been applied to vulnerability detection and software security analysis [19, 39, 33], including prompt-based methods and retrieval-augmented methods [6]. In addition, agentbased frameworks such as CodeAgent [32] explore automated code review. However, these approaches are primarily designed for general vulnerability detection or code review tasks, and lack specialized mechanisms for reasoning about VICs in evolving codebases. The ReAct Agent [38] introduces an iterative reasoning process based on a thought–action–observation loop, enabling LLMs to dynamically acquire additional context during analysis. This design may lead to the accumulation and propagation of errors across reasoning steps, while lacking explicit mechanisms for result verification. Such a design may make intermediate results difficult to independently verify and limit the transparency of the reasoning process. In contrast, role-specialized multi-agent systems have been shown to improve modularity and reasoning reliability by explicitly separating different reasoning objectives [20, 14]. To address these limitations, we propose VIC-RAGENT, an LLM-based multi-agent framework with a multi-stage reasoning process for detecting of VICs. VIC-RAGENT introduces a structured multi-stage reasoning pipeline with role-specialized components, enabling explicit decomposition of commit semantics, intent understanding, and vulnerability verification. VIC-RAGENT analyzes each commit by integrating multiple sources of information, including code diffs, commit messages, and file-level context. The framework adopts a multi-agent analysis workflow. Specifically, a Code Analyst agent extracts structural information, and a Target Analyst agent interprets commit intent. Vulnerability Inspector agents perform multi-stage reasoning to identify potential vulnerabilities. When a vulnerability is confirmed, a Document Specialist agent generates a security report that is stored in the knowledge base, enabling future analyzes to leverage historical vulnerability cases. We evaluate VIC-RAGENT on the V-SZZ dataset [2] across multiple LLMs. Experiments show that VIC-RAGENT consistently improves recall and F1-
Detecting VICs via Multi-Stage Reasoning with LLM-Based Agents Stage 0: Preparation Code Analyst (CA)
Stage 1: Preliminary Inspection (Candidate Generation)
Stage 2: Reanalysis (Type-Guided Verification)
Candidate Fragments + Predicted Vulnerability Types
Vulnerability Inspector vulnerability type - incorporate retrieved case if (VI)
Structural Code Info
Commit Input - Commit message - Code diff - File-level context
For each candidate fragment: - re-check under predicted available
Retrieved Case RAG: Most Similar
Vulnerability Inspector (VI)
Target Analyst (TA)
Guided by - vulnerability taxonomy
Change Intent / Goal Alignment
Historical Case
No suspicious fragments (End)
Refined Fragment-level Analysis Results
Stage 3: Final Decision (Verification + Aggregation) For each filtered candidate fragment: - final fragment-level verification Then: commit-level aggregation
Security Report
3
Not Vulnerable Commit (End) Document Specialist (DS)
Vulnerable Commit
Vulnerability Knowledge Base
- End (if there are no remaining suspicious code fragments) - To Stage 3 (else) Make decisions based on the evidence provided before
Save & End
Vulnerability Inspector (VI)
Fig. 1. The overall framework of VIC-RAGENT.
score across all evaluated LLMs, achieving up to 2× recall improvement over the strongest baseline. Our main contributions are summarized as follows: A multi-agent framework for JIT-VD. We propose VIC-RAGENT, a multi-agent framework that analyzes potential vulnerabilities introduced by commit. A multi-stage reasoning process. We introduce a structured inspection workflow consisting of preliminary analysis, reanalysis, and final decision, enabling coarse-to-fine reasoning for improved vulnerability detection. Knowledge-augmented vulnerability reasoning. We design a vulnerability knowledge base that stores security reports organized by vulnerability types, allowing VIC-RAGENT to leverage historical vulnerability cases through retrieval-augmented reasoning. Empirical evaluation on a real-world dataset. We conduct experiments on a real-world dataset, demonstrating strong detection performance across three LLMs. The implementation of VIC-RAGENT and the dataset are available at: https://github.com/KeLeXueBi/VIC-RAGRENT.
2
Multi-Agent Analysis Framework
2.1
Framework Overview
Fig. 1 illustrates the overall architecture of VIC-RAGENT, a multi-stage reasoning framework for detecting VICs. Table 1 summarizes the responsibilities of each agent in the framework. VIC-RAGENT performs a structured analysis through four sequential stages: preparation, preliminary inspection, reanalysis, and final decision. The output of each stage is progressively refined and passed to subsequent stages.
4
Chen et al. Table 1. Agents in VIC-RAGENT.
Agent
Input
Code Analyst (CA)
Diff + Context Structural summary
Structural analysis
Target Analyst (TA)
Diff + Message Intent summary
Intent understanding
Vulnerability Inspector CA + TA (VI)(stage 1)
Output
Responsibility
Candidate vulnerabil- Security inspection ities
Vulnerability Inspector VI (stage 1) + Refined (VI)(stage 2) Knowledge Base ties
vulnerabili- Knowledge-augmented verification
Vulnerability Inspector VI (stage 2) (VI)(stage 3)
Prediction result
Final decision
Document (DS)
Security report
Knowledge tion
Validation feedback
Process verification
Specialist Verified VIC
Audit Supervisor (AS) Agent outputs
2.2
construc-
Stage 0: Preparation
Given a commit input, the preparation stage performs contextual analysis using two agents. The goal of this stage is to construct structured intermediate representations that capture both structural and semantic aspects of the code changes. The structural representation is constructed by the Code Analyst (CA), which extracts structural information from commit changes to help other agents understand how modified code interacts with the existing system. Given the code diff together with the file-level context, the CA summarizes structural relationships among components, including simplified call graphs and potential data flow patterns. The intent representation is derived by the Target Analyst (TA), which interprets the intentions behind commit changes and aligns the goals described in the commit message with the modified files. The TA extracts the goals described in the commit message and categorizes them into common modification types (e.g., bug fixes, feature additions, refactoring, performance optimizations, or security patches). Inconsistencies between intended goals and implemented changes—such as incomplete implementation or unclear scope—may indicate potential risks. Together, the structural representation and intent representation provide complementary structural and semantic views of the commit. This dual-view representation reduces ambiguity in subsequent reasoning stages and helps the Vulnerability Inspector better interpret the intent and impact of code changes. 2.3
Stage 1: Preliminary Inspection
The preliminary inspection stage can be viewed as a high-recall candidate generation step that approximates an over-complete set of potential vulnerability
Detecting VICs via Multi-Stage Reasoning with LLM-Based Agents
5
hypotheses. Instead of making definitive decisions, this stage intentionally relaxes verification constraints and constructs a superset of plausible vulnerabilityinducing fragments. Based on the structural information extracted by CA and the change intentions identified by TA, the Vulnerability Inspector (VI) examines modified code fragments and analyzes potential vulnerability-inducing patterns across several vulnerability categories. The analysis is not limited to the modified lines; when a change affects function behavior, variable usage, or call flow, the inspector further examines relevant surrounding code to assess potential risks. The inspection follows a category-driven analysis strategy based on common vulnerability types observed in real-world systems. Specifically, VI evaluates code changes with respect to six vulnerability types: I/O Validation, Memory Safety, Web Security, Authentication and Authorization, Resource Management, and File/Path Handling. If suspicious code fragments are identified, VI records the relevant code segments together with their associated vulnerability type and an explanation of the potential risk. These preliminary findings are then passed to the next inspection stage for further verification and refinement. This design is particularly important for VIC detection, where vulnerabilities are often subtle and may not exhibit explicit security violations at the time of introduction. By allowing the model to flag potentially risky patterns with incomplete evidence, this stage prioritizes recall and ensures that suspicious candidates are not prematurely discarded.
2.4
Stage 2: Reanalysis
While the preliminary inspection may produce uncertain or weakly supported candidates, this stage refines them through constrained reasoning over the hypothesis space. Each hypothesis is re-evaluated under additional constraints, including its predicted vulnerability type and contextual consistency with the commit. Compared to Stage 1, which prioritizes coverage, this stage focuses on improving the reliability of hypotheses by enforcing structured constraints. In practice, the Vulnerability Inspector (VI) performs type-guided reanalysis on each candidate fragment based on its predicted vulnerability category. In addition, the system may retrieve similar historical cases from the vulnerability knowledge base using a retrieval-augmented generation (RAG) mechanism. When available, similar historical cases are retrieved from the vulnerability knowledge base and incorporated as additional evidence. If no sufficiently similar cases are found, the analysis proceeds using only the available contextual information. This design enables case-based reasoning while avoiding overdependence on the knowledge base. The output of this stage is a refined set of fragment-level hypotheses, which are passed to the final decision stage.
6
2.5
Chen et al.
Stage 3: Final Decision
Although the reanalysis stage refines candidate hypotheses, uncertainty in LLM reasoning may still affect the final assessment. This stage produces a commitlevel vulnerability decision through structured verification and aggregation. Given the refined hypothesis set, each hypothesis is independently verified under a more conservative reasoning setting. A global decision is then derived by aggregating fragment-level results. In practice, VI performs fragment-level verification under a low-temperature configuration to produce more deterministic and conservative judgments. A commit is classified as vulnerability-inducing if at least one fragment is confirmed as vulnerable; otherwise, it is classified as non-vulnerable. If the commit is determined to be vulnerable, the Document Specialist (DS) generates a security report summarizing the detected issues, which is stored in the vulnerability knowledge base for future retrieval. Throughout the process, the Audit Supervisor (AS) monitors agent outputs and verifies whether they satisfy predefined requirements, ensuring consistency of the multi-stage reasoning process. 2.6
Vulnerability Knowledge Base
To support knowledge-augmented vulnerability reasoning, VIC-RAGENT maintains a vulnerability knowledge base that stores historical vulnerability analysis results generated during previous inspections. This knowledge base enables the system to reuse prior vulnerability cases during the reanalysis stage. The knowledge base is organized using a two-level structure. At the top level, reports are grouped by high-level vulnerability categories, and at the second level, each entry contains a security report describing a VIC. To enable efficient retrieval, VIC-RAGENT maintains a vector index over the stored cases. Each commit is encoded into a fixed-length embedding using a pretrained code model (e.g., CodeBERT [7]), and the embeddings are associated with the corresponding reports for similarity-based retrieval. During the reanalysis, the system retrieves similar historical cases based on embedding similarity. The retrieved cases provide reference examples that support case-based reasoning, helping the model assess whether the current commit exhibits similar vulnerability patterns. Whenever a commit is confirmed to be vulnerable by VIC-RAGENT, its security report and embedding are added to the knowledge base, enabling continuous expansion. To mitigate this issue, the system periodically removes reports corresponding to FP cases, retaining only reliable vulnerability instances and preventing error propagation.
3
Implementation
3.1
LLM Configuration
VIC-RAGENT relies on LLMs to perform reasoning tasks across different agent roles. We evaluate the framework using three LLMs: DeepSeek-V3.2 [18], Qwen-
Detecting VICs via Multi-Stage Reasoning with LLM-Based Agents
7
Table 2. Temperature configuration for different agent roles / Stages. Role / Stage Temperature CA TA VI (Stage 1) VI (Stage 2) VI (Stage 3) DS AS
0.2 0.2 0.4 0.4 0.1 0.3 0.1
Plus [1], and GPT-4o-mini [24]. Each agent role is implemented with the same LLM per experiment. VIC-RAGENT is evaluated using three LLMs: DeepSeekV3.2 [18], Qwen-Plus [1], and GPT-4o-mini [24]. All agent roles use the same underlying LLM within each experiment. Table 2 summarizes the temperature settings adopted for different stages and agent roles. The same prompts and workflows are used across all LLMs. To ensure a fair comparison, all experiments use the same prompts and agent workflows across all LLMs. 3.2
Agent Prompt Design
VIC-RAGENT uses role-specific prompt templates with JSON outputs for intermediate communication; full prompts and implementation details are publicly available. 3.3
Embedding and Retrieval
Commit diffs are encoded using CodeBERT [7] and stored in a vector index. During Stage 2, the current commit is encoded with the same model and matched against historical cases using cosine similarity. Only the most similar case is retrieved and used when similarity exceeds 0.85. Knowledge base maintenance. Since the knowledge base is incrementally updated using model-generated reports, it may be affected by false positive (FP) predictions. To mitigate this issue, we periodically remove reports corresponding to FP cases, retaining only reliable vulnerability instances. For example, if a commit is predicted as vulnerable by the preliminary analysis but the ground truth label indicates it is not, it is considered an FP and removed.
4
Experimental Setup
4.1
Research Questions
To evaluate the effectiveness of VIC-RAGENT for detecting VICs, we design a series of experiments to answer the following research questions.
8
Chen et al.
RQ1: How effective is VIC-RAGENT in detecting VICs compared with existing approaches? RQ2: How does the multi-agent collaboration contribute to vulnerability detection performance? RQ3: What is the computational cost of VIC-RAGENT, and how does it compare with baseline methods? RQ4: Can VIC-RAGENT generalize to unseen vulnerabilities without relying on memorized knowledge? 4.2
Dataset
We evaluate VIC-RAGENT on the V-SZZ dataset [2], which contains manually curated vulnerability-inducing commits (VICs) and vulnerability-fixing commits (VFCs). Following the original evaluation protocol, we start from 360 commits (172 VICs and 188 VFCs). To construct a binary classification dataset for VIC detection, commits labeled as both VIC and VFC are treated as VICs. We further remove commits that exceed the LLM context window, cannot be decoded, cannot be retrieved from the original repositories, or fail during framework execution. After preprocessing, the final dataset contains 241 commits, including 106 VICs and 135 VFCs. Among these commits, approximately 80% belong to paired vulnerability records, meaning that a VFC is associated with one or more corresponding VICs. 4.3
Baselines
We compare VIC-RAGENT with three representative baselines based on LLMs. Direct LLM. In this baseline, the commit information (including the commit message, code diff, and file context) is directly provided to the LLM, which produces a binary prediction indicating whether the commit introduces a vulnerability. This setting represents a vanilla prompting approach without structured reasoning or agent-based decomposition. Chain-of-Thought (CoT). This baseline extends direct prompting by instructing a single LLM to perform explicit step-by-step reasoning before producing the final binary decision. The model is asked to summarize the commit changes, identify security-relevant code fragments, analyze whether the changes may introduce a new vulnerability, and then output a commit-level yes/no prediction. CodeAgent. CodeAgent [32] is an LLM-based agent framework designed for automated code review. In CodeAgent, security analysis is treated as one of the code review subtasks. The framework requires the LLM agent to evaluate the security of a commit based on a predefined set of 25 vulnerability factors proposed by the authors. These factors capture common vulnerability patterns and risky coding practices, guiding the agent to assess whether a commit may introduce security risks.
Detecting VICs via Multi-Stage Reasoning with LLM-Based Agents
9
Table 3. Performance comparison. LLM
4.4
Method
Precision Recall F1-score
Direct CoT DeepSeek-V3.2 CodeAgent VIC-RAGENT
67% 86% 64% 70%
9% 6% 22% 48%
16% 11% 33% 57%
GPT-4o-mini
Direct CoT CodeAgent VIC-RAGENT
18% 57% 41% 50%
8% 15% 60% 75%
11% 24% 49% 60%
Qwen-Plus
Direct CoT CodeAgent VIC-RAGENT
75% 77% 52% 76%
28% 19% 58% 58%
41% 31% 55% 66%
Evaluation Metrics
Following prior work, we evaluate commit-level classification performance using Precision, Recall, and F1-score.
5
Experimental Results
5.1
Overall Performance
Table 3 presents the performance comparison between VIC-RAGENT and baseline methods across three LLMs. Substantial improvement in recall. Across all evaluated LLMs, VICRAGENT consistently achieves significantly higher recall than Direct and CoT baselines. This demonstrates that VIC-RAGENT is substantially more effective in identifying VICs, which is critical in security scenarios where missed vulnerabilities can lead to severe consequences [30]. Limitations of Direct and CoT baselines. Direct and CoT generally achieve high precision but extremely low recall. For instance, CoT with DeepSeekV3.2 achieves a precision of 86% but only 6% recall indicating that these methods are overly conservative. Comparison with CodeAgent. Although CodeAgent improves recall compared to Direct and CoT, it remains inferior to VIC-RAGENT across all LLMs. For example, under DeepSeek-V3.2, CodeAgent achieves an F1-score of 33%, whereas VIC-RAGENT reaches 57%. Similar trends are observed for GPT4o-mini and QWen-Plus. These results suggest that simply introducing agentbased reasoning is insufficient; instead, the structured multi-stage design of VICRAGENT plays a crucial role in improving effectiveness.
10
Chen et al. Table 4. Ablation results of VIC-RAGENT. Method w/o CA w/o TA w/o Stage 2 w/o Stage 3 VIC-RAGENT
Precision Recall F1-score 65% 64% 72% 67% 70%
35% 45% 43% 57% 48%
46% 53% 54% 62% 57%
Table 5. Confusion matrix comparison of ablation variants. Method
TP FP TN FN
w/o CA 37 w/o TA 48 w/o Stage 2 46 w/o Stage 3 60 VIC-RAGENT 51
5.2
20 115 69 24 111 58 18 117 60 30 105 46 22 113 55
Ablation Study
We conduct an ablation study using DeepSeek-V3.2 to evaluate the contribution of each component in VIC-RAGENT. Specifically, we evaluate the following variants: w/o Code Analyst (w/o CA): The CA is removed, and the VI directly analyzes the commit without structural information extracted from code changes. w/o Target Analyst (w/o TA): The TA is removed, and the VI performs vulnerability analysis without explicit guidance on commit intent and goal-file alignment. w/o Stage 2: The reanalysis stage is removed, and the VI makes final decisions solely based on the preliminary inspection without refinement. w/o Stage 3: The final decision stage is removed. The refined results from Stage 2 are directly used as the final prediction, without the additional fragmentlevel verification and commit-level aggregation performed in Stage 3. These variants allow us to evaluate the impact of structural context (CA), intent understanding (TA), Stage 2 reanalysis, and Stage 3 final decision on the overall performance of VIC-RAGENT. Table 4 and Table 5 present the results of the ablation study. Impact of Stage 2 reanalysis. Removing Stage 2 increase precision but reduces recall and F1-score. The confusion matrix shows a decrease in TP (51→46) and an increase in FN (55→60), indicating that more vulnerabilities are missed. This suggests that Stage 2 plays an important role in improving coverage. Given candidate fragments and their predicted vulnerability categories, the model can perform more targeted and type-guided reasoning under a narrowed search space. This refinement step primarily serves to re-examine and disambiguate the suspicious fragments identified in Stage 1, allowing the model to make more informed
Detecting VICs via Multi-Stage Reasoning with LLM-Based Agents
11
Table 6. Average cost per commit evaluated with DeepSeek-V3.2. Method
Query Time (s) Cost in USD
Direct Cot CodeAgent VIC-RAGENT
5 17 80 103
0.0079 0.0083 0.0217 0.0487
decisions on borderline or uncertain cases. As a result, more potential vulnerabilities are retained as positive predictions, which improves recall but may also introduce additional false positives. Therefore, Stage 2 is a key component in the coarse-to-fine reasoning process, improving coverage while introducing a moderate trade-off between recall and precision, and contributing to better overall detection effectiveness. Impact of Stage 3 final decision. Removing Stage 3 increase recall and F1-score. However, the confusion matrix shows that this gain is accompanied by lower prediction reliability. Specifically, when Stage 3 is removed, FP increases from 22 to 30, while TN decreases from 113 to 105. This provides direct evidence that Stage 3 filters out a subset of false positives that remain after Stage 2. At the same time, TP decreases from 60 to 51 and FN increases from 46 to 55 after Stage 3 is enabled, showing that the additional verification step also filters out some true positives. Therefore, Stage 3 introduces a clear trade-off: it sacrifices some recall in exchange for stricter confirmation and improved robustness. Overall, these results show that Stage 3 acts as a conservative filtering mechanism over the refined candidates produced by Stage 2. Its primary role is not to maximize F1-score, but to improve the trustworthiness of final predictions by reducing unreliable positive decisions. In practical vulnerability detection scenarios, this behavior can be desirable because excessive false positives may lead to unnecessary security alerts and increased manual inspection effort. 5.3
Efficiency and Cost Analysis
We evaluate the computational efficiency of VIC-RAGENT by comparing its query latency and monetary cost with baseline methods. Table 6 reports the average cost per commit using DeepSeek-V3.2. The reported cost is calculated based on the cache miss pricing, which provides a conservative upper-bound estimate of the actual cost. Higher cost due to multi-stage reasoning. Compared to Direct and CoT, which require only a single LLM call, VIC-RAGENT incurs a higher cost due to multiple agent interactions across different reasoning stages. This increase is expected, as VIC-RAGENT performs structured multi-stage reasoning instead of single-pass inference. Cost-effectiveness compared to CodeAgent. Compared with CodeAgent, VIC-RAGENT exhibits a moderate increase in cost, while achieving substantially higher recall and F1-score. This suggests that the additional cost is justi-
12
Chen et al. Table 7. Performance on recent CVEs (DeepSeek-V3.2). Method Direct CoT CodeAgent VIC-RAGENT
Precision Recall F1-score 80% 100% 63% 67%
36% 18% 45% 55%
50% 31% 53% 60%
fied by improved detection effectiveness. Although VIC-RAGENT introduces additional LLM calls across multiple stages, the design is suitable for offline or CI-integrated security auditing scenarios, where accuracy is prioritized over latency. The modular structure also allows early-stage filtering to reduce unnecessary downstream analysis. 5.4
Generalization on Recent Vulnerabilities.
To evaluate the generalization ability of VIC-RAGENT under realistic conditions, we conduct experiments on a set of commits associated with recently disclosed vulnerabilities. The evaluation dataset consist of 20 commits collected from 11 CVEs, including 11 VICs and 9 VFCs. Importantly, the selected CVEs are published between March 2025 and March 2026, with the majority (10 CVEs) released after November 2025. These vulnerabilities are likely to be beyond the training cutoff of DeepSeek-V3.2, making it unlikely that the model has prior exposure to these specific cases. Table 7 presents the performance of different methods on commits associated with recently disclosed vulnerabilities using DeepSeek-V3.2. Strong generalization to unseen vulnerabilities. VIC-RAGENT achieves the best overall performance, with the highest recall, outperforming all baselines. This indicates that VIC-RAGENT is more effective in identifying VICs even when vulnerabilities are newly disclosed. Effectiveness of multi-stage reasoning. Compared to CodeAgent, VICRAGENT improves recall and F1-score, demonstrating the benefit of the proposed multi-stage reasoning process. This suggests that structured candidate generation and refinement are essential for detecting vulnerabilities in previously unseen scenarios.
6
Discussion
Effect of Multi-Stage Reasoning. The multi-stage reasoning process follows a coarse-to-fine paradigm. The preliminary inspection stage prioritizes recall by generating candidate vulnerability hypotheses, while the reanalysis stage improves reliability through type-guided verification. Together, these stages balance exploration and verification, contributing to improved detection performance.
Detecting VICs via Multi-Stage Reasoning with LLM-Based Agents
13
Limitations. Despite its effectiveness, VIC-RAGENT has several limitations. First, it relies on LLM reasoning, which may still produce inconsistent results in complex scenarios. Second, the effectiveness of the reanalysis stage depends on the availability and relevance of retrieved cases. Third, the final decision stage introduces a conservative bias that may reduce recall for subtle vulnerabilities. Finally, the framework focuses on commit-level detection and does not address fine-grained tasks such as vulnerability localization or patch generation.
7
Threats to Validity
We discuss internal and external validity threats for VIC-RAGENT. Internally, although the multi-stage design improves reliability, LLMs may produce inconsistent or incorrect judgments. Prompt design and temperature settings can also affect outcomes. Iterative refinement mitigates but does not fully eliminate errors. Externally, our evaluation on the V-SZZ dataset may not generalize to all projects, programming languages, or real-world environments, and performance could vary across different codebases or vulnerability patterns.
8
Related Work
Just-In-Time Vulnerability Detection. Early JIT vulnerability detection approaches rely on handcrafted features and traditional machine learning models [26, 36, 22, 27]. More recent studies has explored deep learning techniques to learn representations from source code and commit diffs, including DeepJIT [9], CodeJIT [21], and HgtJIT [31]. Although these approaches improve semantic modeling, they still struggle to capture developer intent and broader contextual information associated with commits. Code Vulnerability Detection. Code vulnerability detection has evolved from role-based and static-analysis approaches to learning-based methods. Deep learning models such as VulDeePecker [16] and VulDeeLocator [15] learn vulnerability patterns from code representations, while graph-based approaches (e.g., Devign [40], BGNN4VD [3], COCA [4]) model structural dependencies in programs. More recently, transformer-based models such as LineVul [8] leverage pretrained language models for vulnerability prediction. However, these methods primarily focus on static code snippets and are not designed for commit-level vulnerability reasoning. LLM-based Vulnerability Analysis. Recent advances in LLMs have demonstrated strong capabilities in code understanding and vulnerability analysis [34, 11, 13, 19, 39, 17]. Existing approaches include prompt-based methods, retrievalaugmented methods [6], and agent-based framework such as CodeAgent [32]. ReAct-style agents [38] further introduce iterative reasoning through thought– action–observation loops. However, existing LLM-based approaches primarily focus on source-code vulnerability analysis or general code review, and often rely on single-agent or single-pass reasoning. As summarized by recent surveys [39],
14
Chen et al.
current approaches remain sensitive to prompt design and may produce unstable reasoning results. These limitations motivate our work, which focuses on commit-level vulnerability detection and employs structured role decomposition together with multi-stage verification.
9
Conclusion
In this paper, we propose VIC-RAGENT, an LLM-based multi-agent framework for detecting vulnerability-inducing commits. The framework formulates vulnerability detection as a structured multi-stage reasoning process that integrates code diffs, commit messages, and contextual information. Experimental results show that VIC-RAGENT consistently outperforms baseline methods across multiple LLMs, achieving up to 2× higher recall and 1.2–1.7× higher F1-scores compared to the strongest baseline. These results demonstrate the effectiveness of coarse-to-fine reasoning for commit-level vulnerability detection, enabling improved coverage while maintaining reliable decision making. In future work, we plan to explore more adaptive decision strategies and extend the framework to support fine-grained vulnerability classification. Acknowledgments. This work was supported by the National Key R&D Program of China No 2024YFB4506200.
References 1. Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al.: Qwen technical report. arXiv preprint arXiv:2309.16609 (2023) 2. Bao, L., Xia, X., Hassan, A.E., Yang, X.: V-szz: automatic identification of version ranges affected by cve vulnerabilities. In: Proceedings of the 44th international conference on software engineering. pp. 2352–2364 (2022) 3. Cao, S., Sun, X., Bo, L., Wei, Y., Li, B.: Bgnn4vd: Constructing bidirectional graph neural-network for vulnerability detection. Information and Software Technology 136, 106576 (2021) 4. Cao, S., Sun, X., Wu, X., Lo, D., Bo, L., Li, B., Liu, W.: Coca: Improving and explaining graph neural network-based vulnerability detection systems. In: Proceedings of the IEEE/ACM 46th International Conference on Software Engineering. pp. 1–13 (2024) 5. Chen, L., Sun, H., Gao, X., Shi, L., Yang, Y., Xu, Y.: Vulnerability identification by harnessing inter-connected multi-source information. Journal of Systems and Software 241, 113001 (2026) 6. Du, X., Zheng, G., Wang, K., Zou, Y., Wang, Y., Deng, W., Feng, J., Liu, M., Chen, B., Peng, X., et al.: Vul-rag: Enhancing llm-based vulnerability detection via knowledge-level rag. ACM Transactions on Software Engineering and Methodology (2024) 7. Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al.: Codebert: A pre-trained model for programming and natural languages. In: Findings of the association for computational linguistics: EMNLP 2020. pp. 1536–1547 (2020)
Detecting VICs via Multi-Stage Reasoning with LLM-Based Agents
15
8. Fu, M., Tantithamthavorn, C.: Linevul: A transformer-based line-level vulnerability prediction. In: Proceedings of the 19th international conference on mining software repositories. pp. 608–620 (2022) 9. Hoang, T., Dam, H.K., Kamei, Y., Lo, D., Ubayashi, N.: Deepjit: an end-to-end deep learning framework for just-in-time defect prediction. In: 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR). pp. 34–45. IEEE (2019) 10. Iannone, E., Guadagni, R., Ferrucci, F., De Lucia, A., Palomba, F.: The secret life of software vulnerabilities: A large-scale empirical study. IEEE Transactions on Software Engineering 49(1), 44–63 (2022) 11. Jiang, J., Wang, F., Shen, J., Kim, S., Kim, S.: A survey on large language models for code generation. ACM Transactions on Software Engineering and Methodology 35(2), 1–72 (2026) 12. Jiang, M., Jiang, J., Wu, T., Ma, Z., Luo, X., Zhou, Y.: Understanding vulnerability inducing commits of the linux kernel. ACM Transactions on Software Engineering and Methodology 33(7), 1–28 (2024) 13. Li, B., Wu, W., Tang, Z., Shi, L., Yang, J., Li, J., Yao, S., Qian, C., Hui, B., Zhang, Q., et al.: Prompting large language models to tackle the full software development lifecycle: A case study. In: Proceedings of the 31st International Conference on Computational Linguistics. pp. 7511–7531 (2025) 14. Li, X., Wang, S., Zeng, S., Wu, Y., Yang, Y.: A survey on llm-based multi-agent systems: workflow, infrastructure, and challenges. Vicinagearth 1(1), 1–43 (2024) 15. Li, Z., Zou, D., Xu, S., Chen, Z., Zhu, Y., Jin, H.: Vuldeelocator: a deep learningbased fine-grained vulnerability detector. IEEE Transactions on Dependable and Secure Computing 19(4), 2821–2837 (2021) 16. Li, Z., Zou, D., Xu, S., Ou, X., Jin, H., Wang, S., Deng, Z., Zhong, Y.: Vuldeepecker: A deep learning-based system for vulnerability detection. arXiv preprint arXiv:1801.01681 (2018) 17. Lin, J., Mohaisen, D., et al.: From large to mammoth: A comparative evaluation of large language models in vulnerability detection. In: NDSS (2025) 18. Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al.: Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024) 19. Lu, G., Ju, X., Chen, X., Pei, W., Cai, Z.: Grace: Empowering llm-based software vulnerability detection with graph structure and in-context learning. Journal of Systems and Software 212, 112031 (2024) 20. Motwani, S.R., Smith, C., Das, R.J., Rafailov, R., Laptev, I., Torr, P.H.S., Pizzati, F., Clark, R., de Witt, C.S.: Malt: Improving reasoning with multi-agent llm training. arXiv preprint arXiv:2412.01928 (2024) 21. Nguyen, S., Nguyen, T.T., Vu, T.T., Do, T.D., Ngo, K.T., Vo, H.D.: Code-centric learning-based just-in-time vulnerability detection. Journal of Systems and Software 214, 112014 (2024) 22. Ni, C., Wang, W., Yang, K., Xia, X., Liu, K., Lo, D.: The best of both worlds: integrating semantic features with expert features for defect prediction and localization. In: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. pp. 672– 683 (2022) 23. Ni, C., Yin, X., Shen, L., Wang, S.: Learning-based models for vulnerability detection: An extensive study. Empirical Software Engineering 31(1), 18 (2026) 24. OpenAI: Gpt-4o mini: advancing cost-efficient intelligence. https://openai.com/ index/gpt-4o-mini-advancing-cost-efficient-intelligence/ (2024)
16
Chen et al.
25. Oriol, M., Müller, C., Marco, J., Fernandez, P., Franch, X., Ruiz-Cortés, A.: Comprehensive assessment of open source software ecosystem health. Internet of Things 22, 100808 (2023) 26. Pornprasit, C., Tantithamthavorn, C.K.: Jitline: A simpler, better, faster, finergrained just-in-time defect prediction. In: 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR). pp. 369–379. IEEE (2021) 27. Riom, T., Sawadogo, A., Allix, K., Bissyandé, T.F., Moha, N., Klein, J.: Revisiting the vccfinder approach for the identification of vulnerability-contributing commits. Empirical Software Engineering 26(3), 46 (2021) 28. Saha, R.K., Khurshid, S., Perry, D.E.: An empirical study of long lived bugs. In: 2014 Software Evolution Week - IEEE Conference on Software Maintenance, Reengineering, and Reverse Engineering (CSMR-WCRE). pp. 144–153 (2014). https://doi.org/10.1109/CSMR-WCRE.2014.6747164 29. Shen, Y., Gao, X., Sun, H., Guo, Y.: Understanding vulnerabilities in software supply chains. Empirical Software Engineering 30(1), 20 (2025) 30. Steenhoek, B., Rahman, M.M., Jiles, R., Le, W.: An empirical study of deep learning models for vulnerability detection. In: 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). pp. 2237–2248 (2023). https://doi.org/10.1109/ICSE48619.2023.00188 31. Sun, X., Zhou, M., Cao, S., Wu, X., Bo, L., Wu, D., Li, B., Xiang, Y.: Hgtjit: Justin-time vulnerability detection based on heterogeneous graph transformer. IEEE Transactions on Dependable and Secure Computing (2025) 32. Tang, X., Kim, K., Song, Y., Lothritz, C., Li, B., Ezzini, S., Tian, H., Klein, J., Bissyandé, T.F.: Codeagent: Autonomous communicative agents for code review. In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. pp. 11279–11313 (2024) 33. Tian, W., Lin, Y., Gao, X., Sun, H.: Enhanced vulnerability localization: Harmonizing task-specific tuning and general llm prompting. In: 2025 IEEE International Conference on Software Maintenance and Evolution (ICSME). pp. 110–122. IEEE (2025) 34. Wang, J., Ni, T., Lee, W.B., Zhao, Q.: A contemporary survey of large language model assisted program analysis. arXiv preprint arXiv:2502.18474 (2025) 35. Woo, S., Choi, E., Lee, H.: A large-scale analysis of the effectiveness of publicly reported security patches. Computers & Security 148, 104181 (2025) 36. Yang, L., Li, X., Yu, Y.: Vuldigger: A just-in-time and cost-aware tool for digging vulnerability-contributing changes. In: GLOBECOM 2017-2017 IEEE Global Communications Conference. pp. 1–7. IEEE (2017) 37. Yang, Y., Xu, B., Gao, X., Sun, H.: Context-enhanced vulnerability detection based on large language models. ACM Transactions on Software Engineering and Methodology (2025) 38. Yildiz, A., Teo, S.G., Lou, Y., et al.: Benchmarking llms and llm-based agents in practical vulnerability detection for code repositories. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 30848–30865 (2025) 39. Zhou, X., Cao, S., Sun, X., Lo, D.: Large language model for vulnerability detection and repair: Literature review and the road ahead. ACM Transactions on Software Engineering and Methodology 34(5), 1–31 (2025) 40. Zhou, Y., Liu, S., Siow, J., Du, X., Liu, Y.: Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks. Advances in neural information processing systems 32 (2019)