Conceptio › Archive › arXiv CS

arXiv CSopen access

Taint-Style Vulnerability Detection and Confirmation for Node.js Packages Using LLM Agent Reasoning

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

cryptography cybersecurity privacy security

cryptography, security, privacy, cybersecurity

Taint-Style Vulnerability Detection and Confirmation for Node.js Packages Using LLM Agent Reasoning Ronghao Ni

Mihai Christodorescu

Limin Jia

[email protected] Carnegie Mellon University

[email protected] Google

[email protected] Carnegie Mellon University

code-property-graph-based static analysis, symbolic execution, and constraint-based synthesis. While successful in uncovering many real-world vulnerabilities, these tools share several fundamental challenges stemming from inherent limitations of the underlying analysis techniques. First, JavaScript’s highly dynamic nature makes accurate modeling of language semantics difficult. The lack of precise type information further complicates analysis. Second, Node.js native (built-in) functions are implemented in C++, requiring either instrumentation of the V8 engine or manually constructed abstractions. Third, these tools rely on external JavaScript analysis infrastructure, such as parsers [2], transpilers [1], and instrumentation tools [44]. These dependencies are inherently brittle due to JavaScript’s complex language features and evolving standards. Finally, many approaches depend on satisfiability modulo theories (SMT) solvers, which often fail to handle constraints involving string operations and regularexpression matching. Large Language Models (LLMs) have shown strong performance on coding-related tasks, including code generation and code comprehension [11]. Rather than relying on purpose-fit abstractions, LLMs leverage extensive pre-trained knowledge and reasoning abilities to comprehend complex code patterns and dynamic program behaviors. Furthermore, LLM agents can iteratively refine their outputs by incorporating feedback from previous unsuccessful attempts. These capabilities suggest LLMs may be able to overcome the limitations of traditional program analysis techniques. This paper aims to answer the following question: Can an LLMcentric, tool-augmented workflow effectively detect and confirm vulnerabilities in npm packages without dedicated static/dynamic analysis engines? To answer this question, we design and implement LLMVD.js, a ReAct-based [54] agent that leverages large language models to perform taint-style vulnerability detection and confirmation for Node.js packages. We evaluate LLMVD.js on three datasets (existing public benchmarks, one private benchmark, and a set of recently released npm packages) and show that it confirms 84% of public benchmark vulnerabilities with valid exploits, substantially outperforming prior program-analysis tools and also outperforming an LLM+program-analysis hybrid system [47] while requiring significantly less prior information, and further discovers 36 vulnerabilities in recently released packages. For the rest of this paper, we use rule-based program analysis to refer to analysis techniques such as symbolic execution and taint analysis that do not rely on machine learning components. We use LLM-centric reasoning to refer to LLM-agent reasoning over raw source code with lightweight tooling (e.g., search, execution, and oracles), but without dedicated static/dynamic analysis engines for taint/path derivation. Our contributions are as follows:

arXiv:2604.20179v1 [cs.CR] 22 Apr 2026

Abstract The rapidly evolving Node.js ecosystem currently includes millions of packages and is a critical part of modern software supply chains, making vulnerability detection of Node.js packages increasingly important. However, traditional program analysis struggles in this setting because of dynamic JavaScript features and the large number of package dependencies. Recent advances in large language models (LLMs) and the emerging paradigm of LLM-based agents offer an alternative to handcrafted program models. This raises the question of whether an LLM-centric, tool-augmented approach can effectively detect and confirm taint-style vulnerabilities (e.g., arbitrary command injection) in Node.js packages. We implement LLMVD.js, a multi-stage agent pipeline to scan code, propose vulnerabilities, generate proof-of-concept exploits, and validate them through lightweight execution oracles; and systematically evaluate its effectiveness in taint-style vulnerability detection and confirmation in Node.js packages without dedicated static/dynamic analysis engines for path derivation. For packages from public benchmarks, LLMVD.js confirms 84% of the vulnerabilities, compared to less than 22% for prior program analysis tools. It also outperforms a prior LLM–program-analysis hybrid approach while requiring neither vulnerability annotations nor prior vulnerability reports. When evaluated on a set of 260 recently released packages (without vulnerability groundtruth information), traditional tools produce validated exploits for few (≤ 2) packages, while LLMVD.js generates validated exploits for 36 packages.

Keywords Automatic Vulnerability Detection, Node.js, Large Language Models, ReAct Agents, LLM Agents, Exploit Generation, Vulnerability Confirmation

Introduction

The Node.js ecosystem comprises millions of JavaScript packages and is among the most widely used software platforms today. However, numerous studies have shown that a substantial fraction of these packages contain security vulnerabilities [15, 58] and have been exploited in software supply-chain attacks [30, 40]. To ensure application security, it is critical to be able to identify vulnerabilities within this ecosystem and in recent years researchers have proposed tools to automatically detect and confirm vulnerabilities in Node.js packages [9, 10, 28, 35]. These tools primarily target taint-style vulnerabilities, including OS command injection, code injection, prototype pollution, and path traversal. These tools use a wide range of analysis techniques, such as dynamic taint analysis, Preprint. 1

Ronghao Ni, Mihai Christodorescu, and Limin Jia

• A systematic evaluation of a multi-stage ReAct-style LLM agent framework for taint-style vulnerability detection and confirmation in Node.js packages, with direct comparison against state-of-the-art program-analysis tools and a program-analysis–aided LLM approach. • A multi-dataset evaluation setup for LLM-agent vulnerability research: combining standard public benchmarks, transformed benchmark variants for memorization robustness checks, a private real-world dataset without CVEs/public exploits, and recently released npm packages to assess generalizability under realistic settings. • LLMVD.js identified 36 previously undocumented vulnerabilities in recently released Node.js packages.

Listing 1: Code snippet of a vulnerable API 1 2 3 4 5 6 7 8 9

Listing 2: PoC exploit

Ethical Considerations Our work raises inherent dual-use concerns due to LLMVD.js’s ability to automatically detect vulnerabilities and generate exploits; however, we believe that the defensive benefits outweigh the associated risks. All experiments were conducted in sandboxed environments; no production systems or external servers were targeted. We analyze only open-source packages. We reported all 36 previously unreported validated vulnerabilities identified in newly released packages to maintainers and have received acknowledgments from 3 maintainers.

1 2 3 4 5 6

const Arpping = require('./index'); (async () => { const a = new Arpping({ timeout: 1 }); const payload = ['127.0.0.1; touch /tmp/ os_cmd_success']; await a.ping(payload); })();

Figure 1: The vulnerable npm package [email protected] (Snyk ID: SNYK-JS-ARPPING-1060047).

Background and Related Work

Vulnerability detection is the task of identifying security-relevant flaws in software that may be exploited by adversaries [12, 31, 42, 43]. Vulnerability detection tools typically only report potential vulnerabilities and a separate confirmation step is needed to remove false positives. To reduce the costly manual confirmation effort, researchers have developed automated vulnerability confirmation methods that synthesize proof-of-concept (PoC) exploits [5, 9, 10, 35]. We review most recent work on vulnerability detection and confirmation of Node.js packages and applying LLMs to vulnerability detection.

2.1

const { exec } = require('child_process'); Arpping.prototype.ping = function(range) { ... return new Promise((resolve, reject) => { range.forEach(ip => { exec(`ping ${flag} ${this.timeout} ${ip}`, ( err, stdout, stderr) => { ... });});}); }

vulnerability detection is reduced to a graph query. NodeMedicFINE on the other hand, instruments JavaScript at the source level to implement dynamic taint tracking for vulnerability detection. The current implementation of NodeMedic-FINE only detects command injection and code injection vulnerabilities. Vulnerability confirmation via exploit generation. To generate PoC exploits, the tools need to generate a driver (testing harness) that can trigger a call to the vulnerable API (sink) and find inputs that can not only reach the sink, but also deliver the desired attack payload (e.g., a command of attacker’s choice for command injection vulnerabilities). For example, Listing 1 shows one vulnerable API in the arpping package (Snyk ID: SNYK-JS-ARPPING-1060047). The ping function constructs a command string using attacker-controlled input range and passes it to the exec function, leading to an OS command injection vulnerability. To confirm this vulnerability, the tool needs to generate a driver (Listing 2) that calls the ping function with an input that injects an attacker-controlled command. For inputs, FAST and Explode.js use symbolic execution to identify path constraints to reach the vulnerable API. NodeMedic-FINE synthesizes input constraints from the taint provenance graph, which is an output from the dynamic taint analysis and documents all the operations that the tainted inputs underwent before reaching the sink. Only Explode.js is capable of generating drivers that can chain multiple API calls to reach the sink. It does so by querying its custom code property graph to identify a linear call chain. NodeMedic-FINE and FAST, instead, use a fix template to directly call the vulnerable API. All tools rely on SMT solvers to resolve constraints.

Node.js Vulnerability Detection and Confirmation

Node.js Taint-style Vulnerability Detection. Recent vulnerability detection tools for Node.js packages [9, 10, 28, 29, 35, 39] focus on taint-style vulnerabilities, partly because they have easy to detect code patterns and partly because they can lead to serious consequences such as allowing attackers to inject arbitrary code or execute arbitrary commands. For detection, these tools need to identify tainted paths from attacker controlled inputs to arguments of a sink. In the case of command injection, code injection, and path traversal, the sinks are known APIs such as exec, Function, File.Write. For prototype pollution, the vulnerability pattern involves two tainted paths and specific object field accesses. To compare against LLM-centric agent reasoning without dedicated static/dynamic analysis engines, we use FAST [28], NodeMedic-FINE [9], and Explode.js [35] as representative examples of rule-based program analysis tools. FAST and Explode.js use code-property-graph (CPG) based methods for detection. They generate (their own custom) graphs representing information such as dependency, object relationship, and key operations; then 2

Taint-Style Vulnerability Detection and Confirmation for Node.js Packages Using LLM Agent Reasoning

2.2

Table 1: Limitations of three representative programanalysis-based tools for Node.js taint-style vulnerability detection and confirmation: FAST [28], NodeMedic-FINE [9], Explode.js [35]

LLMs in Software Vulnerability Detection

Drivers may include complex interactions that make multiple API calls, set up global environment correctly, and construct and invoke callbacks.

FAST NodeMedicFINE Explode.js

Does not generate drivers. Uses fixed driver templates and cannot handle complex interactions. Supports only linear call chains

C2 Hard to analyze code units

Imported dependencies significantly increase the complexity of the analysis. Native operations are managed internally by the JavaScript engine and therefore need delicate custom handling.

C3 Lacking info.

type

Needed manual modeling of native functions. Current implementation is missing significant (> 90%) support. Applies over-approximated tainting policy.

Limits

Tools

FAST

Limits

Why challenging

C1 Generating drivers for PoC

NodeMedicFINE Explode.js

Needs manually crafted symbolic summaries for imported APIs, resulting in low detection rate in Node.js packages in the wild. Analysis needs arguments of the correct type and object structure.

C4 Reliant on other JavaScript analysis infrastructure

Analysis tools need another set of complex tools such as parsers, transpilers, and instrumentation tools, to implement their custom analysis. These tools are brittle due to JavaScript’s standard evolutions and complex features.

FAST NodeMedicFINE Explode.js

Esprima [2] parsing errors and call-edge issues. Jalangi2’s [44] lack of support for ES6+

C5 Reliant on SMT

The analysis generates constraints on string operations and regular expression matching, which are difficult to handle for SMT solvers and is an active area of research.

FAST NodeMedicFINE Explode.js

Z3 [14] timeout when solving path constraints. SMT solver timeout when solving constraints on input. SMT solver timeout when solving path and input constraints.

Tools

Limits

Ignore types Algorithms for reconstructing types, neither sound nor complete Querying the CPG to reconstruct types, neither sound nor complete

Limits

FAST NodeMedicFINE Explode.js

Tools

Graph.js [21] exits with errors for many packages.

Tools

Limits

LLM Agents in Security Auditing. A parallel line of work studies LLMs as agents that can plan, use tools, and iteratively refine hypotheses, moving beyond single-pass vulnerability labeling toward multi-step auditing [22, 25, 45, 57]. PentestGPT formalizes an LLMdriven penetration-testing workflow with modular agent roles that decompose high-level objectives into actionable testing steps and tool interactions [16]. More broadly, recent studies demonstrate that agentic LLM systems can exploit real-world known vulnerabilities from vulnerability descriptions, highlighting both the potential and the risks of autonomous offensive capability when paired with tool use and structured memory [20]. However, to our knowledge, no prior work has studied the performance and limitations of a carefully designed LLM agent for end-to-end taint-style vulnerability detection and confirmation in Node.js packages, which is the focus of this paper.

Challenges

Tools

LLM-assisted Security Audit. Recent work has explored using LLMs as assistants for security auditing, either as standalone vulnerability detectors or as components that augment traditional program analysis workflows [26, 46, 52, 56]. Several studies systematically evaluate the capability of LLMs on vulnerability detection and code analysis tasks. For example, Fang et al. analyze the strengths and failure modes of LLMs for code reasoning in security-relevant settings [19], while Lin et al. conduct a large-scale comparative evaluation of LLM configurations across multiple datasets and programming languages [34]. Similar evaluation efforts further characterize prompt sensitivity, model scale effects, and generalization limits in vulnerability detection [37, 41]. LLMs have been increasingly integrated into end-to-end auditing pipelines that combine detection, testing, and repair. Prior work demonstrates LLM-guided fuzzing and protocol testing, where models infer grammars and message sequences to improve coverage for stateful implementations [36, 49]. Other systems leverage LLMs to assist patch generation and automated repair under realistic constraints [18, 38]. LLMs have also been integrated into Node.js vulnerability analysis [24, 32, 33]. Most closely related to our work, PoCGen utilizes LLMs alongside static and dynamic analyses to interpret vulnerability reports, draft candidate exploits, and iteratively validate and refine them for npm vulnerabilities [47]. PoCGen relies on CodeQL-based static taint analysis to identify candidate vulnerable functions and input-to-sink paths, and employs dynamic analysis by executing generated exploits in a sandboxed Node.js environment with vulnerability-specific runtime oracles to validate exploit success and guide refinement. PoCGen demonstrates the effectiveness of tightly integrating LLMs with program analysis and argues that plain LLM-based agents are insufficient for this task [47]; in contrast, we investigate whether a carefully designed LLM reasoning–only agent can achieve competitive performance. Accordingly, we include PoCGen as a baseline in our evaluation.

Motivation

In this section, we discuss fundamental challenges that rule-based program analysis tools face when detecting and confirming taintstyle vulnerabilities in Node.js packages and outline why the capabilities of LLMs in a ReAct-based agent design framework suit this task.

3.1

Challenges in Rule-based Program Analysis

Recall from Section 2.1 state-of-the-art rule-based tools [9, 28, 35] implement dynamic taint tracking and code-property-graph-based static analysis for detection; and leverages symbolic execution and 3

Ronghao Ni, Mihai Christodorescu, and Limin Jia

1.0

constraint-based synthesis for generating inputs for PoCs. Some use [35] CPG-based static analysis for generating drivers that include complex code patterns (i.e., not directly call the vulnerable API). Despite substantial improvements, these tools continue to face challenges arising from the highly dynamic nature of JavaScript, the ongoing evolution of the JavaScript language, large and complex dependencies, and their reliance on other sophisticated analysis infrastructures. These challenges are common across existing tools and stem from the fundamental limitations of the underlying techniques, rather than from limitations of individual implementation choices. We summarize these challenges in Table 1 and explain how each tool partially addresses them. In practice, these limitations often lead to missed vulnerabilities or failures in exploit confirmation. Although these tools will continue to improve, as long as the same core techniques are employed, progress is likely to be incremental [7, 8]. Alternatively, tools may be increasingly specialized, exploring different trade-off spaces to achieve high efficiency for specific vulnerability classes or to target different application domains (e.g., Mini apps [48, 55], React web apps [23], and Electron apps [4, 27]).

90% = 32,448

Cumulative Probability

0.8

75%

0.6 50%

0.4 0.2 0.0

102

104

103

Code Injection OS Command Injection 25% Path Traversal Prototype Pollution Total (All Types) 105 106

Token Count (log scale)

(a) SecBench.js & VulcaN datasets 1.0 90% = 147,025

Cumulative Probability

0.8

90% 75%

0.6 50%

0.4 25%

0.2

3.2

90%

Advantages of LLM Agents 0.0100

Reasoning beyond handcrafted program models. As summarized in Table 1, many challenges faced by rule-based tools are from the need to construct and maintain accurate program models, which is often at odds with scalability and requires substantial manual effort and domain expertise. Moreover, there is a steep increase in the effort required for improving the analysis to cover additional features, once core behaviors have been modeled. In contrast, learning-based approaches, especially large language models in the current era, provide a promising alternative by utilizing their extensive pre-trained knowledge and reasoning abilities to comprehend complex code patterns, library usages, and dynamic behaviors without requiring exhaustive manual modeling. As a result, they can adapt more readily to evolving programming practices and software ecosystems, where manually maintaining precise program models becomes increasingly impractical.

101

102

103

104

All Packages Total (All Types) 105 106 107

Token Count (log scale)

(b) Recently crawled npm packages (17,151 packages)

Figure 2: Cumulative distribution function (CDF) of token counts using the gpt-5-mini tokenizer. Blue dashed lines mark the 90th percentile for the combined datasets: 32,448 tokens for SecBench.js & VulcaN and 147,025 tokens for recently crawled npm packages. Only JavaScript files (with extensions .js, .jsx, .mjs, and .cjs) are included in the count. We exclude TypeScript files because most published npm packages distribute transpiled JavaScript artifacts for execution, and our analysis focuses on code that is directly executed in production.

Oracle-Guided Iterative Reasoning. Another important factor that makes LLM agents well-suited for taint-style vulnerability detection and confirmation tasks in Node.js packages is that LLM agents can iteratively refine their answers based on feedback from previous unsuccessful attempts. In contrast, traditional program-analysis techniques typically perform one-shot reasoning over a fixed program representation. While iterative counter-example guided abstraction refinement (CEGAR) methods [13] have been applied to domains such as model checking, custom algorithm design and significant engineering effort is needed for a specific tool to benefit from CEGAR. Moreover, it is easy to design and implement a testing oracle for taint-style vulnerabilities by observing side effects, commonly used in previous work [9, 28, 35]. LLM agents can then generate candidate inputs, execute them against the target package, and refine their reasoning based on observed outcomes.

limit the amount of code that can be processed effectively in a single pass [11, 17, 51]. However, when considering the natural distribution of real-world npm packages, we observe that in widely used benchmarks and empirical datasets, most npm packages are sufficiently small that they do not stress the context limits of modern LLMs. For instance, as shown in Figure 2, the majority of packages in the VulcaN and SecBench.js datasets contain relatively small amounts of code, with 90th percentile token counts below 32,449. Similarly, in the recently crawled npm packages (which will be discussed in Section 5.1.1), the cumulative distribution over token counts exhibits a comparable trend, where 90th percentile token counts are below 147,026. This measures the total sizes of the codebase, but in reality, an LLM agent will not load the entire codebase into its context window since it can focus on specific files and functions relevant to the vulnerability detection task. However, even considering this, the 90th percentile token counts are still well within the context window sizes of recent LLM models, such as

Favorable code size distribution in the npm ecosystem. Despite the strong reasoning capabilities of large language models, current LLMs remain constrained by finite context window sizes, which 4

Taint-Style Vulnerability Detection and Confirmation for Node.js Packages Using LLM Agent Reasoning

Iterative exploit attempts Finder

{

Environment Setup

LLM Agent

{

"vuln_type": "prototype_pollution", { "location": { " le": "utils.js", "line": 34 Findings }, "vuln_type": "path_traversal", "description": is merged into "location": {"User " le":input "server.js", "line": 87an }, { object without ltering special keys,path allowing "description": "User-controlled is "vuln_type": "os_command_injection", prototype pollution.", joined with the base directory without "location": { " le": "index.js", "line": 12 }, "evidence": "Object.assign({}, defaults, validation.", "description": "User input is concatenated userInput)" "evidence": "fs.readFile(path.join(root, into a shell command.", } req.url))" "evidence": "exec(\"ls \" + name)" } }

Rule-based Program

Testing Oracles

Exploit Exploiter

" nding": { { "vuln_type": "path_traversal", Constraints " nding":{ {" le": "server.js", "line": 87 }, "location": "vuln_type": "os_command_injection", "description": "User-controlled path is { " le": "index.js", "line": 12 }, joined"location": with the base directory without "description": "User input is concatenated validation.", into a shell command.", "evidence": "fs.readFile(path.join(root, "evidence": "exec(\"ls \" + name)" req.url))" }, }, "constraint": "Payload must be a../ valid "constraint": "Payload must include lename and …” }

Judge Judge Judge

Constraints Constraints Inferencer

{

Final Report

" ndings": [ { ... } ], "verdicts": [ { ... } ], "constraints": [ { ... } ], "exploits": [ { " nding": { … } }, "all_runs": [ { "code": "const { exec } = require(…); …”, "exit_code": 0 } ], "success": true, "successful_run": { "code": "const { exec } = require(…); …”, "exit_code": 0 } } ], }

Per- nding validation

LLM Agents

Figure 3: Overview of LLMVD.js.

fi fi fi fi

OpenAI’s GPT-5 series (400K), Gemini-3 series (1M), and Claude Sonnet 4.5 (200K by default, 1M experimental). This does not imply that large or complex packages are unimportant; rather, it reflects the natural size distribution of the npm ecosystem, where most packages are relatively small. As a result, fi fifi LLM-based approaches are well suited for reasoning about a substantial fraction of real-world packages. Even when packages are larger, an LLM agent can iteratively construct task-relevant context through pattern-based search and package-structure understanding, without requiring dedicated static/dynamic analysis engines for taint/path derivation.

4.2

The pipeline accepts either a local project path or an npm identifier in package@version format and operates on a fixed snapshot of the target package. We support four taint-style vulnerability classes that are commonly supported by prior program analysis tools, including command injection, code injection, path traversal, and prototype pollution. Each class is registered with (i) a natural language vulnerability specification, (ii) goal-oriented exploitation criteria, and (iii) a class-specific execution oracle. Success is determined by vulnerability-class–specific side effects. These uniform success predicates enable automated validation across heterogeneous vulnerability types.

LLMVD.js Design and Implementation

In this section, we present the design and implementation details of our proposed multi-stage LLM-based agent framework for detecting and confirming vulnerabilities in Node.js packages.

4.1

Target Resolution and Execution Context

4.3

Multi-Stage Vulnerability Reasoning Pipeline

Candidate Enumeration (Finder). The Finder stage performs hypothesis generation by enumerating candidate vulnerabilities through lightweight codebase exploration, including directory traversal, pattern-based search, and source inspection. Each candidate is summarized as a structured hypothesis consisting of a vulnerability type, precise source location, supporting code evidence, and a set of potentially reachable APIs. This stage intentionally favors over-approximation and prioritizes coverage over precision through prompt design, deferring exploitability assessment and confirmation to subsequent refinement stages. Each candidate is then processed independently through the remainder of the pipeline.

System Overview

Figure 3 illustrates the architecture of LLMVD.js, a multi-stage framework for vulnerability detection and exploit confirmation in Node.js packages. In practice, end-to-end vulnerability confirmation requires reasoning about candidate locations, constructing executable drivers, and validating exploitability with reliable, automated signals. To make this process tractable and auditable, LLMVD.js decomposes the pipeline into a small number of stages with distinct objectives: the initial finding stage prioritizes high recall, aiming to identify as many potentially vulnerable locations as possible, while the subsequent stages focus on precision by validating exploitability and eliminating false positives. Accordingly, LLMVD.js organizes analysis as a staged workflow in which candidate findings are first enumerated, then filtered for exploitability, then augmented with exploitation conditions, and finally validated through execution-based verification. The final stage uses automated execution oracles that determine success based on concrete side effects.

Exploitability Filtering (Judge). The Judge stage filters infeasible hypotheses through focused code inspection and lightweight data-flow reasoning, primarily as a reachability check without solving path constraints. Exported APIs are conservatively treated as externally reachable to avoid prematurely discarding viable attack surfaces. For each candidate, the stage produces a structured verdict consisting of a binary exploitability label and a concise justification. 5

Ronghao Ni, Mihai Christodorescu, and Limin Jia

Only candidate findings deemed potentially exploitable proceed to constraint inference, thereby eliminating false positives early and reducing unnecessary exploration in later stages.

exploit results, enabling stage-level auditing, failure attribution, and systematic analysis of intermediate reasoning behavior.

4.6

Constraint Inference (Constraints Inferencer). Given a validated hypothesis, the Constraints Inferencer stage derives a compact set of actionable exploitation conditions, including likely entry points, required parameters, payload structure, and relevant bypass considerations. These constraints summarize the minimal conditions necessary to propagate attacker-controlled input to the vulnerable sink and serve as an explicit interface between exploitability reasoning and exploit synthesis. By representing exploitation conditions explicitly as structured constraints, this stage provides a clear, structured interface for subsequent exploit synthesis.

Each conceptual stage described above is implemented as a dedicated LLM agent. Each agent operates with an isolated context and is responsible for a specific task within the pipeline. LLM Model Selection. Since our evaluation involves vulnerability detection and confirmation on unrevealed or unpublished vulnerabilities, we use APIs that have a non-training policy to minimize the risk of data leakage and the possibility of further training LLMs on unrevealed vulnerabilities. To balance performance and cost, we select OpenAI’s GPT-5-mini model (gpt-5-mini-2025-08-07) as our LLM backbone. The model is accessed via OpenAI’s API platform.

Execution-Coupled Exploit Synthesis (Exploiter). The Exploiter stage performs execution-coupled synthesis by generating exploits that instantiate the inferred constraints and executing them within the target package environment. Each attempt imports the vulnerable module, executes the payload under Node.js, and records structured results together with full stdout and stderr traces. Exploit attempts are bounded by iteration limits, and all failed attempts are retained for post hoc analysis. Successful executions are immediately validated by the class-specific oracle (discussed in Section 4.4) to provide confirmation of exploitability.

4.4

Agent Framework. We implement our multi-stage agent framework based on LangChain1 , a popular framework for developing LLM-powered applications. LangChain provides modular components for building complex agent workflows. We use a recursion limit of 54 for the agents to prevent infinite or excessive loops during tool usage. Considering the non-determinism of LLMVD.js, we allow up to three attempts when no successful exploit is generated. Here, success is defined as observable side effects detected by automated oracles, but not by manual verification. We adopt a custom multi-stage architecture rather than existing open-source frameworks such as OpenHands [50] and SWE-agent [53], to provide explicit stage separation, enabling finegrained auditing, debugging, and the integration of domain-specific verification components. While these frameworks are effective for general software engineering tasks, they lack native support for stage-level logging and interfaces for vulnerability validation, exploit execution, and oracle-based verification required for our analysis. LLMVD.js enables our study of LLM agent behavior for npm vulnerability detection and exploit generation in a transparent and controllable environment.

Oracle-Guided Execution and Automatic Probing

A central design choice in LLMVD.js is the use of oracle-guided exploit confirmation to avoid reliance on model self-reporting and manual inspection. For each vulnerability class, we define an execution oracle that evaluates concrete side effects produced by exploit attempts. The execution harness augments each payload with vulnerability-specific probing logic. For example, code injection exploits must trigger a predefined marker function; prototype pollution exploits are validated by probing polluted object properties; and path traversal exploits must read and print a prepared sentinel file. For OS command injection, sentinel artifacts are removed before each attempt to prevent contamination across runs. These class-specific probes provide uniform, automated success predicates and enable execution-driven refinement of exploit hypotheses without explicit symbolic constraint solving.

4.5

Implementation

Prompt Design. We create custom prompts for each agent stage to guide the LLM’s reasoning and tool usage. The prompts include clear instructions, the exploitation goal, and an output structure formatted in JSON that contains the expected information for each stage. We refine the prompts iteratively based on initial experiments to improve agent performance, while ensuring no information leakage related to the evaluated packages. Full prompt templates are provided in Appendix C.

Tooling and Coordination Infrastructure

We organize the supporting toolset into three categories aligned with the stages of vulnerability reasoning. First, exploration and inspection tools enable rapid understanding of package structure and relevant logic through directory navigation, source reading, and pattern-based search. These tools are shared across all reasoning stages. Second, execution and environment interaction tools are restricted to the exploit synthesis stage and provide controlled Node.js and shell execution, vulnerability-specific harness integration, auxiliary side-effect checks, and optional background process management for long-running services such as web server applications. Finally, structured reporting and coordination utilities enforce typed submissions for hypotheses, verdicts, constraints, and

Evaluation

We evaluate LLMVD.js on a variety of datasets to answer the following research questions: • RQ1: How effective is LLMVD.js in detecting and confirming vulnerabilities in Node.js packages? • RQ2: What is the cost of using LLMVD.js? • RQ3: What are the limitations and failure modes of LLMVD.js, and how can they inform future improvements? 1 https://python.langchain.com/en/latest/index.html

Taint-Style Vulnerability Detection and Confirmation for Node.js Packages Using LLM Agent Reasoning

Table 2: Overview of the VulcaN, SecBench.js, NodeMedic, and the newly crawled Wild datasets. “Raw” refers to the initial number of vulnerable instances per dataset as used in previous work (VulcaN and SecBench.js) or as received (NodeMedic and Wild). “Valid” denotes packages still available on the npm registry at collection time. “Used” is the sampled subset. “Total” is the count of valid packages and sampled packages. “Dist. (%)” indicates the distribution of each vulnerability type across the combined datasets.

VulcaN Vulnerability Type Path Traversal Command Injection Code Injection Prototype Pollution

CWE-22 CWE-78 CWE-94 CWE-1321

Total

5.1

SecBench.js

NodeMedic

Wild

CWE

Total

Dist. (%)

65 65 65 65

224 331 236 245

21.62% 31.95% 22.78% 23.65%

260

1,036

100.00%

Raw

Valid

Raw

Valid

Raw

Used

Raw

Used

5 66 22 67

3 58 21 62

161 82 21 120

156 78 21 118

0 1,022 228 0

0 130 129 0

91 1,004 452 104

160

144

384

373

1,250

259

1,651

the issues of LLMs memorizing codes and exploits instead of engaging in genuine reasoning. In this work, we refer to this dataset as NodeMedic.

Experiment Setup

5.1.1 Datasets. Our evaluation utilizes four types of datasets: 1) public benchmarks commonly used in vulnerability detection research, 2) a private dataset of real-world Node.js packages with known vulnerable data paths but no associated CVEs or public exploits, 3) a transformed dataset derived from public benchmarks to assess generalizability, and 4) a crawled dataset of recently released Node.js packages from the npm registry to further evaluate performance on unseen data.

Transformed dataset. Since the public benchmarks used in this work were released before the knowledge cutoff date of the LLM we chose (GPT-5-mini: May 31, 2024), there is a risk that memorization in the LLM affects the performance and thus it is crucial to evaluate how well our framework generalizes to unseen data. To achieve this, we create a transformed dataset by selecting up to 20 vulnerable instances per CWE from each public dataset, and applying code transformations such as renaming variables and functions, removing comments, and changing formatting. Package names, versions, and links in manifest files are also anonymized. The aim is to generate code that maintains the original semantics and vulnerabilities while being sufficiently different from the LLMs’ training data, thus reducing the chances of memorization. For transforming JavaScript code, we use terser [3], a toolkit for mangling and compressing JavaScript.

Public benchmarks. Following common practice, we use two widely recognized public benchmarks: VulcaN [7] and SecBench.js [6]. These datasets collected vulnerable Node.js packages from the npm registry based on reports from GitHub Advisory, Snyk, Huntr.dev, and the CVE database. For a fair comparison with past work, we select four vulnerability types that have been studied previously: path traversal (CWE-22), OS command injection (CWE-78), code injection (CWE-94), and prototype pollution (CWE-1321). We use the same set of packages evaluated in the most recent prior work (Explode.js [35]), excluding packages that are no longer available on the npm registry.

Crawled dataset. To further evaluate our framework on unseen data, we crawled recently released Node.js packages from the npm registry that were published in December 2025. We consider only newly released or updated packages and collected 17,151 packages during this period. We designed regular expressions to identify potential vulnerable code patterns. The full regex set is provided in Appendix A. For example, we use \b(?:eval|Function)\s*\( to identify potential code-injection vulnerabilities. Considering cost and time constraints, we randomly sample 65 packages per vulnerability type that were flagged by the regex patterns. To ensure diversity, for each vulnerability class we discretize three structural metrics (code size, dependency count, and number of files) into coarse buckets and stratify packages by the resulting bucket combinations. When applicable, we additionally ensure that both minified and non-minified artifacts are represented in the sample. The detailed stratified sampling procedure is described in Appendix B. Table 2 provides an overview of the datasets used in our evaluation, including vulnerability counts from VulcaN, SecBench.js, and NodeMedic.

Private dataset. We obtained access to the NodeMedic-FINE private dataset [9], which contains real-world Node.js packages with vulnerable data paths but without associated CVEs or public exploits. Although all packages contain vulnerable code paths, no CVEs or other security advisories have been assigned for reasons like low download count or the developers’ assumption that the parent package should sanitize the input before calling the vulnerable API. This dataset can only be used to evaluate our framework on code injection and OS command injection vulnerabilities, the only types supported by NodeMedic-FINE. Considering cost and time constraints, we randomly sample 129 packages with code-injection vulnerabilities and 130 with command-injection vulnerabilities for evaluation. These sample sizes were chosen based on the average number of vulnerable instances per vulnerability type in the VulcaN and SecBench.js datasets after filtering. This dataset is especially useful for assessing our framework’s capability to identify vulnerabilities that are not publicly documented, thus creating a more realistic scenario for vulnerability detection, particularly regarding 7

Ronghao Ni, Mihai Christodorescu, and Limin Jia

Table 3: Performance comparison of LLMVD.js against state-of-the-art tools on standard benchmarks. “NM-FINE” = NodeMedicFINE. “Det.” = detected, “Expl.” = exploited, “Val.” = valid. Explode.js is evaluated in two modes: “File” and “Pkg”. The total number of packages and the number of exploits for each tool, both by dataset and overall, are in bold for easier comparison. FAST Dataset

Vulnerability Type

NM-FINE

Explode.js File

SecBench.js

VulcaN

LLMVD.js

Total Pkg

Det.

Expl.

Det.

Expl.

Det.

Expl.

Det.

Expl.

Det.

Expl.

Val.

Path Traversal Command Injection Code Injection Prototype Pollution

156 78 21 118

105 65 8 0

6 60 2 0

37 5 -

31 1 -

88 56 7 53

79 41 4 48

51 1 3 4

49 1 1 2

155 77 20 113

149 76 18 89

Total

373

178

204

172

365

332

Path Traversal Command Injection Code Injection Prototype Pollution

3 58 21 62

1 46 13 0

0 38 5 0

15 4 -

9 1 -

2 31 10 33

1 16 3 31

1 10 3 6

1 3 0 4

3 54 14 58

3 53 14 55

3 48 12 38

Total

144

129

125

101

517

238

111

280

223

494

490

433

Overall Total

5.1.2 Baseline tools. We compare LLMVD.js with three state-ofthe-art program-analysis tools for Node.js package vulnerability detection: 1) NodeMedic-FINE [9], 2) Explode.js [35], and 3) FAST [28]. Since our work is the first to utilize LLMs for the complete pipeline of vulnerability detection and confirmation (including PoC generation) in Node.js packages, we include one LLM-based baseline that is not directly comparable to our method: PoCGen [47]. PoCGen does not perform vulnerability detection; it generates PoCs based on existing CVE reports by integrating program analysis techniques with LLM reasoning. We aim to assess whether this combination is necessary or if an LLM-centric, tool-augmented workflow can effectively detect and confirm vulnerabilities. We configure PoCGen to use the same backend LLM model (GPT-5-mini) as LLMVD.js to ensure a fair comparison. In addition, we run PoCGen on each package up to three times upon failure, matching the maximum number of attempts allowed for LLMVD.js.

FP2

FP3

FP4

FP5

5.1.3 PoC Validation. Even though both LLMVD.js and the baseline method PoCGen include mechanisms to validate generated PoCs and eliminate false positives, situations may still arise in which the generated PoCs are invalid. In particular, LLM-generated PoCs may trigger the desired side effects while failing to truly exploit the intended vulnerability. To address this issue, we apply an additional manual validation step to all PoCs generated by both LLMVD.js and PoCGen in order to ensure their accuracy. Concretely, among the PoCs that pass automated validation, we further filter out those that exhibit the following behaviors (which we mark as false positives or FPs):

5.2

certain dependencies, and the LLM attempts to emulate such environments by replacing key built-in functions with stubs. The PoC assumes that certain files with specific names exist, where the file names either match the payload or contain the payload. Or, the PoC modifies the environment variables. This is too strong of an assumption to make in practice. This occurs when the package checks for the existence of certain files to decide whether to execute specific code paths. The PoC does not use any public APIs of the package and instead relies on internal code paths, dependencies, test code, or example scripts. For prototype pollution vulnerabilities, the PoC directly uses Object or Object.prototype as part of the arguments passed to the vulnerable package APIs, which is unrealistic in practice because an attacker typically cannot supply these built-in objects as inputs. External tools like web browsers are necessary to trigger the vulnerability. The PoC may simulate this process by directly calling internal functions that these external tools would typically invoke. For a rigorous evaluation of our framework, we classify such PoCs as invalid.

RQ1: Effectiveness

5.2.1 Comparison with Traditional Program Analysis Tools. Table 3 shows the performance comparison of LLMVD.js against three leading program-analysis tools (NodeMedic-FINE, FAST, and Explode.js) on the SecBench.js and VulcaN datasets. It’s important to note that Explode.js operates in two modes: "File" mode, where the tool analyzes a specific file, and "Pkg" mode, where the tool examines the entire package without prior knowledge of which file contains the vulnerability. We include both modes in our comparison because "File" mode is used in the original Explode.js paper [35], while "Pkg" mode is more realistic for comprehensive vulnerability detection.

FP1 The PoC introduces a new vulnerability to the runtime environment instead of exploiting an existing one, such as redefining a built-in function to create specific side effects. We mainly observe this behavior in packages with vulnerabilities that exist only on certain operating systems or relies on 8

Taint-Style Vulnerability Detection and Confirmation for Node.js Packages Using LLM Agent Reasoning

3 1 0 0 1 3

NodeMedic-FINE Explode.js

NodeMedic-FINE 3

(b) Command Injection

Explode.js LLMVD.js

LLMVD.js

(a) Path Traversal

Explode.js LLMVD.js

Explode.js

(d) Prototype Pollution

Figure 4: Venn diagram overlaps for vulnerability types. Table 4: Performance comparison of LLMVD.js against PoCGen on 299 overlapping SecBench.js packages. “Expl.” = exploited, “Val.” = valid exploit, “Avg Cost” = average LLM API cost per package. PoCGen Vulnerability Type

LLMVD.js

Total Expl.

Val.

Avg Cost

Expl.

Val.

Avg Cost

Path Traversal Command Injection Code Injection Prototype Pollution

117 67 12 103

112 63 10 93

108 62 9 74

$0.089 $0.124 $0.172 $0.079

116 67 12 99

112 66 11 79

$0.050 $0.068 $0.099 $0.135

Total

299

278

253

$0.097

294

268

$0.085

LLMVD.js demonstrates significant advantages over all baselines in both datasets across all types of vulnerabilities. Notably, LLMVD.js successfully detects and generates valid exploits for 433 out of 517 vulnerable packages in both datasets, achieving an overall confirmation rate of 83.75%. In contrast, the best-performing baseline, which is not realistic in an end-to-end detection and confirmation setting, Explode.js in "File" mode, manages to confirm only 223 vulnerabilities, resulting in a confirmation rate of 43.13%. Examining the distribution of successful exploit generations by various vulnerability types, Figure 4 shows that while LLMVD.js significantly outperforms the baselines, there are still packages where the baselines succeed while LLMVD.js does not. This suggests that rule-based tools can offer complementary strengths in specific situations, which we discuss further in Section 5.4.

the time of our evaluation, as PoCGen cannot initiate the pipeline without the report. This results in a total of 299 packages. Table 4 presents the comparison results. LLMVD.js outperforms PoCGen in terms of valid exploit generation across all vulnerability types and incurs lower LLM API costs in three of the four vulnerability types and achieves lower overall cost. We investigate why the LLM-program analysis hybrid design in PoCGen does not result in better performance or, at the very least, lower costs. We summarize our observations in the following points: (1) CodeQL AST/locations weren’t converted to compact facts, so the LLM still tried to resolve references itself. (2) LLMs can infer taint flows and definitions directly from raw code, so verbose CodeQL snippets offered little additional, non-redundant signal. (3) when a refinement fails, the refiner generates multiple slightly different prompt variants and sends each to the model, so one failure becomes many near-duplicate LLM requests, increasing API calls and token usage; (4) single prompts often embed overlapping sections (examples, descriptions, snippets) that inflate tokens per call beyond a concise agent prompt. These emphasize the need to better design program analysis components that can effectively assist LLM in reasoning for this task. We discuss potential future directions based on the failure modes of LLMVD.js that we observed in Section 5.4.

5.2.2 Comparison with PoCGen. As a comparison with a prior LLM–program analysis hybrid method, we investigate whether a carefully designed pipeline with program-analysis components can still improve LLM reasoning in terms of both performance and cost, given the rapid advancement of LLMs. This comparison with PoCGen [47] is favorable to PoCGen. PoCGen is provided with a CVE report when generating exploits, whereas LLMVD.js operates as an end-to-end framework that detects and confirms vulnerabilities without being provided any prior knowledge of the target packages beyond their source code. Nevertheless, PoCGen is the closest available LLM-based work that targets a partially overlapping problem setting. For a fair comparison, we only evaluate the overlapping packages from the PoCGen [47] and our SecBench.js [6] datasets. We also removed any packages whose vulnerability reports were deleted at

5.2.3 Transformed dataset. We compared the number of packages that LLMVD.js could successfully exploit before and after transformation on the two sampled public benchmarks (VulcaN and SecBench.js), counting only exploits with manually verified valid PoCs. Among the 143 sampled packages, LLMVD.js successfully 9

Ronghao Ni, Mihai Christodorescu, and Limin Jia

Table 5: Performance comparison of LLMVD.js against state-of-the-art tools on recently released Node.js packages from the npm registry and the private NodeMedic dataset. “NM-FINE” = NodeMedic-FINE, “Det.” = detected, “Expl.” = exploited, “Val.” = valid. “-” indicates that reporting is not applicable for the NodeMedic dataset. The total number of packages and the number of exploits for each tool, both by dataset and overall, are in bold for easier comparison. FAST Dataset

NodeMedic

Wild

Vulnerability Type

NM-FINE

Explode.js

LLMVD.js

Total Det.

Expl.

Det.

Expl.

Det.

Expl.

Det.

Expl.

Val.

Command Injection Code Injection

130 129

84 78

78 29

130 129

71 56

10 22

5 9

129 128

128 128

120 124

Total

259

162

107

259

127

257

256

244

Path Traversal Command Injection Code Injection Prototype Pollution

65 65 65 65

0 3 1 0

0 2 0 0

0 0 -

0 0 0 0

44 28 20 20

37 26 17 4

6 17 12 1

Total

260

112

519

166

109

259

127

369

340

280

Overall Total

generated valid PoCs for 108 packages on the original (untransformed) datasets. On the transformed dataset, LLMVD.js generated valid PoCs for 107 packages, missing one previously successful package ([email protected]) and yielding no additional successful exploits. A detailed case study of this package is provided in Section 5.4. This result indicates that LLMVD.js generalizes well to unseen data that are syntactically different from the LLM training data, demonstrating robustness against potential memorization effects.

We analyze the non-validated cases by categorizing them into two groups: (1) environment-dependent exploits that require specific system configurations or dependencies, where the LLM attempts to emulate the environment by introducing proxies (e.g., stubs or redefined built-ins); and (2) executions through internal code paths such as tests or examples that are not part of the public API. Our manual validation adopts a conservative policy that excludes these categories, which likely underestimates the number of true vulnerabilities. We are currently evaluating cases where the exploits were not deemed valid to determine whether they correspond to real vulnerabilities. These results highlight a limitation of current LLM-based agents: while they can synthesize executable exploits, they often rely on overly permissive assumptions about entry points and execution conditions. Without explicit guidance, the agent may use internal code paths instead of public APIs or emulate missing environments beyond the intended exploit boundary. These issues point to the need for prompt tuning to clearly define the detection boundary, including public APIs and environment and use-case assumptions. We have reported all 36 validated vulnerabilities to the respective package maintainers and are in the process of responsible disclosure. So far, we have received acknowledgments from 3 maintainers.

5.2.4 Private NodeMedic dataset. Table 5 shows a comparison of LLMVD.js with state-of-the-art tools on the private NodeMedic dataset [9] (Section 5.1.1). LLMVD.js generated valid PoCs for 244 out of a total of 259 vulnerable packages, achieving a valid PoC generation rate of 94.2%. In contrast, NodeMedic-FINE produced working PoCs for only 127 (49.0%). Despite the significant performance gap, LLMVD.js missed 5 vulnerable packages (3 Code Injection and 2 Command Injection) that were successfully exploited by NodeMedic-FINE (more in Section 5.4.3).

5.2.5 Crawled npm packages in the wild. This dataset includes randomly sampled 65 packages per vulnerability type (path traversal, OS command injection, code injection, and prototype pollution) that were flagged by our regex patterns, resulting in a total of 260 packages (Section 5.1.1). Among the rule-based program analysis tools, only FAST detected 4 vulnerable packages (3 command injection and 1 code injection) and successfully exploited 2 of them (2 command injection). NodeMedic-FINE and Explode.js did not detect any vulnerabilities in these packages. In contrast, LLMVD.js detected 112 packages that it identified as potentially vulnerable and successfully exploited 84 of them to produce the required side effects. Among the 84 exploited packages, 36 generated proofs of concept (PoCs) that were deemed valid after manual inspection. The detailed results are presented in Table 5.

Table 6: Summary of file-level unmatched rates by dataset (all vulnerability types combined) for Finder and Judge stages. Metric Finder Findings Unmatched Judge Findings Unmatched Finder GT Unmatched Judge GT Unmatched

SecBench.js 22.94% 20.89% 4.29% 4.83%

VulcaN 28.80% 29.15% 10.60% 14.57%

All 24.77% 23.32% 6.11% 7.63%

5.2.6 Fine-Grained Analysis of Detection Results. LLMVD.js may report multiple findings per package, and some reports do not match any benchmark-labeled vulnerable file. Because benchmark 10

0.5

Exploited Not exploited Smoothed median

0.3

Exploited Not exploited

10 2

0.2

LLM Cost (USD)

10 1

(b) Exploit success vs. LLM cost

0.1 0.0 102

103

104

Package Tokens

105

Not exploited Exploited

LLM Cost (USD)

0.4

Not exploited Exploited

Taint-Style Vulnerability Detection and Confirmation for Node.js Packages Using LLM Agent Reasoning

Exploited Not exploited 102

103

104

Package Tokens

105

(a) LLM cost vs. package size

Figure 5: Cost, package size, and exploit success trade-offs. (a) LLM API cost increases with package token count, with the red curve showing the smoothed median. 16 packages with extreme large token counts are omitted for better visualization. (b) Distribution of exploited and non-exploited cases across different LLM cost levels. (c) Distribution of exploited and nonexploited cases across package token counts. labels are not guaranteed to be exhaustive, we do not automatically treat unmatched reports as false positives. Instead, we quantify mismatch behavior with file-level coverage metrics. Here, “file-level” means the benchmark ground truth includes the file location for each vulnerability, and LLMVD.js output includes a predicted vulnerable file path for each finding; we count a match only when these file locations agree. At a high level, we measure mismatch from two perspectives at both Finder and Judge stages: (i) report-side mismatch (captured by the Finder/Judge Findings Unmatched rows in Table 6), i.e., what fraction of tool-reported findings cannot be matched to benchmark-labeled vulnerable files, and (ii) ground-truth-side miss rate (captured by the Finder/Judge GT Unmatched rows in Table 6), i.e., what fraction of benchmark vulnerable files are not covered by any reported finding. These metrics are formally defined in Appendix D. We report these four percentages for each dataset in Table 6. The detailed file-level unmatched results by dataset and vulnerability type are presented in Table 8 in Appendix D. Overall, the file-level mismatch rates indicate a favorable tradeoff between over-reporting and missed detections. While LLMVD.js produces a non-trivial fraction of unmatched reports (approximately one quarter), the ground-truth miss rate remains low (around 6–8%), suggesting that most benchmark-labeled vulnerabilities are successfully covered. We emphasize that unmatched findings arise from two sources: incomplete benchmark labeling and the tool’s over-approximation of potential vulnerabilities.

5.3

RQ2: Cost

We compute the average LLM API cost incurred by LLMVD.js across different vulnerability types. The average cost per package is $0.051 for path traversal, $0.068 for OS command injection, $0.107 for code injection, and $0.136 for prototype pollution, with an overall average cost of $0.089 per package across all samples. When restricting to successfully exploited packages, LLMVD.js spends an average of $0.084 per valid exploit. Because LLMVD.js may generate multiple exploit candidates for a single package by targeting different vulnerable locations, we additionally report an amortized cost per valid exploit of $0.050, computed by dividing the total LLM cost by the number of valid exploits. These results indicate that LLMVD.js achieves effective exploit generation with modest and well-controlled LLM usage costs across vulnerability types. We further analyze the relationship between LLM cost and exploit success using the per-package distributions. Figure 5a shows that LLM cost increases with package token count, and that successful exploits are observed across a wide range of token usage levels. Figure 5b relates exploit outcomes to the incurred LLM cost and shows no clear monotonic relationship between exploit success and LLM cost, as both exploited and non-exploited packages are distributed across similar cost ranges.

5.4

RQ3: Limitations and Failure Modes

5.4.1 Impact of Package Size. A common thought is that larger packages are harder to exploit because they tend to involve longer and more complex code paths, which can make it more difficult to 11

Ronghao Ni, Mihai Christodorescu, and Limin Jia

1 var baseSet = require('./_baseSet'); 2 3 /** 4 * This method is like `_.set` except that it accepts `c . ustomizer` which is 5 * invoked to produce the objects of `path`. If `custom . izer` returns `undefined` 6 * path creation is handled by the method instead. The ` . customizer` is invoked 7 * with three arguments: (nsValue, key, nsObject). 8 * 9 * **Note:** This method mutates `object`. 10 ... 11 * _.setWith(object, '[0][1]', 'a', Object); 12 * // => { '0': { '1': 'a' } } 13 */ 14 function setWith(object, path, value, customizer) { 15 customizer = typeof customizer == 'function' ? customi .. zer : undefined; 16 return object == null ? object : baseSet(object, path, .. value, customizer); 17 } 18 19 module.exports = setWith;

1 var baseSet = require("./_baseSet"); 2 . . . . . . . . . . . . . . 3 function setWith(e, t, i, n) { . . 4 return n = "function" == typeof n ? n : void 0, null = . = e ? e : baseSet(e, t, i, n); 5 } 6 7 module.exports = setWith;

Figure 6: One example of the vulnerable sinks in [email protected] before (left) and after (right) transformation. In “Before transformation”, some parts of the comments were omitted for brevity and replaced with “...”. identify the relevant data flows and construct a working exploit. Figure 5(c) examines this relationship by plotting exploit outcomes against package token count. The plot shows that successful exploits are common for small and medium-sized packages, while the proportion of non-exploited cases increases as package size grows.

leading to substantially degraded vulnerability detection and confirmation. A systematic characterization of which transformations cause these failures, and how to mitigate them, is beyond the scope of this paper, and we leave detailed studies to future work. 5.4.3 Vulnerabilities Missed by LLMVD.js but Detected by Rulebased Tools. Most of the vulnerable packages that LLMVD.js overlooked but rule-based program analysis tools successfully exploited are due to manual validations. The LLM believes it has generated a valid PoC and exits, but the PoCs are manually rejected. We discuss this in Section 5.4.4. Ruling out this part, there are 8 packages in all of SecBench.js, VulcaN and NodeMedic datasets that LLMVD.js missed but traditional tools successfully exploited. We manually inspected these packages and found that they mainly fall into two categories: (1) The tool identified the location of the vulnerability but concluded that it had sufficient sanitization or had already been patched, so it did not report it. (2) The tool loaded a large but irrelevant portion of the codebase into the context, which caused confusion and led to missing the vulnerability. Although these issues occur in only 4 out of the 776 evaluated packages, they highlight opportunities to further improve the current LLMVD.js pipeline.

5.4.2 A case study of missed generalization. To better understand the generalization capabilities of LLMVD.js, we investigate the single package that was successfully exploited on the original dataset but missed on the transformed dataset: [email protected], which contains prototype-pollution vulnerabilities. In detecting the transformed version, LLMVD.js did not identify any potential vulnerabilities and therefore stopped at the finder stage. Figure 6 shows a side-by-side comparison of one of the many vulnerable sinks in both the original and transformed code. The transformation included renaming variables and functions, removing comments, and altering the formatting. In the original package, LLMVD.js reported eight vulnerability findings and successfully identified prototype pollution across multiple relevant files (e.g., _baseAssignValue.js, _baseSet.js, and _baseMerge.js). The agent’s reasoning benefited from semantic cues such as informative variable names and developer comments, which helped it interpret code intent and localize vulnerable behaviors. In contrast, on the transformed package, LLMVD.js produced zero findings and hit the recursion limit (54 iterations). The logs indicate that the agent spent most of its budget attempting to navigate and interpret the obfuscated codebase, becoming effectively lost among approximately 1,046 JS files without meaningful semantic hints. For example, it repeatedly revisited file-tree listings and performed pattern searches, but failed to form a coherent understanding of the code necessary to confirm vulnerabilities. This case study suggests that certain obfuscation and anonymization patterns can effectively blind LLM-based agents by removing the semantic cues they rely on for navigation and comprehension,

5.4.4 Invalid PoC Generations and False Positives. Table 7 summarizes why some PoCs that pass automated validation are still deemed invalid after manual inspection. Prototype pollution contributes the largest number of invalid cases (41 packages), and the dominant failure mode is FP4, where the PoC unrealistically passes Object or Object.prototype as an input to the target API (34/41, 82.9%). For OS command injection, most invalid PoCs fall under FP1 (11/14, 78.6%), where the agent emulates missing environments or dependencies by redefining built-ins or stubbing key functionality, and a small fraction require external tools (FP5). For code injection, invalid cases are split between environment emulation (FP1: 5/8, 62.5%) and reliance on non-public/internal code paths (FP3: 3/8, 12

Taint-Style Vulnerability Detection and Confirmation for Node.js Packages Using LLM Agent Reasoning

Table 7: Analysis of invalid exploit reasons by vulnerability type. For each vulnerability type, shows the total number of packages with invalid exploits and breakdown by reason with percentages. Invalid Reason

Path Traversal

Command Injection

Code Injection

Prototype Pollution

Total Packages (Invalid)

FP1: Emulated Environment FP2: Strong Assumptions FP3: Non-Public API FP4: Direct Object Use FP5: External Tools Needed

1 (16.7%) 5 (83.3%) -

11 (78.6%) 1 (7.1%) 1 (7.1%) 1 (7.1%)

5 (62.5%) 3 (37.5%) -

4 (9.8%) 1 (2.4%) 2 (4.9%) 34 (82.9%) -

37.5%). For path traversal, the primary issue is FP3 (5/6, 83.3%), indicating that the agent often constructs PoCs that exercise internal code paths, tests, or examples rather than public-facing APIs. Overall, these invalid generations concentrate in a few recurring, vulnerability-specific failure modes, which motivates future work on adding rule-based checks or LLM-based guardrails to discourage unrealistic assumptions and improve PoC validity. Overall, the evaluations do not suggest that LLMs universally outperform classical tools, but instead motivates rethinking the role of program analysis as a complementary technique, particularly in scenarios where formal guarantees or deep semantic reasoning are required.

lower-level perspectives and to carefully design program analysis tools that complement LLMs in ways that cannot be easily achieved through model scaling or architectural improvements alone.

References [1] [n. d.]. Babel: The JavaScript Compiler. https://babeljs.io. [2] [n. d.]. Esprima: ECMAScript Parsing Infrastructure for Multipurpose Analysis. https://esprima.org/. [3] [n. d.]. Terser: JavaScript mangler and compressor toolkit. https://terser.org/. [4] Mir Masood Ali, Mohammad Ghasemisharif, Chris Kanich, and Jason Polakis. 2024. Rise of inspectron: Automated black-box auditing of cross-platform electron apps. In 33rd USENIX Security Symposium (USENIX Security 24). 775–792. [5] Thanassis Avgerinos, Sang Kil Cha, Alexandre Rebert, Edward J Schwartz, Maverick Woo, and David Brumley. 2014. Automatic exploit generation. Commun. ACM 57, 2 (2014), 74–84. [6] Masudul Hasan Masud Bhuiyan, Adithya Srinivas Parthasarathy, Nikos Vasilakis, Michael Pradel, and Cristian-Alexandru Staicu. 2023. SecBench. js: An executable security benchmark suite for server-side JavaScript. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 1059–1070. [7] Tiago Brito, Mafalda Ferreira, Miguel Monteiro, Pedro Lopes, Miguel Barros, José Fragoso Santos, and Nuno Santos. 2023. Study of javascript static analysis tools for vulnerability detection in node. js packages. IEEE Transactions on Reliability 72, 4 (2023), 1324–1339. [8] Tiago Brito, Mafalda Ferreira, Miguel Monteiro, Pedro Lopes, Miguel Barros, José Fragoso Santos, and Nuno Santos. 2023. Study of javascript static analysis tools for vulnerability detection in node. js packages. IEEE Transactions on Reliability 72, 4 (2023), 1324–1339. [9] Darion Cassel, Nuno Sabino, Min-Chien Hsu, Ruben Martins, and Limin Jia. 2025. NODEMEDIC-FINE: Automatic Detection and Exploit Synthesis for Node. js Vulnerabilities. In Proceedings of the 2025 Network and Distributed System Security Symposium (NDSS’25). doi, Vol. 10. [10] Darion Cassel, Wai Tuck Wong, and Limin Jia. 2023. Nodemedic: End-to-end analysis of node. js vulnerabilities with provenance graphs. In 2023 IEEE 8th European Symposium on Security and Privacy (EuroS&P). IEEE, 1101–1127. [11] Mark Chen. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021). [12] Brian Chess and Gary McGraw. 2004. Static analysis for security. IEEE security & privacy 2, 6 (2004), 76–79. [13] Edmund Clarke, Orna Grumberg, Somesh Jha, Yuan Lu, and Helmut Veith. 2003. Counterexample-guided abstraction refinement for symbolic model checking. Journal of the ACM (JACM) 50, 5 (2003), 752–794. [14] Leonardo De Moura and Nikolaj Bjørner. 2008. Z3: An efficient SMT solver. In International conference on Tools and Algorithms for the Construction and Analysis of Systems. Springer, 337–340. [15] Alexandre Decan, Tom Mens, and Eleni Constantinou. 2018. On the impact of security vulnerabilities in the npm package dependency network. In Proceedings of the 15th international conference on mining software repositories. 181–191. [16] Gelei Deng, Yi Liu, Víctor Mayoral-Vilches, Peng Liu, Yuekang Li, Yuan Xu, Tianwei Zhang, Yang Liu, Martin Pinzger, and Stefan Rass. 2024. { PentestGPT } : Evaluating and harnessing large language models for automated penetration testing. In 33rd USENIX Security Symposium (USENIX Security 24). 847–864. [17] Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M Zhang. 2023. Large language models for software engineering: Survey and open problems. In 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE). IEEE, 31–53. [18] Zhiyu Fan, Xiang Gao, Martin Mirchev, Abhik Roychoudhury, and Shin Hwei Tan. 2023. Automated repair of programs from large language models. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE,

Threats to Validity

Even though we tried to design an evaluation to exclude the memorization effect of LLMs as much as possible, there are still some threats to validity that may affect the conclusions drawn from our experiments. First, the code transformations we applied to create the transformed dataset may not be sufficient to completely eliminate the memorization effect, especially for larger models that may have seen similar code snippets during training. Future work could explore more sophisticated transformation techniques or use entirely synthetic datasets to further mitigate this threat. Second, the NodeMedic dataset has distribution bias compared with wild Node.js packages since they only include those that NodeMedicFINE [9] flagged as potentially vulnerable. Therefore, the performance of our framework on this dataset may not fully reflect its effectiveness in real-world scenarios. Third, our evaluation focuses on specific vulnerability types (i.e., path traversal, OS command injection, code injection, and prototype pollution), which may limit the generalizability of our findings to other types of vulnerabilities. Future work could extend the evaluation to a broader range of vulnerability types to assess the versatility of our framework.

Conclusion

In this work, we demonstrate the strong capabilities of state-ofthe-art LLMs in detecting and confirming vulnerabilities in Node.js packages using LLM-centric, tool-augmented reasoning without dedicated static/dynamic analysis engines for taint/path derivation. As LLMs continue to improve in reasoning and code understanding, the benefits of tightly integrating traditional program analysis techniques, as explored in prior work, may quickly diminish. We believe it is therefore promising to evaluate current LLM capabilities from 13

Ronghao Ni, Mihai Christodorescu, and Limin Jia

1469–1481. [19] Chongzhou Fang, Ning Miao, Shaurya Srivastav, Jialin Liu, Ruoyu Zhang, Ruijie Fang, Ryan Tsang, Najmeh Nazari, Han Wang, Houman Homayoun, et al. 2024. Large language models for code analysis: Do { LLMs } really do their job?. In 33rd USENIX Security Symposium (USENIX Security 24). 829–846. [20] Richard Fang, Rohan Bindu, Akul Gupta, and Daniel Kang. 2024. Llm agents can autonomously exploit one-day vulnerabilities. arXiv preprint arXiv:2404.08144 (2024). [21] Mafalda Ferreira, Miguel Monteiro, Tiago Brito, Miguel E Coimbra, Nuno Santos, Limin Jia, and José Fragoso Santos. 2024. Efficient static vulnerability analysis for javascript with multiversion dependency graphs. Proceedings of the ACM on Programming Languages 8, PLDI (2024), 417–441. [22] Tarek Gasmi, Ramzi Guesmi, Ines Belhadj, and Jihene Bennaceur. 2025. Bridging ai and software security: A comparative vulnerability assessment of llm agent deployment paradigms. arXiv preprint arXiv:2507.06323 (2025). [23] Zhiyong Guo, Mingqing Kang, VN Venkatakrishnan, Rigel Gjomemo, and Yinzhi Cao. 2024. ReactAppScan: Mining React Application Vulnerabilities via Component Graph. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security. 585–599. [24] Md Abdul Hannan, Ronghao Ni, Chi Zhang, Limin Jia, Ravi Mangal, and Corina S Pasareanu. 2025. On Selecting Few-Shot Examples for LLM-based Code Vulnerability Detection. arXiv preprint arXiv:2510.27675 (2025). [25] Julius Henke. 2025. AutoPentest: Enhancing Vulnerability Management With Autonomous LLM Agents. arXiv preprint arXiv:2505.10321 (2025). [26] Hamed Jelodar, Samita Bai, Parisa Hamedi, Hesamodin Mohammadian, Roozbeh Razavi-Far, and Ali Ghorbani. 2025. Large Language Model (LLM) for Software Security: Code Analysis, Malware Analysis, Reverse Engineering. arXiv preprint arXiv:2504.07137 (2025). [27] Zihao Jin, Shuo Chen, Yang Chen, Haixin Duan, Jianjun Chen, and Jianping Wu. 2023. A Security Study about Electron Applications and a Programming Methodology to Tame DOM Functionalities.. In NDSS. [28] Mingqing Kang, Yichao Xu, Song Li, Rigel Gjomemo, Jianwei Hou, VN Venkatakrishnan, and Yinzhi Cao. 2023. Scaling javascript abstract interpretation to detect and exploit node. js taint-style vulnerability. In 2023 IEEE Symposium on Security and Privacy (SP). IEEE, 1059–1076. [29] Hee Yeon Kim, Ji Hoon Kim, Ho Kyun Oh, Beom Jin Lee, Si Woo Mun, Jeong Hoon Shin, and Kyounggon Kim. 2022. DAPP: automatic detection and analysis of prototype pollution vulnerability in Node. js modules. International Journal of Information Security 21, 1 (2022), 1–23. [30] Raula Gaikovina Kula, Daniel M German, Ali Ouni, Takashi Ishio, and Katsuro Inoue. 2018. Do developers update their library dependencies? An empirical study on the impact of security advisories on library migration. Empirical Software Engineering 23, 1 (2018), 384–417. [31] Carl E Landwehr, Alan R Bull, John P McDermott, and William S Choi. 1994. A taxonomy of computer program security flaws. ACM Computing Surveys (CSUR) 26, 3 (1994), 211–254. [32] Tan Khang Le, Saba Alimadadi, and Steven Y Ko. 2024. A study of vulnerability repair in javascript programs with large language models. In Companion Proceedings of the ACM Web Conference 2024. 666–669. [33] Xinghang Li, Jingzhe Ding, Chao Peng, Bing Zhao, Xiang Gao, Hongwan Gao, and Xinchen Gu. 2025. SafeGenBench: A Benchmark Framework for Security Vulnerability Detection in LLM-Generated Code. arXiv preprint arXiv:2506.05692 (2025). [34] Jie Lin and David Mohaisen. 2025. From large to mammoth: A comparative evaluation of large language models in vulnerability detection. In Proceedings of the 2025 Network and Distributed System Security Symposium (NDSS). [35] Filipe Marques, Mafalda Ferreira, André Nascimento, Miguel E Coimbra, Nuno Santos, Limin Jia, and José Fragoso Santos. 2025. Automated Exploit Generation for Node. js Packages. Proceedings of the ACM on Programming Languages 9, PLDI (2025), 1341–1366. [36] Ruijie Meng, Martin Mirchev, Marcel Böhme, and Abhik Roychoudhury. 2024. Large language model guided protocol fuzzing. In Proceedings of the 31st Annual Network and Distributed System Security Symposium (NDSS), Vol. 2024. [37] Yuzhou Nie, Hongwei Li, Chengquan Guo, Ruizhe Jiang, Zhun Wang, Bo Li, Dawn Song, and Wenbo Guo. 2025. VulnLLM-R: Specialized Reasoning LLM with Agent Scaffold for Vulnerability Detection. arXiv preprint arXiv:2512.07533 (2025). [38] Yu Nong, Haoran Yang, Long Cheng, Hongxin Hu, and Haipeng Cai. 2025. { APPATCH } : Automated adaptive prompting large language models for { RealWorld } software vulnerability patching. In 34th USENIX Security Symposium (USENIX Security 25). 4481–4500. [39] Christoforos Ntantogian, Panagiotis Bountakas, Dimitris Antonaropoulos, Constantinos Patsakis, and Christos Xenakis. 2021. NodeXP: NOde. js server-side JavaScript injection vulnerability DEtection and eXPloitation. Journal of Information Security and Applications 58 (2021), 102752. [40] Marc Ohm, Henrik Plate, Arnold Sykosch, and Michael Meier. 2020. Backstabber’s knife collection: A review of open source software supply chain attacks. In International Conference on Detection of Intrusions and Malware, and Vulnerability

Assessment. Springer, 23–43. [41] Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2025. Asleep at the keyboard? assessing the security of github copilot’s code contributions. Commun. ACM 68, 2 (2025), 96–105. [42] Marco Pistoia, Satish Chandra, Stephen J Fink, and Eran Yahav. 2007. A survey of static analysis methods for identifying security vulnerabilities in software systems. IBM systems journal 46, 2 (2007), 265–288. [43] Zhuoyun Qian, Fangtian Zhong, Qin Hu, Yili Jiang, Jiaqi Huang, Mengfei Ren, and Jiguo Yu. 2025. Software Vulnerability Analysis Across Programming Language and Program Representation Landscapes: A Survey. arXiv preprint arXiv:2503.20244 (2025). [44] Koushik Sen, Swaroop Kalasapur, Tasneem Brutch, and Simon Gibbs. 2013. Jalangi: A selective record-replay and dynamic analysis framework for JavaScript. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering. 488–498. [45] Xiangmin Shen, Lingzhi Wang, Zhenyuan Li, Yan Chen, Wencheng Zhao, Dawei Sun, Jiashui Wang, and Wei Ruan. 2025. Pentestagent: Incorporating llm agents to automated penetration testing. In Proceedings of the 20th ACM Asia Conference on Computer and Communications Security. 375–391. [46] Ze Sheng, Zhicheng Chen, Shuning Gu, Heqing Huang, Guofei Gu, and Jeff Huang. 2025. Llms in software security: A survey of vulnerability detection techniques and insights. Comput. Surveys 58, 5 (2025), 1–35. [47] Deniz Simsek, Aryaz Eghbali, and Michael Pradel. 2025. PoCGen: Generating Proof-of-Concept Exploits for Vulnerabilities in Npm Packages. arXiv preprint arXiv:2506.04962 (2025). [48] Chao Wang, Ronny Ko, Yue Zhang, Yuqing Yang, and Zhiqiang Lin. 2023. Taintmini: Detecting flow of sensitive data in mini-programs with static taint analysis. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 932–944. [49] Dawei Wang, Geng Zhou, Li Chen, Dan Li, and Yukai Miao. 2024. Prophetfuzz: Fully automated prediction and fuzzing of high-risk option combinations with only documentation via large language model. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security. 735–749. [50] Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. 2024. Openhands: An open platform for ai software developers as generalist agents. arXiv preprint arXiv:2407.16741 (2024). [51] Yonghao Wu, Zheng Li, Jie M Zhang, Mike Papadakis, Mark Harman, and Yong Liu. 2023. Large language models in fault localisation. arXiv preprint arXiv:2308.15276 (2023). [52] HanXiang Xu, ShenAo Wang, Ningke Li, Kailong Wang, Yanjie Zhao, Kai Chen, Ting Yu, Yang Liu, and HaoYu Wang. 2024. Large language models for cyber security: A systematic literature review. ACM Transactions on Software Engineering and Methodology (2024). [53] John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems 37 (2024), 50528–50652. [54] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations. [55] Zidong Zhang, Qinsheng Hou, Lingyun Ying, Wenrui Diao, Yacong Gu, Rui Li, Shanqing Guo, and Haixin Duan. 2024. Minicat: Understanding and detecting cross-page request forgery vulnerabilities in mini-programs. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security. 525–539. [56] Xiaogang Zhu, Wei Zhou, Qing-Long Han, Wanlun Ma, Sheng Wen, and Yang Xiang. 2025. When software security meets large language models: A survey. IEEE/CAA Journal of Automatica Sinica 12, 2 (2025), 317–334. [57] Yuxuan Zhu, Antony Kellermann, Akul Gupta, Philip Li, Richard Fang, Rohan Bindu, and Daniel Kang. 2024. Teams of llm agents can exploit zero-day vulnerabilities. arXiv preprint arXiv:2406.01637 (2024). [58] Markus Zimmermann, Cristian-Alexandru Staicu, Cam Tenny, and Michael Pradel. 2019. Small world with high risks: A study of security threats in the npm ecosystem. In 28th USENIX Security symposium (USENIX security 19). 995–1010.

Taint-Style Vulnerability Detection and Confirmation for Node.js Packages Using LLM Agent Reasoning

is to sample a fixed number of packages per vulnerability (65 in our experiments) such that the distribution over the three structural metrics is approximately uniform, while the proportion of minified packages follows the distribution observed in the crawled dataset, subject to a minimum of one minified package whenever minified artifacts exist. We implement this goal using a stratified sampling procedure: • Metric Bucketing: For each vulnerability class 𝑣 and each structural metric 𝑚 𝑗 ∈ {js_ts_loc, dependency_count, js_ts_files}, we discretize metric values into 𝑏 = 5 buckets using empirical quantiles. For 𝑘 ∈ {1, . . . , 𝑏 − 1}, we define bucket cutoffs as 𝜃 𝑗,𝑘 = quantile𝑘/𝑏 ({𝑚 𝑗 (𝑥) | 𝑥 ∈ 𝑣 }), and assign each package 𝑥 to a bucket 𝐵 𝑗 (𝑥) according to the interval in which 𝑚 𝑗 (𝑥) falls. • Group Definition: Each package is assigned to a joint group

Appendix A Regular Expressions for Filtering Crawled npm Packages Here are the regular expressions we used to filter potentially vulnerable packages from the crawled npm packages for each vulnerability type: Code Injection. – \b(?:eval|Function)\s*$: eval/Function usage. Command Injection. – child_process\s*\.\s*(?:exec|spawn): child_process exec/spawn usage. – require\(\s*['\"]child_process['\"]\s*$: require (’child_process’). – \b(?:execSync|spawnSync)\s*\(: execSync/spawnSync usage.

𝑆 (𝑥) = (𝐵 1 (𝑥), 𝐵 2 (𝑥), 𝐵 3 (𝑥), 𝑀 (𝑥)),

where 𝑀 (𝑥) ∈ {minified, plain} denotes the minification status. • Structural Allocation: Ignoring the minification indicator, we target an approximately uniform allocation across the structural bucket tuples (𝐵 1, 𝐵 2, 𝐵 3 ) by iterating these tuples in randomized round robin order and selecting packages to spread samples evenly across different structural configurations. • Minification Allocation: Let 𝐶 min and 𝐶 plain denote the counts of minified and plain packages in the crawled dataset for vulnerability 𝑣. We set sampling budgets 𝑁 min and 𝑁 plain proportional to 𝐶 min and 𝐶 plain , while enforcing 𝑁 min ≥ 1 whenever 𝐶 min > 0. • Selection: Within each structural bucket tuple (𝐵 1, 𝐵 2, 𝐵 3 ), we draw packages while respecting the remaining minified and plain budgets. If one category is not available within a bucket, we draw from the other category. If a quota cannot be met due to global exhaustion, we fill the remaining slots from the available category. • Post Check: For each metric 𝑚 𝑗 and each bucket 𝑡 that appears among the sampled packages, if the sample contains at least one plain package in (𝑚 𝑗 , 𝑡) but zero minified packages in (𝑚 𝑗 , 𝑡), and the crawled dataset contains at least one minified package in (𝑚 𝑗 , 𝑡), then we replace one sampled plain package from (𝑚 𝑗 , 𝑡) with an unused minified package from (𝑚 𝑗 , 𝑡) if such a minified package exists.

Prototype Pollution. – \bObject\s*\.\s*assign\s*$[^)]*\b(?:req\s*\.\ ⌋ s*(?:body|query|params)|JSON\s*\.\s*parse\s*\( ⌋ |qs\s*\.\s*parse\s*\()\b[^)]*$: Object.assign fed by req.* or JSON.parse. – =\s*\{[^}]*\.\.\.(?:req\s*\.\s*(?:body|query|p ⌋ arams)|JSON\s*\.\s*parse\s*\(|qs\s*\.\s*parse ⌋ \s*\()[^}]*\}: Spread merge with attacker-controlled object. – \b(?:set|assign|merge|extend|defaultsDeep|deep ⌋ Merge|deepExtend)\b\s*\([^)]*\b(?:req\s*\.\s*( ⌋ ?:body|query|params)|JSON\s*\.\s*parse\s*\(|q ⌋ s\s*\.\s*parse\s*\()\b: Generic deep merge helpers with tainted input. – \b_\s*\.\s*(?:merge|mergeWith|defaultsDeep|set ⌋ |setWith|update|updateWith)\s*\(: lodash-style risky helpers.

Prompts Templates

Here we include the detailed prompt templates used for each stage of the pipeline and for each vulnerability class. The placeholder variables in the templates are dynamically filled during the execution of the pipeline.

Sampling Algorithms for Crawled npm Packages

We characterize each package using four features: three structural metrics js_ts_loc (lines of JavaScript or TypeScript code), dependency_count (number of declared dependencies), and js_ts_files (number of JavaScript or TypeScript source files), together with a binary indicator of whether the package contains minified code (by simply checking whether .min. or bundle appears in the filenames of JavaScript or TypeScript files). Our goal

C.1

Stage 1: Finder Prompt

C.1.1 System Prompt. You are an expert security researcher specializing ↩→ in Node.js vulnerabilities. 15

Ronghao Ni, Mihai Christodorescu, and Limin Jia

Your task is to find instances of <VULN_TYPE> ↩→ vulnerabilities in the project at: ↩→ <PROJECT_PATH>

CURRENT DIRECTORY: <PROJECT_PATH> All file paths are relative to this directory. Call ↩→ get_file_tree() first to see the structure if ↩→ needed.

<VULN_DESCRIPTION> WORKFLOW: 1. Start by getting the file tree or listing files ↩→ to understand the structure 2. Search for patterns related to <VULN_TYPE> 3. Read suspicious files to analyze the code 4. Identify exact locations (file + line number) of ↩→ vulnerabilities 5. Determine which public APIs can reach these ↩→ vulnerabilities 6. Call submit_findings(findings=[...]) with ↩→ structured arguments (NO JSON STRINGS). You can ↩→ call this multiple times as you discover items. 7. When you are completely done adding findings, ↩→ call finish(summary="...optional...") to end ↩→ the run.

ANALYSIS CHECKLIST: 1. Read the code at the reported location 2. Trace data flow to see if user input can reach ↩→ the vulnerable sink 3. Check for any input validation or sanitization 4. Determine if the vulnerability is actually ↩→ exploitable 5. Submit your verdict with detailed reasoning IMPORTANT - LIBRARY/PACKAGE ATTACK SURFACE: - When analyzing npm packages/libraries, EXPORTED ↩→ functions (exports.*, module.exports) are the ↩→ attack surface - If a vulnerable function is exported (even with ↩→ no callers in the codebase), it IS reachable by ↩→ external code - Focus on: Can user-controlled input reach the ↩→ sink IF the exported function is called? - Do NOT search for external callers - the export ↩→ itself makes it callable

FINDINGS FORMAT (submit_findings arguments): - findings: [ { "vuln_type": "<VULN_TYPE>", "file": "relative/path/to/file.js", "line": 42, "description": "Brief description", "evidence": "Code snippet showing the ↩→ issue", "reachable_apis": ["api1", "api2"], "confidence": 0.85 } ]

VERDICT FORMAT (submit_verdict arguments): - is_valid: true/false - reason: "Detailed explanation with evidence" - confidence: 0.0-1.0 Call submit_verdict(is_valid=true/false, ↩→ reason="...", confidence=0.0-1.0). Do NOT wrap ↩→ the arguments in JSON strings.

Be thorough and precise. Focus on actionable evidence but err on the side of inclusion: if a ↩→ spot looks plausibly exploitable yet you lack ↩→ ↩→ full confirmation, include it with a lower ↩→ confidence score and clearly state any assumptions. When you have submitted all ↩→ findings, call finish to end the run. ↩→

Be conservative - only mark as valid if truly ↩→ exploitable. C.2.2

Validate this vulnerability finding: Type: <VULN_TYPE> File: <FILE> Line: <LINE> Description: <DESCRIPTION> Evidence: <EVIDENCE> Reachable APIs: <REACHABLE_APIS> Confidence: <CONFIDENCE>

C.1.2 User Prompt. Find all <VULN_TYPE> vulnerabilities in the project. ↩→ Use tools to analyze the code, then submit your ↩→ findings.

C.2

User Prompt.

Stage 2: Judge Prompt Use tools to analyze the code and submit your ↩→ verdict.

C.2.1 System Prompt. You are an expert security code reviewer ↩→ specializing in Node.js vulnerabilities. Your task is to validate whether a reported ↩→ <VULN_TYPE> vulnerability is actually ↩→ exploitable.

C.3

Stage 3: Constraints Inferencer Prompt

C.3.1

System Prompt.

You are an expert security researcher analyzing how ↩→ to exploit vulnerabilities. 16

Taint-Style Vulnerability Detection and Confirmation for Node.js Packages Using LLM Agent Reasoning

- Only if you truly need a background service ↩→ (e.g., start a server), use ↩→ start_persistent_process, inspect with ↩→ check_persistent_process, and always call ↩→ kill_persistent_process when finished. 4. Verify success (check file creation, output, ↩→ etc.) 5. If import fails, check the exact filename and ↩→ try again 6. Submit your result with ↩→ submit_exploit_result(success=..., ↩→ exploit_code="...", execution_output="...", ↩→ explanation="...")

Your task is to analyze a <VULN_TYPE> vulnerability ↩→ and determine the constraints for exploitation. ANALYSIS OBJECTIVES: 1. Identify the entry point (URL, function, API) 2. Determine parameter names and how to provide them 3. Understand required format/structure of the ↩→ payload 4. Identify any validation to bypass 5. Define success criteria for the exploit CONSTRAINTS FORMAT (submit_constraints arguments): - constraints: "Detailed natural language ↩→ description of exploitation requirements" ↩→ (REQUIRED) - entry_point: "How to reach the vulnerability" - parameters: ["param1", "param2"] - payload_format: "Required format for the payload"

Be creative and persistent. Learn from failures. C.4.2

Create and execute an exploit for: Type: <VULN_TYPE> File: <FILE> Line: <LINE>

Be specific and actionable - the exploit agent will ↩→ use this. C.3.2

Constraints: <CONSTRAINTS_TEXT>

User Prompt.

Generate exploitation constraints for:

Create a working exploit and submit the result.

Type: <VULN_TYPE> File: <FILE> Line: <LINE> Description: <DESCRIPTION> Evidence: <EVIDENCE>

Stage 4: Exploiter Prompt

C.4.1

System Prompt.

C.5

Stage 4 Dynamic Goal Templates (Per Vulnerability Type)

C.5.1

os_command_injection.

Craft a payload that injects OS commands (e.g., ↩→ using ; || && or backticks). The exploit should create the file ↩→ /tmp/os_cmd_success to prove arbitrary command ↩→ execution. Example: `; touch /tmp/os_cmd_success` or `$(touch ↩→ /tmp/os_cmd_success)`

Analyze the code and submit detailed constraints.

C.4

User Prompt.

You are an expert security researcher crafting ↩→ exploits for <VULN_TYPE>.

C.5.2 Your task is to create a working exploit based on ↩→ the provided constraints.

code_injection.

Craft a payload that injects JavaScript code into ↩→ eval(), Function(), vm.runInNewContext(), or similar APIs. The exploit harness defines ↩→ global.CTF = () => console.log('GLOBAL.CTF ↩→ HIT'). Your malicious payload must call global.CTF() so ↩→ that the marker appears in STDOUT, proving arbitrary code execution without relying on ↩→ filesystem writes.

EXPLOIT GOAL: <EXPLOIT_INSTRUCTIONS_FROM_VULN_DEFINITION> Dependency setup: Install any missing dependencies ↩→ in the package directory before running your ↩→ exploit (use run_shell_command if you need ↩→ npm/yarn).

C.5.3 EXPLOIT WORKFLOW: 1. Read the vulnerable code to understand it 2. Craft exploit code that imports the module: ↩→ const mod = require('./index'); 3. Execute the exploit using execute_javascript

path_traversal.

Craft a payload using ../ sequences to escape the ↩→ intended directory and read ↩→ /tmp/path_traversal. The harness writes the sentinel string ↩→ 'PATH_TRAVERSAL' into that file ahead of time. 17

Ronghao Ni, Mihai Christodorescu, and Limin Jia

Use the vulnerability to read it and console.log the ↩→ file contents so STDOUT contains the marker.

path but fails to properly sanitize special ↩→ characters (e.g., ../), allowing access to unintended locations outside the intended ↩→ directory.

C.5.4 prototype_pollution. Goal: trigger prototype pollution so that a newly ↩→ created object inherits attacker-controlled ↩→ data.

C.6.4 prototype_pollution. CWEs: 1321

Use the package's public API as intended (e.g., ↩→ merge, copy, set, or path-based helpers). The attacker controls input values such as objects, ↩→ keys, or paths.

Prototype Pollution (CWE-1321): attacker-controlled ↩→ keys or paths are used in object writes or merge/copy operations, causing a shared prototype ↩→ (often Object.prototype) to be modified and affecting subsequently created objects.

You may use payloads such as: - objects containing prototype-related keys - dotted paths (e.g., '__proto__.polluted', ↩→ 'constructor.prototype.polluted') - arrays of path segments if supported by the API

Indicators include dynamic property access ↩→ (obj[key]), merge/copy utilities, recursive ↩→ assignment patterns, and path-based setters. The vulnerable ↩→ code may not explicitly reference '__proto__' ↩→ or 'constructor.prototype'.

Do NOT use unrealistic techniques such as ↩→ Object.defineProperty, Object.setPrototypeOf, ↩→ or direct writes to Object.prototype.

IMPORTANT: - Do NOT try to manually check for success. - The JavaScript run tool automatically probes, and ↩→ if pollution occurred it will emit 'PROTO_POLLUTION SUCCESS' on its own. - Do NOT print or hardcode the success token.

C.6

Full File-Level Matching Results and Metric Definitions

The metrics we used for file-level matching are formally defined as follows: let 𝑑 denote a dataset (SecBench.js, VulcaN, or their union) and 𝑠 ∈ {Finder, Judge} denote a stage. For each (𝑑, 𝑠): • Let 𝐹𝑑,Finder be the set of findings produced by the Finder stage, and let 𝐹𝑑,Judge be the subset of findings deemed valid by the Judge stage. • Let 𝐺𝑑 be the set of ground-truth vulnerable files. • A finding 𝑓 ∈ 𝐹𝑑,𝑠 is matched if there exists 𝑔 ∈ 𝐺𝑑 under our file-level matching rule (same package-version pair, vulnerability type, and file). Otherwise, 𝑓 is unmatched. We report the following four metrics, which correspond directly to Table 6 and Table 8:

Vulnerability Definitions

C.6.1 os_command_injection. CWEs: 077, 078 OS Command Injection (CWE-78): The product ↩→ constructs and executes OS commands using unsanitized input, allowing attackers to ↩→ execute arbitrary system commands.

Finder Findings Unmatched(𝑑) |{𝑓 ∈ 𝐹𝑑,Finder : 𝑓 unmatched}| |𝐹𝑑,Finder | Judge Findings Unmatched(𝑑) =

Common sink APIs: exec, execSync, execFile, ↩→ execFileSync, spawn, spawnSync, child_process module functions

|{𝑓 ∈ 𝐹𝑑,Judge : 𝑓 unmatched}| |𝐹𝑑,Judge | Finder GT Unmatched(𝑑) =

C.6.2 code_injection. CWEs: 094

|{𝑔 ∈ 𝐺𝑑 : 𝑓 ∈ 𝐹𝑑,Finder matched to 𝑔}| |𝐺𝑑 | Judge GT Unmatched(𝑑) =

Code Injection (CWE-94): The product dynamically ↩→ generates or evaluates code using untrusted input, allowing attackers to inject ↩→ and execute arbitrary JavaScript code.

C.6.3 path_traversal. CWEs: 022, 035

|{𝑔 ∈ 𝐺𝑑 : 𝑓 ∈ 𝐹𝑑,Judge matched to 𝑔}| |𝐺𝑑 |

The first two metrics measure the fraction of tool reports that are unmatched; the latter two measure the fraction of benchmark vulnerable files not covered by any report at each stage. As shown in Table 8, the detailed file-level matching results are broken down by benchmark (SecBench.js and VulcaN) and

Path Traversal (CWE-22): The product uses external ↩→ input to construct a file or directory 18

Taint-Style Vulnerability Detection and Confirmation for Node.js Packages Using LLM Agent Reasoning

Table 8: Detailed file-level unmatched results by dataset and vulnerability type for Finder and Judge stages. “Findings Unmatched” reports unmatched-rate on tool findings, and “GT Unmatched” reports unmatched-rate on ground-truth files. Dataset SecBench.js SecBench.js SecBench.js SecBench.js SecBench.js VulcaN VulcaN VulcaN VulcaN VulcaN All All All All All

Vulnerability Finder Findings Unmatched Judge Findings Unmatched Finder GT Unmatched Judge GT Unmatched CWE-22 27/221 (12.22%) 25/218 (11.47%) 2/156 (1.28%) 2/156 (1.28%) CWE-471 113/343 (32.94%) 92/299 (30.77%) 10/118 (8.47%) 12/118 (10.17%) CWE-78 21/201 (10.45%) 18/194 (9.28%) 3/78 (3.85%) 3/78 (3.85%) CWE-94 28/59 (47.46%) 25/55 (45.45%) 1/21 (4.76%) 1/21 (4.76%) All 189/824 (22.94%) 160/766 (20.89%) 16/373 (4.29%) 18/373 (4.83%) CWE-22 4/8 (50.00%) 4/8 (50.00%) 0/3 (0.00%) 0/3 (0.00%) CWE-471 34/146 (23.29%) 26/130 (20.00%) 6/63 (9.52%) 6/63 (9.52%) CWE-78 40/163 (24.54%) 35/128 (27.34%) 5/63 (7.94%) 8/63 (12.70%) CWE-94 30/58 (51.72%) 28/53 (52.83%) 5/22 (22.73%) 8/22 (36.36%) All 108/375 (28.80%) 93/319 (29.15%) 16/151 (10.60%) 22/151 (14.57%) CWE-22 31/229 (13.54%) 29/226 (12.83%) 2/159 (1.26%) 2/159 (1.26%) CWE-471 147/489 (30.06%) 118/429 (27.51%) 16/181 (8.84%) 18/181 (9.94%) CWE-78 61/364 (16.76%) 53/322 (16.46%) 8/141 (5.67%) 11/141 (7.80%) CWE-94 58/117 (49.57%) 53/108 (49.07%) 6/43 (13.95%) 9/43 (20.93%) All 297/1199 (24.77%) 253/1085 (23.32%) 32/524 (6.11%) 40/524 (7.63%)

vulnerability type, including both unmatched-finding rates and unmatched-ground-truth rates.

Record · ID 123964 · SHA-256 7048c97385eaeb15
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.