arXiv CSopen access

ATTAIN: Automated Exploit Failure Analysis through Trace-Driven Diff Analysis

arXiv:2606.09060v1 [cs.SE] 8 Jun 2026

ATTAIN: Automated Exploit Failure Analysis through Trace-Driven Diff Analysis Xinwei Mao

Zirui Chen

School of Software Technology, Zhejiang University Ningbo, China [email protected]

The State Key Laboratory of Blockchain and Data Security, Zhejiang University Hangzhou, China [email protected]

Xing Hu∗

Xin Xia

The State Key Laboratory of Blockchain and Data Security, Zhejiang University Hangzhou, China [email protected]

The State Key Laboratory of Blockchain and Data Security, Zhejiang University Hangzhou, China [email protected]

Abstract Exploits are widely used to check whether library vulnerabilities appear in different versions and to mark affected version ranges. Exploit-based checks sometimes fail because exploits stop running on many versions after API or environment changes. Commit-based methods, such as SZZ-style analysis, sometimes miss the right introduce commits and spread labels incorrectly along long version chains. These problems leave many affected versions unlabeled or wrongly labeled and make manual exploit failure analysis very expensive and impractical at scale. We present Attain, a trace-driven diff analysis framework with three modules to assess vulnerability presence across evolving library versions. The modules are trace construction, diff exploration, and affected-version judgment. The trace construction module executes an exploit across historical library versions and compares their behaviors to capture cross-version execution divergences. Using these divergences, the diff exploration module guides an LLM through a finite-state tool loop to autonomously search over version changes and collect vulnerability-relevant diff hunks. The affectedversion judgment module reasons over the collected evidence to determine whether the vulnerability exists in each version and outputs the affected version range. We evaluate Attain on an extensive dataset comprising 224 CVEs and 25,943 library versions across 128 libraries. Attain achieves an F1-score of 93.24%, outperforming the commit-based methods V-SZZ and LLM4SZZ by 116.28% and 33.30% respectively. Attain uses short tool-guided prompts and a fixed number of iterations, keeping token usage low. It matches or surpasses existing methods on frequent CWE types, including cases where exploit runs fail ∗ Corresponding authors

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. Conference’17, Washington, DC, USA © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-x-xxxx-xxxx-x/YYYY/MM https://doi.org/10.1145/nnnnnnn.nnnnnnn

for non-vulnerability reasons or commit messages do not clearly delimit affected versions.

CCS Concepts • Security and privacy → Software security engineering.

Keywords Library Vulnerabilities, Exploit Trace, Affected Version ACM Reference Format: Xinwei Mao, Zirui Chen, Xing Hu, and Xin Xia. 2026. ATTAIN: Automated Exploit Failure Analysis through Trace-Driven Diff Analysis. In . ACM, New York, NY, USA, 13 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn

1

Introduction

Vulnerabilities in open-source libraries have become a critical security concern in modern software development, because these libraries are widely reused to reduce redundant implementation effort and accelerate software delivery [24, 47, 53, 59, 63], allowing vulnerable upstream components to affect numerous downstream projects [3–5, 18, 25, 27, 32, 38–40, 58, 60–62]. To assess these risks, downstream maintainers need to determine whether their projects depend on vulnerable library versions [6, 7]. A commonly adopted practice is to consult the affected version ranges disclosed by vulnerability databases, such as NVD and Snyk. However, recent studies have shown that these databases often suffer from inaccuracies and inconsistencies [9, 13, 22]. To address this challenge, researchers have proposed patch-based approaches [2, 53] that locate vulnerability-related information from security patches and then assess whether these statements or their surrounding structures exist in historical versions. However, these techniques still face challenges when the introducing commit and the fixing patch are structurally distant. For example, CVE-2023-51080 in Hutool is introduced in version 5.8.22, where the commit [33] adds a recursive fallback path in toBigDecimal. When receiving malformed numeric inputs, such as ‘NaN’, this recursive path repeatedly invokes number parsing and eventually causes a ‘StackOverflowError’. The vulnerability is fixed in version 5.8.25 by adding an assertion to reject invalid parsed numbers. Existing patch-based approaches, such as Vision [53], treat

Conference’17, July 2017, Washington, DC, USA

all versions before 5.8.25 as affected due to failing to identify the actual vulnerability-introducing change. To address the above challenge, we need to accurately identify vulnerability-introducing diff hunks beyond structural similarity to the fixing patch. We make two key observations: ❶ vulnerability exploit execution traces across versions provide useful signals for localizing related functions; and ❷ Deep learning, especially large language models (LLMs), which are widely used due to their capabilities in various code-related tasks [14, 15, 17, 26, 44, 49, 55–57, 64], can reason about whether a change may introduce vulnerabilityrelated behavior. Based on these observations, we propose Attain, an LLM-guided framework that autonomously searches crossversion changes based on exploit execution behavior, collects relevant diff hunks, and determines affected versions based on the collected evidence. Attain consists of three modules. Given a public exploit and historical library versions, the trace construction module executes a public exploit across historical library versions and compares their behaviors to capture cross-version execution divergences if the exploit triggers the vulnerability in one version but fails to reproduce or compile in the nearby version. Second, for each version where the exploit is not reproduced, the diff exploration module uses these divergences to guide an autonomous search over version changes and collect vulnerability-relevant diff hunks. Third, the affected-version judgment module reasons over the collected evidence to determine whether the vulnerability exists in each version and outputs the affected version range. We evaluate Attain on an extensive dataset comprising 224 CVEs and 25,943 library versions across 128 Java libraries. Attain achieves an F1-score of 93.24%, outperforming the commit-based methods V-SZZ and LLM4SZZ by 116.28% and 33.30% respectively. Meanwhile, Attain uses short trace-guided prompts and bounded iterations to keep token costs low enough for practical deployment. Our ablation and CWE-level analyses show that trace-diff guidance, dynamic diff search, rule-based aggregation, and versionchain backfill all contribute to effectiveness and coverage. These components are particularly helpful when exploits fail for nonvulnerability reasons or commit messages do not clearly delimit affected versions and traces. This paper makes the following main contributions: • We propose Attain, a trace-driven diff analysis framework that autonomously searches cross-version changes from exploit execution traces, collects vulnerability-relevant diff hunks, and determines affected versions from the collected evidence. The source code and dataset of Attain are publicly available in the online replication package [34]. • We conduct an evaluation on 224 CVEs and 25,943 library versions across 128 libraries. Attain achieves an F1-score of 93.24%, outperforming the commit-based methods V-SZZ and LLM4SZZ by 116.28% and 33.30% respectively. The remainder of this paper is organized as follows. Section 2 motivates our work. Section 3 presents Attain’s three modules. Section 4 describes the experimental setup. Section 5 reports results against baselines, with ablation and per-CWE analyses. Section 6 discusses typical failures and threats. Section 7 reviews related work. Section 8 summarizes the paper.

Xinwei Mao, Zirui Chen, Xing Hu, and Xin Xia

2

Motivation

This section explains why identifying affected versions is challenging. We first show that patch-based approaches are inaccurate when the introducing and fixing commits are structurally distant, motivating the use of exploits. We then discuss why failed exploit runs are ambiguous and lead to low recall. Finally, we outline how Attain analyzes exploit failures along a version chain to strengthen affected-version identification.

2.1

Exploit Signals

Existing patch-based approaches locate vulnerability-related information from security patches and check its presence in historical versions [53]. However, they become inaccurate when the vulnerability-introducing commit and the fixing patch are structurally distant. For example, CVE-2023-51080 in Hutool introduces a recursive fallback path in version 5.8.22 and fixes it with an assertion in 5.8.25. In contrast, running the exploit on 5.8.22 directly triggers the vulnerability, providing unambiguous evidence. This illustrates that successful exploit executions yield high-precision affected-version signals.This example also clarifies why we choose exploit-driven analysis as the core signal. For CVE-2023-51080, patch-only matching over-approximates affected versions because the introducing and fixing commits are structurally distant, while exploit execution directly tests vulnerability semantics on each concrete version. Moreover, exploit outcomes remain useful even when they are negative: with cross-version traces and diffs, a failed run can be diagnosed as pre-introduction, true fix, or exploit-version mismatch, turning ambiguous negatives into actionable evidence and improving recall without sacrificing precision. However, a failed exploit run is ambiguous: it may mean the vulnerability has not yet been introduced, has already been fixed, or remains present but the exploit fails due to breaking changes. Existing workflows rarely distinguish these cases, leading to low recall or requiring extensive manual analysis.

2.2

Motivating Examples

Our key observation is that exploit failures do not occur in isolation: they happen along a version chain where code and behavior evolve over time. Attain combines cross-version execution traces with version diffs to explain exploit failures at the level of concrete code changes, aiming to separate two categories: (1) pre-introduction failures, where the vulnerable behavior has not yet been introduced; and (2) breaking-change failures, where the vulnerability persists but API or environment changes prevent the exploit from running. We next discuss one representative case for each category. 2.2.1 Case 1: Pre-introduction. CVE-2023-1436 in jettison affects version 1.3.1, and our exploit explicitly targets the constructor JSONObject(Map map). The exploit builds a self-referential input map and invokes JSON object construction; inside the constructor, nested maps are recursively wrapped (e.g., myHashMap.put(k, new JSONObject((Map) v));), which repeatedly re-enters the same object graph. As a result, the exploit triggers an infinite recursion and eventually a StackOverflowError in version 1.3.1.

ATTAIN: Automated Exploit Failure Analysis through Trace-Driven Diff Analysis

base: jettison-1.3.1

…

Conference’17, July 2017, Washington, DC, USA

Reference(vulnerable path) - reaches RCE

compare: jettison-1.2

src/main/java/org/codehaus/jettison/json/JSONObject.java

Entry (Test Code)

Earlier Version(crashes) - AIOOBE

XMLTest.testFromXML() XStream.fromXML(xml)

@@ -225,16 +224,6 @@ public JSONObject(Map map) { 225 226 227 228

-

229 230 231 232 233

237 238

if (v instanceof Collection) { myHashMap.put(k, new JSONArray((Collection) v)); }

-

234 235 236

if (v instanceof Map) { myHashMap.put(k, new JSONObject((Map) v)); }

227

XStream Deserailization

this.myHashMap = (map == null) ? new LinkedHashMap() : new LinkedHashMap(map) // ensure a pure hierarchy of JSONObjects and JSONArrays for (Object k : myHashMap.keySet()) { Object v = myHashMap.get(k);

224 225 226

}

TreeUnmarshaller

TreeUnmarshaller

AbstractReflectionConverter

AbstractReflectionConverter

(…)Converters

(…)Converters

ReflectionProvider

FieldDictionary (com.thoughtworks.xstream. core.util.FieldDictionary)

Fileds(com.thoughtworks.xstream. core.util.Fields)

Field Resolution Infrastructure

OrderRetainingMap(fieldCache)

OrderRetainingMap(fieldCache)

}

Array-based bookkeeping Java.lang.reflect.Field

Figure 1: Pre-introduction failure case for CVE-2023-1436 in org.codehaus.jettison:jettison, comparing versions 1.3.1 (reference, affected) and 1.2 (target, not affected).

ArrayIndexOutofBoundException

Gadget Chain & Outcome Gadget Chain Program terminates abnormally (PoC crashes) Remote Code Execution

Running the same exploit on version 1.2 does not reproduce the stack overflow. As shown in Figure 1, Attain analyzes the crossversion traces and diffs: in the reference trace (1.3.1), the execution reaches the recursive path that causes the stack overflow; in the target trace (1.2), the execution diverges earlier and never enters this path, and the corresponding diff shows that the vulnerable recursive logic is absent. Based on this combination, Attain classifies 1.2 as a pre-introduction failure: the exploit fails because the vulnerable behavior has not yet been introduced. 2.2.2 Case 2: Breaking Change. CVE-2021-21351 in xstream is a deserialization vulnerability where a crafted XML payload drives the reflective deserialization pipeline of XStream, potentially leading to remote code execution. Both versions 1.4.6 and 1.4.5 are affected. In version 1.4.6 (reference), the exploit follows the solid path in Figure 2, traversing the reflective deserialization pipeline and reaching the RCE-enabling gadget chain. In version 1.4.5, the exploit crashes early with ArrayIndexOutOfBoundsException inside OrderRetainingMap.entrySet, before reaching the vulnerable reflection logic. To distinguish this from a true non-affected case, Attain again uses a multi-signal check. First, the failure signature is a runtime crash in field-resolution infrastructure (AIOOBE), not a clean disappearance of vulnerable semantics. Second, trace alignment shows that both versions share the same upper reflective pipeline (TreeUnmarshaller and AbstractReflectionConverter) and diverge around FieldDictionary/OrderRetainingMap. Third, the version diff highlights behavior changes in helper code such as core.util.Fields, while the core vulnerable deserialization chain is still present. Together, these signals indicate exploit invalidation due to breaking changes: version 1.4.5 remains affected, but this exploit instance no longer runs to completion on that environment.

Figure 2: Breaking-change failure case for CVE-2021-21351 in com.thoughtworks.xstream:xstream, comparing versions 1.4.6 (reference, affected) and 1.4.5 (target, affected).

3

Proposed Approach

We now describe Attain, our approach to cross-version affectedversion determination. The method operates through three modules: the Trace Construction Module, which captures cross-version execution divergences; the Diff Exploration Module, which recovers vulnerability-relevant evidence from version diffs; and the Affected-Version Judgment Module, which reasons over the evidence and extends labels along the version chain. Figure 3 presents the overall workflow.

3.1

Trace Construction Module

Given a public exploit and the historical versions of a library, the trace construction module executes the exploit across versions and compares their behaviors. Its goal is to capture cross-version execution divergences when the exploit triggers the vulnerability in one version but fails to reproduce or compile in a nearby version. When multiple exploits exist for the same CVE, each exploit runs independently through the entire pipeline. We start from a known CVE and its publicly available exploit. We also have two versions of the same library. One is the reference version 𝑣𝑏 , where the exploit can trigger the vulnerability. The other is the target version 𝑣𝑡 , where the exploit behaves differently. Attain runs the exploit on both versions and records execution traces. These traces form an execution trace chain.

Conference’17, July 2017, Washington, DC, USA

For each version, Attain uses the Maven testing framework to run the exploit. A lightweight Java agent based on ASM bytecode manipulation instruments every non-abstract method within the target library’s package namespace, recording fully qualified class name, method name, and descriptor on each method entry. The execution trace chain includes three parts: • Method execution sequence: the method call path during exploit execution, shown as fully qualified class and method names. • Dependency tree: the complete dependency graph obtained from the build system. • Execution observations: whether compilation or runtime succeeds or fails, together with exception types and assertion failure messages. Trace divergence. Let 𝑇 (𝑣) denote the method enter sequence obtained by running the exploit on version 𝑣. A trace divergence 𝛿 is computed by element-wise comparison of 𝑇 (𝑣𝑏 ) and 𝑇 (𝑣𝑡 ): the system identifies the first index where the two sequences differ, and also records the set of methods unique to each version. Formally: 𝛿 = 𝑇 (𝑣𝑏 ) \ 𝑇 (𝑣𝑡 ) A trace divergence can appear in multiple forms: a compilation failure, a runtime exception, a difference in the method call sequences, or an early termination of one trace relative to the other. The two trace chains may differ from each other. This difference directly shows behavioral differences between the two versions. If the exploit triggers the vulnerability on 𝑣𝑏 but behaves differently on 𝑣𝑡 , then the vulnerability trigger path has diverged across the two versions. Recursive calls naturally produce repeated entries in the trace and are handled by the element-wise comparison without special treatment. For extremely large traces (exceeding 1 GB), the system switches to a lightweight mode that infers divergence from execution observations and trace file sizes rather than full element comparison. Version pair preprocessing. Before running the trace comparison, Attain needs to decide which version pairs to analyze and which commits are the relevant patches. The first half of this pipeline resolves version boundaries. It starts from the ground truth execution results. The system finds the upstream GitHub repository for each CVE, then sorts all tested versions by Maven release timestamp and constructs a branch-aware version tree. A version is placed on the main release chain unless its timestamp regresses relative to its predecessor, in which case it is attached to the corresponding maintenance branch. Adjacent version pairs are defined as consecutive versions along this tree where the exploit outcome flips from triggering the vulnerability to not triggering it, or vice versa. Versions not present in the execution results are excluded; unresolved versions are concatenated to the nearest resolved predecessor. Across the 224 CVEs, this algorithm generates 845 boundary version pairs, of which 459 unique pairs are selected for LLM-based analysis. Maven version strings do not always match GitHub tag names. The system queries the GitHub Tags API for each repository and matches Maven versions to GitHub tags by prefix completion, dot-to-underscore conversion, and strict pattern matching. For each version pair, the system calls the GitHub Compare API to get the list of commits between the two tags.

Xinwei Mao, Zirui Chen, Xing Hu, and Xin Xia

The second half of the pipeline filters commits and builds patch context. Given the commit list and the CVE description, the system uses dual-encoder retrieval to rank commit messages by their similarity to the CVE description. The top-𝑘 candidates are then passed to an LLM, which selects the most likely fix commits. Next, the system fetches the full commit details from GitHub, filtering out non-code files such as documentation, configuration, test files, and build scripts. Only commits with remaining valid files are kept. For each remaining commit, the system parses the diff hunks and fetches the surrounding source lines from GitHub. It applies two levels of pruning. Line-level pruning removes comments, import statements, and logging calls. Hunk-level pruning discards test files and empty diffs. Finally, a bi-encoder model scores each file-level diff against the CVE description. Files scoring above a threshold are labeled as patch. If no file passes the threshold, an LLM checks all candidates in descending score order and stops at the first positive verdict, yielding the final set of patch diffs.

3.2

Diff Exploration Module

For each version where the exploit is not reproduced, the diff exploration module uses the observed divergences to guide an autonomous search over version changes and collect vulnerabilityrelevant diff hunks. The divergence is passed to the LLM as a structured summary containing the first point of trace mismatch (with the diverging method signatures), the sets of methods unique to each version, and the execution observations. In a multi-exploit setting, each exploit independently feeds its own divergence into this module. Version diff. Let 𝐷 (𝑣𝑏 , 𝑣𝑡 ) denote the set of all diff hunks between the source code or bytecode of 𝑣𝑏 and 𝑣𝑡 . Each diff hunk ℎ ∈ 𝐷 is a localized code change. It includes the file path, the changed lines, and the surrounding context. Trace-diff context. A trace-diff context 𝐶𝑖 links an observed 𝛿𝑖 between 𝑣𝑏 and 𝑣𝑡 to its search state 𝑠𝑖 and the collected evidence hunks. It is defined as: 𝐶𝑖 = (𝛿𝑖 , 𝑠𝑖 , 𝐸𝑖concrete, 𝐸𝑖fallback ) Here 𝐸𝑖concrete contains diff hunks that directly explain 𝛿𝑖 . 𝐸𝑖fallback contains supplementary evidence such as trace-diff summaries, dependency-tree diffs, and build configuration diffs. These files provide contextual information but cannot by themselves prove that the vulnerability is absent. Evidence repair. When the collected evidence 𝐸𝑖concrete is empty, the system triggers the evidence repair procedure. This procedure selects the highest-scoring candidate hunks from 𝐷 (𝑣𝑏 , 𝑣𝑡 ) based on term matching against the scenario keywords and previously observed symbols. The repair result is denoted as: repair

𝐸𝑖

= top-𝑘 (score(ℎ, 𝛿𝑖 ) | ℎ ∈ 𝐷)

where score(ℎ, 𝛿𝑖 ) measures the relevance between hunk ℎ and 𝛿𝑖 . When a trace divergence 𝛿𝑖 is detected, the system constructs the corresponding 𝐶𝑖 and searches 𝐷 (𝑣𝑏 , 𝑣𝑡 ) for the evidence hunks associated with 𝐶𝑖 . Since 𝐷 (𝑣𝑏 , 𝑣𝑡 ) may contain many candidate hunks, directly recovering the evidence that explains 𝛿𝑖 is difficult. We therefore use a tool-augmented dynamic prompting approach

ATTAIN: Automated Exploit Failure Analysis through Trace-Driven Diff Analysis

Public Exploit Library Versions

Module 1: Trace Construction

1 Cross-Version Execution Setup

CVE Description GitHub Metadata

Baseline Version 𝒗𝒃

Maven Test

Baseline Version 𝒗𝒕

Maven Test

Java Agent

Role: Investigate diffs using traces & context Trace Divergence 𝛅 ++-

Version Diff D(𝒗𝒃 , 𝒗𝒕 )

Concrete Evidence

Java Agent

Trace Chain

Crash No Build

1 Target Prioritize High-impact divergence

4 Reflect Assess evidence quality & next steps

1 Case-Level Judgement

T(𝒗𝒃 )

Trace Divergence 𝛅

T(𝒗𝒕 )

𝛅 = T(𝒗𝒃 ) \ T(𝒗𝒕 )

Available Tools

Find Candidate hunks & related context

KEY Keyword Search

3 Inspect

Read File

++Load Diff Context

— — — — — — — — +- +- +- +-

Search Code Context

Concrete Evidence

Evidence Repair

Fallback Evidence

Select hunks if concrete evidence is empty — —

High-Signal Hunks

Candidate Hunks

Read & analyze hunks in context

…

— — +-

— — +-

Repair Evidence

…

+-

Module 3: Affected-Version Judgment Absent

Aggregated Results

Breakage LLM Judge

3 Divergence Output

Module 2: Diff Exploration 2 Search

Presence Score

Fallback Evidence Repair Evidence

2 Execution Trace Chains

Reproduced (Success)

Trace Chain

Fail

Core Dynamic Prompting Loop LLM Diff Analyst

Conference’17, July 2017, Washington, DC, USA

Neutral

2 Version-Chain Backfill v1

v2

v3

Flip Check v4 v5

Failure Reason Verdict

3 Final Output v6

Affected Version Range

v4 – v6 Not Affected Inconclusive

Affected

Figure 3: The overall framework of Attain.

Role: Locate diff hunks that explain exploit failure from trace divergence signals. Input Context: • Regression Scenario: [scenario_type] on [artifact], comparing 𝑣𝑏 with 𝑣𝑡 . • Execution Observations: exploit outcome on 𝑣𝑏 and 𝑣𝑡 . • Available Evidence: version diffs, trace summary, failure excerpt, dependency tree. Task: Search the version diff space to identify code changes explaining the observed trace divergence. Prioritize concrete code diffs; use auxiliary evidence only when no code change matches. Output Requirements: • Diagnosis: explanation of the exploit failure. • Selected Hunks: specific diff excerpts explaining the divergence. • Supporting Evidence: auxiliary observations (trace, dependencies). • Open Questions: remaining uncertainties.

Figure 4: The Prompt for Diff Exploration

(Figure 4) to recover 𝐸𝑖concrete for 𝐶𝑖 ; when direct evidence is unavailable, the procedure falls back to 𝐸𝑖fallback and may subsequently repair derive 𝐸𝑖 . 3.2.1 Role. The LLM plays the role of a dependency version regression analyst. Its core task is to recover 𝐸𝑖concrete from 𝐷 (𝑣𝑏 , 𝑣𝑡 ) for each 𝐶𝑖 . These retained hunks form a minimal set of high-signal inputs for the next step of vulnerability presence judgment. The specific responsibilities are: • Understanding the behavioral signal represented by 𝛿𝑖 and the associated search state 𝑠𝑖 in 𝐶𝑖 , including compilation failure and runtime exceptions.

• Locating candidate hunks ℎ ∈ 𝐷 and organizing the retained evidence as 𝐸𝑖concrete or 𝐸𝑖fallback . • Telling apart evidence that the vulnerability is absent from exploit breakage evidence within the retained evidence set. Evidence that the vulnerability is absent means changes that remove or block the vulnerable path. Exploit breakage evidence means changes that only break the exploit harness but do not affect whether the vulnerability is still present. 3.2.2 Goals. The goals of the dynamic prompting stage are as follows. • Recover 𝐸𝑖concrete for each 𝐶𝑖 by identifying the hunks in 𝐷 (𝑣𝑏 , 𝑣𝑡 ) that best explain 𝛿𝑖 . • Look at API-level changes first, including method signature changes and class removals. • When API diffs are too shallow and contain only class signatures, field descriptors, or other declaration-only changes, go deeper into bytecode-level diffs to see the method-body context. • When no direct hunk can be retained in 𝐸𝑖concrete , allow citing supplementary evidence in 𝐸𝑖fallback . This evidence must be clearly marked as fallback evidence and cannot alone prove that the vulnerability is absent. 3.2.3 Available tools. The LLM can use the following capabilities to recover evidence from 𝐷 (𝑣𝑏 , 𝑣𝑡 ). • Keyword search. This capability narrows the candidate space by retrieving file-line pairs associated with keywords derived from 𝛿𝑖 and 𝑠𝑖 . It returns a ranked list of candidates that helps identify promising hunks ℎ ∈ 𝐷. • Read file. This capability inspects the local content of a candidate hunk ℎ within a narrow window, allowing the model to verify whether the candidate should be retained in 𝐸𝑖concrete .

Conference’17, July 2017, Washington, DC, USA

• Load diff context. This capability resolves a candidate diff hunk ℎ to its surrounding reference and target code context, enabling the model to assess whether the change belongs to 𝐸𝑖concrete . • Search code context. This capability recovers broader program structure relevant to 𝐶𝑖 , such as class-level or method-level context, when the local diff fragment is insufficient. 3.2.4 State. The core of tool-augmented dynamic prompting is a finite-state iterative loop. At each round, the LLM decides its next action based on the current state. The state transitions work over the current 𝐶𝑖 as follows. • Determine collection target. The input is the scenario type, failure indicators, and trace keywords. The LLM reads the overview of the version diff and the failure symptoms, then initializes the search state 𝑠𝑖 in 𝐶𝑖 with target classes, methods, or exception symbols as starting points. • Construct search query. The LLM uses the current 𝑠𝑖 together with 𝛿𝑖 -based keywords and failure-excerpt keywords. It then performs keyword-based retrieval over the analysis workspace to identify candidate hunks ℎ ∈ 𝐷, producing a ranked list of candidate diff paths by relevance score. • Reflect. The LLM examines the current 𝐶𝑖 , including the retrieved candidates and any retained evidence. It makes three checks: Is the current evidence enough to explain 𝛿𝑖 ? If yes, go to the next step. Are the API diffs too shallow and only show declarations? If so, it retrieves expanded diff context to refine 𝐸𝑖concrete or gathers broader code context to recover the surrounding program structure. Are some important classes missing? If so, update 𝑠𝑖 and go back to construct search query. During reflection, the LLM must respect the following constraints: • The output must contain exactly four sections: Diagnosis, Selected Hunks, Supporting Evidence, and Open Questions. • Declaration-only changes in API diffs must not be used as sufficient evidence to finalize the answer. • Constructor, collection, map, and self-reference style vulnerabilities must be expanded to the complete changed method or constructor context. • Lower priority should be given to logger-only diffs, anonymous inner-class diffs, and enum-reordering diffs. Update target and query. If reflection finds that the retained evidence in 𝐶𝑖 is not enough, the LLM extracts new symbols such as class names and method names from the current diff content, updates 𝑠𝑖 , and merges new keywords from 𝛿𝑖 . It then returns to construct search query with the updated 𝑠𝑖 . If reflection finds that 𝐸𝑖concrete is sufficient, the LLM outputs the final answer as a structured report comprising four sections Diagnosis, Selected Hunks, Supporting Evidence, and Open Questions. If the Selected Hunks section has no valid evidence path, the sysrepair tem derives 𝐸𝑖 . This ensures the output always contains citable evidence for subsequent judgment. The iterative loop stops when one of these conditions is met: • The maximum tool-call round budget is reached. The budget changes based on scenario type. Compile failure is limited to one round. Process failure is limited to two rounds. Trace diff allows more rounds for deeper exploration.

Xinwei Mao, Zirui Chen, Xing Hu, and Xin Xia

Role: Classify individual evidence items collected during diff exploration. Input Context: • Vulnerability: [CVE description] in [artifact], comparing 𝑣𝑏 with 𝑣𝑡 • Evidence Item: a diff hunk, trace summary, or failure excerpt. Task: Classify whether the item indicates one of: • Fix: the change removes or blocks the vulnerable path. • Exploit Breakage: the change breaks the exploit but does not remove the vulnerability. • Neutral: insufficient information to determine. Output Requirements: • judgment: fix | exploit breakage | neutral • confidence: high | medium | low • reason: short justification for the classification.

Figure 5: The Prompt for Classfying Evidence Role: Determine whether a vulnerability persists across library versions. Input Context: • Vulnerability: [CVE description] in [artifact], comparing 𝑣𝑏 with 𝑣𝑡 • Evidence: typed evidence items from the prior stage. Task: Decide whether the vulnerability still exists in 𝑣𝑡 . Apply a logicfirst interpretation: direct removal of the vulnerable path counts as a fix, not only explicit validation. Output Requirements: • score: 0-100 (0 = absent, 100 = persists) • reason: vulnerability not present | exploit invalid | undetermined. • verdict: no | possibly yes | unknown • confidence: high | medium | low

Figure 6: The Prompt for Case-Level Vulnerability Judgment • The LLM stops calling tools and directly produces the final structured response. repair • 𝐸𝑖 has been derived. repair

Once 𝐸𝑖concrete or 𝐸𝑖 has been derived, the evidence set is passed to the next stage for vulnerability presence judgment.

3.3

Affected-Version Judgment Module

Finally, the affected-version judgment module reasons over the collected evidence to determine whether the vulnerability exists in each version and outputs the affected version range. It consists of a case-level vulnerability judgment module and a version-chain backfill module. 3.3.1 Case-Level Vulnerability Judgment. After the dynamic prompting stage constructs 𝐶𝑖 and recovers its retained evidence, the system needs to judge whether the vulnerability still exists in the target version and why the exploit failed. This process has three modules: single-hunk evidence typing, case-level vulnerability judgment, and rule-based label compression. Each retained hunk ℎ in 𝐸𝑖concrete is individually examined by an LLM (Figure 5); when direct evidence is unavailable, the judgment repair instead relies on the derived set 𝐸𝑖 . The LLM assigns one of three

ATTAIN: Automated Exploit Failure Analysis through Trace-Driven Diff Analysis

evidence types: evidence that the vulnerability is absent, meaning the code change removes or blocks the vulnerable path; exploit breakage evidence, meaning the code change breaks the exploit harness but does not imply that the underlying vulnerability is absent; or neutral, meaning the evidence is not sufficient to classify. The LLM also outputs a confidence level and a short reasoning chain for each ℎ, which serve as input to the subsequent case-level judgment. After all retained hunks have been typed, a second LLM call (Figure 6) combines them into a case-level judgment. The LLM receives the typed evidence set together with the corresponding 𝐶𝑖 . It outputs a vulnerability presence score in [0, 100], a failure reason label (vulnerability not present, exploit invalid due to version change, or inconclusive), and a vulnerability presence verdict (no, possibly yes, or inconclusive). It also outputs separate strength ratings for evidence that the vulnerability is not present and for exploit breakage evidence, giving a complete picture. The case-level LLM outputs are further compressed into a binary label: affected or not affected. This compression uses a rule-based procedure that considers the vulnerability presence score, the verdict, the scenario type, the failure subtype, and the trace-quality signals encoded in 𝐶𝑖 . For example, when the trace has an effective difference and the LLM verdict is vulnerability not present, the rule may still change the label to not affected with low confidence if the trace evidence is valid and the first divergence is not noise. This rule module ensures that borderline cases are handled conservatively rather than being forced into a binary decision. 3.3.2 Version Chain Backfill. The previous stages produce seed labels only for the version pairs that were directly analyzed. Attain extends these labels along the version chain to cover the remaining versions. A version is eligible for label extension only when it has not yet received a label from any earlier stage. Extension is confined within a region, defined as (CVE, Library, Execution outcome, exploit). The exploit dimension isolates labels across different exploits. If multiple exploits disagree on the same version, Attain adopts a conservative safety-first strategy, defaulting to affected. Before extension starts, if a version has not yet received a label and the exploit can trigger the vulnerability on that version, the system directly assigns the affected label as a safe default. Within each region, the extension follows the semantics of each label type. Labels associated with the patch commit extend from the patch version toward later versions. Labels associated with the vulnerability introduction commit extend from the introduction version toward earlier versions. Labels that indicate the vulnerability is present extend in both directions along the chain. The most critical case occurs when an affected label is about to extend across a version boundary where the exploit execution outcome flips. This means the exploit triggers the vulnerability on one version but does not on its adjacent version. Such a flip suggests that the vulnerability behavior changes between the two versions, so simply copying the label may be wrong. In this case, the system invokes an LLM to evaluate whether the vulnerability truly persists on the other side of the flip. The LLM receives the version information and the current judgment details. It returns a confidence score between 0 and 1. If the LLM returns very high confidence that the label should not extend, the system blocks the

Conference’17, July 2017, Washington, DC, USA

extension for that version. Otherwise, the extension proceeds. This LLM-based check ensures that label extension does not override a genuine behavior change revealed by the execution outcome flip.

4

Experimental Setup

Research Questions. To evaluate the performance of our approach for cross-version vulnerability presence detection, our experiment aims to answer three research questions: • RQ1 (Effectiveness): How effective is Attain for cross-version vulnerability detection compared to existing approaches? • RQ2 (Ablation Study): How much does each component in Attain contribute to the overall detection performance? • RQ3 (CWE-Level Analysis): How does Attain perform across different CWE vulnerability categories? We address RQ1 to evaluate the overall effectiveness of our method and its advantage over existing approaches. We address RQ2 to measure how much each key component helps. These components include the trace-guided diff exploration module, the evidence and rule modules in vulnerability judgment, and the version chain backfill module. We address RQ3 to check whether Attain keeps its advantage across different CWE categories and to find out where each method has strengths or weaknesses.

4.1

Dataset

We evaluate Attain on the largest publicly available Java exploit dataset [7]. This dataset contains 259 exploits spanning 224 CVE vulnerabilities across 128 libraries in 41 categories, covering 25,943 library versions (e.g., HTTP clients, XML processors, Object Serialization), with 57 libraries ranking in the top 1,000 Maven artifacts. The vulnerabilities cover 61 CWEs (19 in the CWE Top 25) and carry an average CVSS score of 7.75 (62 Critical, 101 High). Notably, the CWEs in this dataset cover at least one CWE for 76.33% of all Maven vulnerabilities disclosed up to 2025, and all major weakness Pillars under the CWE Research Concepts view. Each CVE has a corresponding exploit and a set of manually verified affected versions, making the dataset suitable for evaluating cross-version vulnerability presence detection. To build version pairs for analysis, Attain looks at the execution results for each CVE. It finds adjacent versions where the exploit outcome changes, such as from triggering the exploit successfully to failing to trigger it. These boundary version pairs are the main analysis targets. For each pair, Attain collects execution traces on both the reference version 𝑣𝑏 and the target version 𝑣𝑡 . It also generates version diffs and runs the full pipeline to produce a vulnerability presence judgment.

4.2

Baselines

To the best of our knowledge, no prior work has directly addressed the problem of detecting vulnerability presence across Java library versions based on execution traces and version diffs. Existing studies on vulnerability version tracking rely on commit analysis or exploit reproduction alone. We include the following three baselines: • V-SZZ. This is a variant of the SZZ algorithm [46] for vulnerability version tracking [2]. V-SZZ analyzes version control history

Conference’17, July 2017, Washington, DC, USA

to find commits that fix vulnerabilities. It then labels versions based on whether they come before or after the fix commit. • LLM4SZZ. This baseline improves the traditional SZZ approach with LLM-based commit analysis [48]. It uses a large language model to decide whether a commit is a vulnerability fix. This replaces the keyword-based matching in traditional SZZ. • Exploit. This baseline uses the exploit execution result directly as the detection label. If the exploit triggers the vulnerability, the version is labeled as affected. Otherwise, the version is labeled as not affected. This baseline only relies on exploit reproduction. It does not perform any cross-version analysis.

4.3

Evaluation Metrics

Consistent with prior work, we use precision, recall, and F1-score as the main evaluation metrics. Each method produces a label for each version: affected or not affected. Versions that a method cannot confidently classify are treated as not affected in the main evaluation. This follows the convention that unconfirmed versions are considered safe. For Attain, fine-grained labels such as affected at the introduction commit and affected at the patch commit are unified into binary labels. A label indicating vulnerability presence is mapped to affected; otherwise, not affected. The ground truth comes from the dataset [7], which identifies the fixing and inducing commits and manually verifies each version’s code. A predicted label is correct if it matches the ground truth. For the CWE-level analysis in RQ3, we group all CVEs by their primary CWE tag. We take the top 10 most frequent CWE categories. The rest are grouped into an others category. For each CWE group and each method, we compute precision, recall, and F1 separately. This produces a comparison across vulnerability types.

4.4

Implementation Details

All experiments run on Ubuntu 20.04.6 LTS. We use Java 11 as the runtime, following the same configuration as Wu et al. [53]. We run exploits with Apache Maven for dependency resolution. To collect execution traces, we inject a lightweight Java agent at runtime that records method-level traces. For LLM selection, we use DeepSeek-V3 (snapshot 0324) as the primary model. We choose this model because it is open-source, cost-effective, and has demonstrated strong code understanding capabilities. For comparison, running all 224 cases with a proprietary model such as GPT-5.4 would cost approximately $114, whereas DeepSeek-V3 completes the same analysis for roughly $7. This model handles the dynamic prompting module, the single-hunk evidence typing, and the case-level vulnerability judgment respectively. All LLM calls go through the standard OpenAI-compatible API interface with a 180-second timeout. DeepSeek-V3 was used with temperature 0.5/0.2/0.0 across different stages, while context length followed provider defaults and prompt size was controlled via application-level truncation.

5

Experimental Evaluation

We evaluate the performance of Attain from three perspectives. First, we assess its effectiveness and compare it with baseline methods on the largest dataset of Java library vulnerability exploits, and

Xinwei Mao, Zirui Chen, Xing Hu, and Xin Xia

analyze its strengths and limitations. Second, we conduct an ablation study to demonstrate the contributions of each component to the overall performance. Finally, we analyze the effectiveness of Attain across different CWE types.

5.1

Effectiveness

5.1.1 Performance. We evaluate the effectiveness of Attain on a dataset containing 25,943 library versions, of which 13,109 are vulnerability-affected. As shown in Table 1, Attain achieves an F1score of 93.24%, outperforming all baselines. In particular, Attain identifies 11,741 true positives, which is 699 more than the strongest baseline Exploit (11,042). This corresponds to an improvement of 5.33% in recall, demonstrating the ability of Attain to detect affected versions that exploit execution alone misses. Table 1: Comparison of Different Approaches Tool V-SZZ LLM4SZZ Exploit Attain

TP

FP

FN

Pre. (%)

Rec.(%)

F1 (%)

4,482 7,815 11,042 11,741

3,200 1,419 89 335

8,627 5,294 2,067 1,368

58.34 84.63 99.20 97.23

34.19 59.62 84.23 89.56

43.11 69.95 91.11 93.24

Compared to Exploit, Attain achieves a relative improvement of 2.34% in F1-score, demonstrating its robustness in handling cases where the exploit fails for reasons unrelated to the vulnerability. Exploit achieves near-perfect precision (99.20%) because it only labels a version as affected when the exploit actually triggers. However, its recall is limited to 84.23%: when the exploit fails to run on a target version due to API incompatibility or runtime environment changes, Exploit defaults to not affected, missing genuinely affected versions. Attain overcomes this limitation by analyzing the root cause of exploit failure through trace-diff exploration and vulnerability judgment, correctly identifying cases where the vulnerability persists despite exploit breakage. The commit-based approaches show weaker performance. VSZZ achieves only 43.11% in F1-score, with both low precision (58.34%) and low recall (34.19%). Its low precision indicates many false positives: V-SZZ marks versions as affected based on commit ancestry, but not every version before a fix commit is actually vulnerable. Its low recall indicates that it misses many affected versions, likely because the introducing commit is not correctly identified or because multiple introducing commits exist. LLM4SZZ improves over V-SZZ by using LLM-based commit analysis, raising F1-score to 69.95% and precision to 84.63%. This suggests that LLM-based commit screening reduces false introducing-commit identification. However, its recall remains low at 59.62%, indicating that commit-based methods still inherently struggle to cover all affected versions comprehensively. 5.1.2 Strength. Compared to Exploit, Attain demonstrates superior adaptability in handling cases where exploit execution fails for non-vulnerability reasons. When a exploit triggers an exception such as NoSuchMethodError or ClassNotFoundException on the target version, Exploit labels the version as not affected, as the exploit did not successfully trigger. However, such failures are often

ATTAIN: Automated Exploit Failure Analysis through Trace-Driven Diff Analysis

caused by API-level changes that break the exploit harness rather than actual vulnerability fixes. Attain leverages trace-diff analysis to identify the specific code change that causes the exploit failure, and then determines whether the change removes the vulnerable path or merely breaks the exploit harness. This enables Attain to correctly label affected versions that Exploit misses. Additionally, Attain leverages a structured evidence collection process that captures both fix evidence and exploit breakage evidence from version diffs. This design enables it to accommodate a broader range of code modifications related to vulnerability presence. This advantage becomes more prominent in complex vulnerability scenarios, such as deserialization and path traversal, where the fix involves subtle semantic changes that are difficult to detect from commit messages alone. 5.1.3 Limitations. While Attain demonstrates strong performance in detecting vulnerability presence across library versions, it has several limitations. First, its recall on certain CWE types remains relatively low, particularly CWE-79 (64.24%) and CWE-94 (68.18%). These vulnerability types involve dynamic code generation and string manipulation, where the execution trace may not directly reveal the vulnerability trigger path. When the trace divergence does not align with the actual fix location, Attain may fail to collect sufficient evidence for a correct judgment. Second, Attain encounters difficulties when the version diff contains a large number of changes that are unrelated to the vulnerability. In such cases, even with trace-diff guidance, the LLM may select evidence that appears relevant but does not accurately explain the trace divergence. This can lead to false positives when the selected evidence is incorrectly classified as exploit breakage rather than neutral.

5.2

Ablation Study

Our ablation study aims to achieve two goals: (1) to demonstrate that each component in our design contributes to the overall detection performance, and (2) to analyze how each component affects precision and recall differently. We construct four ablated variants of Attain: • Attain-tracediff (trace-free): We remove the trace-diff summary and the version-diff overview from the initial prompt. The LLM only sees the failure excerpt from the target version. It must search for relevant diffs on its own. This ablation measures how much the trace-guided context helps. • Attain-diffsearch (random evidence): We replace the LLMselected evidence with randomly sampled diffs from the same version-diff space. This measures how much precise diff retrieval contributes. • Attain-aggregation (no rule): We remove the rule-based label compression and use only the raw LLM judgment with simple threshold mapping. This measures how much the rule module contributes. • Attain-backtrack (no backfill): We disable the version chain backfill module. Only the seed labels from the first two modules are used for evaluation. No forward or backward extension is performed along the version chain. This ablation measures how much the full-coverage backfill contributes.

Conference’17, July 2017, Washington, DC, USA

All ablation variants use the same dataset, ground truth, and evaluation metrics as the full Attain pipeline. This ensures a fair comparison. Table 2: Ablation Study on Attain Tool Attain Attain-tracediff Attain-diffsearch Attain-aggregation Attain-backtrack

TP

FP

FN

Pre.(%)

Rec.(%)

F1(%)

11,741 11,087 11,208 11,433 11,066

335 373 719 320 104

1,368 2,022 1,901 1,676 2,043

97.23 96.75 93.97 97.28 99.07

89.56 84.58 85.50 87.21 84.42

93.24 90.25 89.54 91.97 91.16

We observe from Table 2 that Attain achieves the highest F1score, outperforming the best ablated variant by 1.27%. This confirms that each component in our design contributes to the overall performance. Among the variants, removing the dynamic diff search causes the largest F1 drop of 3.70%, followed by trace-diff guidance at 2.99%, version chain backfill at 2.08%, and rule-based aggregation at 1.27%. Although the success rates degrade when removing any single module, they all remain notably higher than the commitbased baselines (V-SZZ: 43.11%, LLM4SZZ: 69.95%), indicating that the core pipeline of Attain brings substantial benefits even without any single component. Attain-tracediff removes the trace-diff summary and the versiondiff overview from the initial prompt. Without trace-diff guidance, the LLM must start its search from the failure excerpt alone, making it harder to locate the correct diff hunks that explain the trace divergence. This results in the largest recall drop of 4.98% among the first three variants, with 654 additional false negatives. The precision only drops slightly by 0.48%, suggesting that the LLM can still make accurate judgments when it finds the right evidence, but it frequently fails to find it without the trace-diff context. Beyond quantifying component importance, this variant also serves as a control against potential data leakage. Since the LLM was pre-trained on large-scale public data including CVE databases and GitHub repositories, it may have encountered some of the evaluated CVEs during pre-training. However, Attain-tracediff retains the CVE identifier in its prompt while removing trace-diff evidence; a model relying on memorized CVE-version associations would show minimal degradation under this condition. The substantial F1 and recall drops indicate that Attain genuinely depends on behavioral evidence rather than memorized CVE knowledge. The Attain-diffsearch variant reinforces this conclusion: replacing LLM-selected evidence with random diffs causes the largest F1 drop of 3.70%, confirming that precise evidence retrieval matters beyond what the model could infer from the CVE identifier alone. Attain-aggregation removes the rule-based label compression. The recall drops by 2.35% while precision increases only slightly by 0.05%. This is because the rule module helps recover borderline cases that the LLM alone classifies with low confidence, by considering additional signals such as trace quality, scenario type, and failure subtype that the raw LLM score alone does not capture. Attain-backtrack disables the version chain backfill module. The recall drops by 5.14%, the largest recall drop among all variants, while precision rises to 99.07%. This trade-off is expected: the backfill module extends seed labels to uncovered versions along the

Conference’17, July 2017, Washington, DC, USA

Xinwei Mao, Zirui Chen, Xing Hu, and Xin Xia

Table 3: Results of Our Effectiveness Evaluation w.r.t CWE Types (#V. denotes the number of vulnerabilities of a CWE type) Tool

Metric

CWE-502 #V.=8,082

CWE-22 #V.=2,300

CWE-611 #V.=1,687

CWE-787 #V.=1,465

CWE-770 #V.=1,094

CWE-79 #V.=1,013

CWE-20 #V.=773

CWE-400 #V.=526

CWE-94 #V.=499

CWE-835 #V.=488

OTHERS #V.=8,783

V-SZZ

Pre.(%) Rec.(%) F1(%)

65.67 11.33 19.33

35.54 61.92 45.16

96.34 17.73 29.95

73.82 24.12 36.36

93.95 38.32 54.44

34.29 55.63 42.42

47.54 20.71 28.86

41.67 37.69 39.58

7.69 11.93 9.35

64.62 87.26 74.25

65.74 58.58 61.95

LLM4SZZ

Pre.(%) Rec.(%) F1(%)

92.90 57.84 71.30

70.54 76.56 73.42

96.93 85.19 90.68

97.73 48.87 65.15

79.32 42.27 55.15

96.74 58.94 73.25

75.35 57.86 65.45

70.66 59.30 64.48

100.00 72.16 83.83

76.37 88.54 82.01

75.91 57.53 65.45

Exploit

Pre.(%) Rec.(%) F1(%)

99.52 90.32 94.70

100.00 84.55 91.63

100.00 86.08 92.52

98.16 93.51 95.78

100.00 73.68 84.85

89.39 52.98 66.53

100.00 80.71 89.33

98.41 93.47 95.88

100.00 57.95 73.38

100.00 87.90 93.56

99.15 78.80 87.81

Attain

Pre.(%) Rec.(%) F1(%)

99.18 92.59 95.77

93.50 87.67 90.49

99.19 95.85 97.49

98.16 93.51 95.78

100.00 75.66 86.14

90.65 64.24 75.19

94.96 80.71 87.26

92.54 93.47 93.00

100.00 68.18 81.08

100.00 93.63 96.71

94.85 88.71 91.68

version chain, increasing coverage at the cost of a small number of over-extensions. The 231 additional false positives are modest compared to the 675 recovered true positives. The high precision of the seed labels before backfill (99.07%) also validates the quality of the first two modules’ output.

commit analysis is particularly effective for code injection vulnerabilities where fix commits have clear semantic signals. However, LLM4SZZ struggles with CWE-770 (F1 = 55.15%) and CWE-787 (F1 = 65.15%), where the relationship between commits and vulnerability presence is less direct.

5.3

This section reflects on what we have learned from the study. We first highlight key success cases, including comparisons with exploit-only baselines and a patch-based approach. We then summarize typical failure cases and their root causes. After that, we discuss the cost of using large language models. Finally, we outline the main threats to the validity of our findings.

6 CWE-Level Analysis

Table 3 shows the per-CWE results for the top 10 CWE categories and an others group. Attain outperforms or matches all baselines in F1-score on 7 out of 11 CWE groups. The largest improvements over Exploit appear in CWE types where exploit execution frequently fails for non-vulnerability reasons. For CWE-94 (Code Injection), Attain achieves F1 of 81.08% compared to 73.38% for Exploit, a gain of 7.70%. This is driven by a recall improvement from 57.95% to 68.18%. Code injection vulnerabilities often involve complex trigger paths that break across versions due to API changes rather than actual fixes. The trace-diff analysis of Attain can identify such cases and correctly label them as affected. For CWE-79 (XSS), Attain improves F1 from 66.53% to 75.19%, with recall rising from 52.98% to 64.24%. XSS exploits are sensitive to minor changes in output encoding or input validation, which often break the exploit harness without fixing the underlying vulnerability. The largest improvement over commit-based methods appears in CWE-502 (Deserialization), the largest CWE group with 8,082 vulnerabilities. Attain achieves F1 of 95.77%, compared to 71.30% for LLM4SZZ and 19.33% for V-SZZ. Deserialization vulnerabilities often span multiple library versions with subtle changes in serialization logic. Commit-based methods struggle because the relevant fix commits are not easily identifiable from commit messages alone. VSZZ achieves particularly low precision (65.67%) and recall (11.33%) on CWE-502, indicating that both fix-commit identification and version coverage are problematic for this vulnerability type. The CWE-level results also reveal a clear divide between commitbased methods and execution-based methods. V-SZZ performs poorly on most CWE types, with F1 below 50.00% on 8 out of 11 groups. Its lowest F1 is 9.35% on CWE-94, where commit analysis fails to identify code injection fixes. LLM4SZZ improves over V-SZZ but still lags behind execution-based methods on most CWE types. Interestingly, LLM4SZZ achieves competitive F1 on CWE-94 (83.83%) compared to Exploit (73.38%), suggesting that LLM-based

6.1

Discussion

Success Cases

Attain achieves an F1-score of 93.24% across 25,943 versions, outperforming commit-based and exploit-only baselines. The 2.34% F1 gain over the exploit-only baseline translates to 620 additional affected versions identified—versions where the exploit fails to trigger but the vulnerability persists. While GitHub Advisory Database already covers 92.6% of these, 46 versions across five CVEs remain unlisted in any advisory. For instance, CVE-2022-25845 (Fastjson) affects versions from 1.2.10 to 1.2.24, yet GitHub Advisory omits 34 early versions in this range. The exploit baseline misses them because the exploit fails to build on these versions due to API changes; Attain correctly recovers them through trace-diff analysis. On the 82 CVEs common with VISION [53] (12,041 versions), Attain attains 93.27% F1 versus VISION’s 85.75%, as Attain anchors analysis on exploit behavior rather than patch structure, avoiding over-approximation when introducing and fixing commits are structurally distant.

6.2

Qualitative Failure Analysis

Although the overall performance is strong, we also find clear limits. We group the most common failure patterns into three types. These patterns help us understand where Attain still needs improvement. Fragile test and environment. Failures in the trace construction module (Module 1) arise when the exploit cannot compile or run on the target version, leaving Attain with little to no execution trace. For example, in CVE-2013-7285 (XStream), the exploit fails to build on version 1.0 due to API incompatibility, so Attain

ATTAIN: Automated Exploit Failure Analysis through Trace-Driven Diff Analysis

has no behavioral signal and remains unlabeled across many early versions. These cases highlight that the current pipeline does not repair build or dependency issues to recover trace signals. Misinterpretation of evidence. Failures in the evidence typing and case-level judgment modules (Module 2) occur when the LLM correctly locates relevant code changes but draws the wrong conclusion. For instance, in CVE-2024-1597 (PostgreSQL JDBC), Attain identifies code differences and confidently judges the vulnerability as fixed, when in reality the affected logic persists under a different execution path. Such cases show that the model can still misattribute changes as a complete vulnerability removal. Gaps along version chains. Failures in the version chain backfill module (Module 3) arise from incomplete coverage. For example, CVE-2022-25845 (Fastjson) has 110 versions where Attain produces no label because the seed labels from difftrace never reached these boundary versions, and the backfill rules conservatively avoid propagating labels across execution-outcome boundaries. This trade-off between completeness and caution is inherent to the region-based propagation design.

6.3

Threats to Validity

The main threats to validity come from the nature of version-level vulnerability analysis in Attain. External validity. Our study focuses on Java library vulnerabilities with runnable exploits, recoverable releases, and accessible version histories. Although this setting is broad, it does not cover many real-world cases, such as closed-source software, deployed applications, or projects with missing history. The reliance on publicly available exploits also limits the scope to vulnerabilities for which a working PoC exists. Therefore, our findings may not directly generalize to all software ecosystems. Internal validity. Another threat lies in how we decide whether a version is affected. This is not always directly visible: a version may still contain the bug even if one exploit no longer works, and the result of an exploit may change with configuration, dependencies, or runtime environment. In addition, datasets usually provide only a small number of exploits and limited historical evidence. The LLM judgments may also vary across different prompts or model versions. As a result, some affected-version labels may still be uncertain. This problem is common in version-level vulnerability studies and is not specific to our approach.

7

Related Work

Exploit-Based Analysis. Exploit migration adapts existing exploits to different software versions for vulnerability assessment and reproduction. AEM [21] aligns execution points across Linux kernel versions to reproduce exploitation behaviors. VulScope [11] leverages directed fuzzing to migrate exploits between versions. SyzBridge [65] addresses environmental differences between upstream and downstream kernels. Evocatio [20] automatically generates exploits to expose unknown bug capabilities. These approaches rely on fuzzing and explicit execution trace mapping, which is timeconsuming and challenging. Dai et al. utilized exploit migration with directed fuzzing to identify affected versions, but the vulnerability may lack a exploit [13, 35]. Moreover, existing exploit-based

Conference’17, July 2017, Washington, DC, USA

approaches treat execution failure as evidence that the vulnerability is absent, without distinguishing genuine fixes from exploit breakages caused by environmental changes. Exploit migration also shares similarities with test migration [1, 16, 28, 42–44] and API migration [10, 12], which can partially address exploit failures caused by API changes but typically focus on function-level repair and overlook environmental factors. Affected Version Identification. Various approaches have been proposed to identify library versions affected by an OSS vulnerability [2, 13, 19, 36, 37, 45, 54]. Report-based methods such as NER [13] extract version information from vulnerability reports, but are limited by report quality and completeness [13, 35, 37]. SZZbased approaches trace vulnerability-introducing changes through patch analysis: the original SZZ [46] and its improvement V-SZZ [2] backtrack deleted lines in patches to identify vulnerability-inducing versions, but fail when patches only contain added lines. Patchbased approaches [19, 45] identify affected versions by measuring patch line presence in target versions or leveraging developer logs, but they ignore the context of modified lines and require manual verification. Vision [53] captures patch context by encoding critical methods and statements into weighted inter-procedural dependency graph signatures. VFC analysis methods [29–31] extract vulnerability features from fixing commits, but rely heavily on predefined rules. VERCATION [8] leverages LLMs and AST-based code clone detection to improve vulnerability characterization. Vulnerable code clone detection approaches such as VUDDY [23], MVP [54], MOVERY [51], V1SCAN [50], V0Finder [52], and VCCFinder [41] generate syntactic or semantic signatures to detect recurring vulnerabilities, but these signatures often contain irrelevant code.

8

Conclusion

In this work, we presented Attain, a trace-driven diff analysis framework for automated exploit failure analysis across evolving library versions. It comprises three modules: trace construction builds version context from exploit executions, diff exploration uses a tool-augmented LLM loop to collect vulnerability evidence from diffs, and affected-version judgment applies version-chain backfill to label versions. On 224 CVEs and 25,943 versions from 128 Java libraries, Attain achieves an F1-score of 93.24%, outperforming V-SZZ and LLM4SZZ by 116.28% and 33.30% respectively, while keeping token costs low via short trace-guided prompts and bounded iterations. Ablation and CWE analyses show that tracediff guidance, dynamic diff search, rule-based aggregation, and version-chain backfill all contribute to effectiveness, particularly when exploits fail for non-vulnerability reasons or commits do not clearly delimit affected versions. Attain still faces challenges on vulnerability types with weak or noisy trace signals and in tangled diffs. Future work may explore richer trace collection, stronger robustness to unrelated changes, and broader application beyond Java. Overall, Attain makes exploit-based vulnerability assessment more reliable across versions.

Acknowledgement This research is supported by the Fundamental Research Funds for the Central Universities (No. 226-2025-00171). We also thank the anonymous reviewers for their insightful suggestions.

Conference’17, July 2017, Washington, DC, USA

References [1] Nadia Alshahwan, Jubin Chheda, Anastasia Finogenova, Beliz Gokkaya, Mark Harman, Inna Harper, Alexandru Marginean, Shubho Sengupta, and Eddy Wang. 2024. Automated Unit Test Improvement using Large Language Models at Meta. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering (Porto de Galinhas, Brazil) (FSE 2024). Association for Computing Machinery, New York, NY, USA, 185–196. doi:10.1145/3663529.3663839 [2] Lingfeng Bao, Xin Xia, Ahmed E. Hassan, and Xiaohu Yang. 2022. V-SZZ: automatic identification of version ranges affected by CVE vulnerabilities. In Proceedings of the 44th International Conference on Software Engineering (Pittsburgh, Pennsylvania) (ICSE ’22). Association for Computing Machinery, New York, NY, USA, 2352–2364. doi:10.1145/3510003.3510113 [3] Gabriele Bavota, Gerardo Canfora, Massimiliano Di Penta, Rocco Oliveto, and Sebastiano Panichella. 2015. How the apache community upgrades dependencies: an evolutionary study. Empirical Software Engineering 20 (2015), 1275–1317. [4] Zirui Chen, Xing Hu, Puhua Sun, Xin Xia, and Xiaohu Yang. 2025. Generating Mitigations for Downstream Projects to Neutralize Upstream Library Vulnerability. arXiv:2503.24273 [cs.SE] https://arxiv.org/abs/2503.24273 [5] Zirui Chen, Xing Hu, Xin Xia, Yi Gao, Tongtong Xu, David Lo, and Xiaohu Yang. 2024. Exploiting Library Vulnerability via Migration Based Automating Test Generation. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (Lisbon, Portugal) (ICSE ’24). Association for Computing Machinery, New York, NY, USA, Article 228, 12 pages. doi:10.1145/3597503. 3639583 [6] Zirui Chen, Zhipeng Xue, Jiayuan Zhou, Xing Hu, Xin Xia, and Xiaohu Yang. 2025. Diffploit: Facilitating Cross-Version Exploit Migration for Open Source Library Vulnerabilities. arXiv:2511.12950 [cs.SE] https://arxiv.org/abs/2511.12950 [7] Zirui Chen, Qi Zhan, Jiayuan Zhou, Xing Hu, Xin Xia, and Xiaohu Yang. 2026. A Large-scale Empirical Study on the Generalizability of Disclosed Java Library Vulnerability Exploits. arXiv:2603.25997 [cs.SE] https://arxiv.org/abs/2603.25997 [8] Yiran Cheng, Ting Zhang, Lwin Khin Shar, Shouguo Yang, Chaopeng Dong, David Lo, Shichao Lv, Zhiqiang Shi, and Limin Sun. 2025. VERCATION: Precise Vulnerable Open-source Software Version Identification based on Static Analysis and LLM. IEEE Transactions on Software Engineering (2025). [9] Roland Croft, M. Ali Babar, and M. Mehdi Kholoosi. 2023. Data Quality for Software Vulnerability Datasets. In Proceedings of the 45th International Conference on Software Engineering (Melbourne, Victoria, Australia) (ICSE ’23). IEEE Press, 121–133. doi:10.1109/ICSE48619.2023.00022 [10] Barthélémy Dagenais and Martin P Robillard. 2011. Recommending adaptive changes for framework evolution. ACM Transactions on Software Engineering and Methodology (TOSEM) 20, 4 (2011), 1–35. [11] Jiarun Dai, Yuan Zhang, Hailong Xu, Haiming Lyu, Zicheng Wu, Xinyu Xing, and Min Yang. 2021. Facilitating Vulnerability Assessment through PoC Migration. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security (Virtual Event, Republic of Korea) (CCS ’21). Association for Computing Machinery, New York, NY, USA, 3300–3317. doi:10.1145/3460120.3484594 [12] Danny Dig and Ralph Johnson. 2006. How do APIs evolve? A story of refactoring. Journal of software maintenance and evolution: Research and Practice 18, 2 (2006), 83–107. [13] Ying Dong, Wenbo Guo, Yueqi Chen, Xinyu Xing, Yuqing Zhang, and Gang Wang. 2019. Towards the detection of inconsistencies in public security vulnerability reports. In Proceedings of the 28th USENIX Conference on Security Symposium (Santa Clara, CA, USA) (SEC’19). USENIX Association, USA, 869–885. [14] Yi Gao, Xing Hu, Tongtong Xu, Jiali Zhao, Xiaohu Yang, and Xin Xia. 2026. DepRadar: Agentic Coordination for Context Aware Defect Impact Analysis in Deep Learning Libraries. arXiv:2601.09440 [cs.SE] https://arxiv.org/abs/2601. 09440 [15] Yi Gao, Xing Hu, Xiaohu Yang, and Xin Xia. 2025. Automated unit test refactoring. Proceedings of the ACM on Software Engineering 2, FSE (2025), 713–733. [16] Luca Gazzola, Daniela Micucci, and Leonardo Mariani. 2018. Automatic software repair: A survey. In Proceedings of the 40th International Conference on Software Engineering. 1219–1219. [17] Konstantin Grotov, Sergey Titov, Yaroslav Zharov, and Timofey Bryksin. 2024. Untangling Knots: Leveraging LLM for Error Resolution in Computational Notebooks. arXiv:2405.01559 [cs.SE] https://arxiv.org/abs/2405.01559 [18] Runzhi He, Hao He, Yuxia Zhang, and Minghui Zhou. 2023. Automating Dependency Updates in Practice: An Exploratory Study on GitHub Dependabot. IEEE Trans. Softw. Eng. 49, 8 (Aug. 2023), 4004–4022. doi:10.1109/TSE.2023.3278129 [19] Yongzhong He, Yiming Wang, Sencun Zhu, Wei Wang, Yunjia Zhang, Qiang Li, and Aimin Yu. 2024. Automatically Identifying CVE Affected Versions With Patches and Developer Logs. IEEE Transactions on Dependable and Secure Computing 21, 2 (2024), 905–919. doi:10.1109/TDSC.2023.3264567 [20] Zhiyuan Jiang, Shuitao Gan, Adrian Herrera, Flavio Toffalini, Lucio Romerio, Chaojing Tang, Manuel Egele, Chao Zhang, and Mathias Payer. 2022. Evocatio: Conjuring Bug Capabilities from a Single PoC. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security (Los Angeles, CA,

Xinwei Mao, Zirui Chen, Xing Hu, and Xin Xia

USA) (CCS ’22). Association for Computing Machinery, New York, NY, USA, 1599–1613. doi:10.1145/3548606.3560575 [21] Zheyue Jiang, Yuan Zhang, Jun Xu, Xinqian Sun, Zhuang Liu, and Min Yang. 2023. AEM: Facilitating Cross-Version Exploitability Assessment of Linux Kernel Vulnerabilities. In 2023 IEEE Symposium on Security and Privacy (SP). 2122–2137. doi:10.1109/SP46215.2023.10179286 [22] Hyeonseong Jo, Jinwoo Kim, Phillip Porras, Vinod Yegneswaran, and Seungwon Shin. 2021. GapFinder: Finding Inconsistency of Security Information From Unstructured Text. IEEE Transactions on Information Forensics and Security 16 (2021), 86–99. doi:10.1109/TIFS.2020.3003570 [23] Seulbae Kim, Seunghoon Woo, Heejo Lee, and Hakjoo Oh. 2017. Vuddy: A scalable approach for vulnerable code clone discovery. In 2017 IEEE symposium on security and privacy (SP). IEEE, 595–614. [24] Raula Gaikovina Kula, Daniel M German, Ali Ouni, Takashi Ishio, and Katsuro Inoue. 2018. Do developers update their library dependencies? An empirical study on the impact of security advisories on library migration. Empirical Software Engineering 23 (2018), 384–417. [25] Raula Gaikovina Kula, Daniel M German, Ali Ouni, Takashi Ishio, and Katsuro Inoue. 2018. Do developers update their library dependencies? An empirical study on the impact of security advisories on library migration. Empirical Software Engineering 23 (2018), 384–417. [26] Jia Li, Zhuo Li, Huangzhao Zhang, Ge Li, Zhi Jin, Xing Hu, and Xin Xia. 2022. Poison Attack and Defense on Deep Source Code Processing Models. arXiv:2210.17029 [cs.SE] https://arxiv.org/abs/2210.17029 [27] Siyuan Li, Yongpan Wang, Chaopeng Dong, Shouguo Yang, Hong Li, Hao Sun, Zhe Lang, Zuxin Chen, Weijie Wang, Hongsong Zhu, and Limin Sun. 2023. LibAM: An Area Matching Framework for Detecting Third-Party Libraries in Binaries. ACM Trans. Softw. Eng. Methodol. 33, 2, Article 52 (Dec. 2023), 35 pages. doi:10.1145/3625294 [28] Xiangyu Li, Marcelo d’Amorim, and Alessandro Orso. 2019. Intent-Preserving Test Repair . In 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST). IEEE Computer Society, Los Alamitos, CA, USA, 217–227. doi:10.1109/ICST.2019.00030 [29] Zhen Li, Deqing Zou, Shouhuai Xu, Hai Jin, Hanchao Qi, and Jie Hu. 2016. Vulpecker: an automated vulnerability detection system based on code similarity analysis. In Proceedings of the 32nd annual conference on computer security applications. 201–213. [30] Zhen Li, Deqing Zou, Shouhuai Xu, Hai Jin, Yawei Zhu, and Zhaoxuan Chen. 2021. Sysevr: A framework for using deep learning to detect software vulnerabilities. IEEE Transactions on Dependable and Secure Computing 19, 4 (2021), 2244–2258. [31] Zhen Li, Deqing Zou, Shouhuai Xu, Xinyu Ou, Hai Jin, Sujuan Wang, Zhijun Deng, and Yuyi Zhong. 2018. Vuldeepecker: A deep learning-based system for vulnerability detection. arXiv preprint arXiv:1801.01681 (2018). [32] Shuhan Liu, Jiayuan Zhou, Xing Hu, Filipe Roseiro Cogo, Xin Xia, and Xiaohu Yang. 2025. An Empirical Study on Vulnerability Disclosure Management of Open Source Software Systems. ACM Trans. Softw. Eng. Methodol. 34, 7, Article 214 (Aug. 2025), 31 pages. doi:10.1145/3716822 [33] Looly. [n. d.]. Introducing Commit of CVE-2023-51080. https://github.com/ chinabugotech/hutool/commit/c45b3f [34] Xinwei Mao. 2026. ATTAIN: Automated Exploit Failure Analysis through Trace-Driven Diff Analysis. GitHub repository. https://github.com/Cirno-9lab/ATTAIN_REPLICATION [35] Dongliang Mu, Alejandro Cuevas, Limin Yang, Hang Hu, Xinyu Xing, Bing Mao, and Gang Wang. 2018. Understanding the reproducibility of crowd-reported security vulnerabilities. In Proceedings of the 27th USENIX Conference on Security Symposium (Baltimore, MD, USA) (SEC’18). USENIX Association, USA, 919–936. [36] Viet Hung Nguyen, Stanislav Dashevskyi, and Fabio Massacci. 2016. An automatic method for assessing the versions affected by a vulnerability. Empirical Software Engineering 21, 6 (2016), 2268–2297. [37] Viet Hung Nguyen and Fabio Massacci. 2013. The (un) reliability of nvd vulnerable versions data: An empirical experiment on google chrome vulnerabilities. In Proceedings of the 8th ACM SIGSAC symposium on Information, computer and communications security. 493–498. [38] Shengyi Pan, Lingfeng Bao, Xin Xia, David Lo, and Shanping Li. 2023. Finegrained Commit-level Vulnerability Type Prediction by CWE Tree Structure. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). 957–969. doi:10.1109/ICSE48619.2023.00088 [39] Shengyi Pan, Lingfeng Bao, Jiayuan Zhou, Xing Hu, Xin Xia, and Shanping Li. 2024. Towards More Practical Automation of Vulnerability Assessment. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (Lisbon, Portugal) (ICSE ’24). Association for Computing Machinery, New York, NY, USA, Article 148, 13 pages. doi:10.1145/3597503.3639110 [40] Shengyi Pan, Jiayuan Zhou, Filipe Roseiro Cogo, Xin Xia, Lingfeng Bao, Xing Hu, Shanping Li, and Ahmed E. Hassan. 2022. Automated unearthing of dangerous issue reports. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Singapore, Singapore) (ESEC/FSE 2022). Association for Computing Machinery, New York, NY, USA, 834–846. doi:10.1145/3540250.3549156

ATTAIN: Automated Exploit Failure Analysis through Trace-Driven Diff Analysis

[41] Henning Perl, Sergej Dechand, Matthew Smith, Daniel Arp, Fabian Yamaguchi, Konrad Rieck, Sascha Fahl, and Yasemin Acar. 2015. Vccfinder: Finding potential vulnerabilities in open-source projects to assist code audits. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security. 426–437. [42] Shanto Rahman, Sachit Kuhar, Berk Cirisci, Pranav Garg, Shiqi Wang, Xiaofei Ma, Anoop Deoras, and Baishakhi Ray. 2025. UTFix: Change Aware Unit Test Repairing using LLM. Proc. ACM Program. Lang. 9, OOPSLA1, Article 85 (April 2025), 26 pages. doi:10.1145/3720419 [43] Shanto Rahman and August Shi. 2024. FlakeSync: Automatically Repairing Async Flaky Tests. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (Lisbon, Portugal) (ICSE ’24). Association for Computing Machinery, New York, NY, USA, Article 136, 12 pages. doi:10.1145/3597503. 3639115 [44] Ahmadreza Saboor Yaraghi, Darren Holden, Nafiseh Kahani, and Lionel Briand. 2025. Automated Test Case Repair Using Language Models. IEEE Trans. Softw. Eng. 51, 4 (Feb. 2025), 1104–1133. doi:10.1109/TSE.2025.3541166 [45] Youkun Shi, Yuan Zhang, Tianhan Luo, Xiangyu Mao, and Min Yang. 2022. Precise (un) affected version analysis for web vulnerabilities. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering. 1–13. [46] Jacek Śliwerski, Thomas Zimmermann, and Andreas Zeller. 2005. When do changes induce fixes? ACM sigsoft software engineering notes 30, 4 (2005), 1–5. [47] Synopsys. [n. d.]. OPEN SOURCE SECURITY AND RISK ANALYSIS REPORT 2023. https://www.synopsys.com/software-integrity/resources/analyst-reports/ open-source-security-risk-analysis.html [48] Lingxiao Tang, Jiakun Liu, Zhongxin Liu, Xiaohu Yang, and Lingfeng Bao. 2025. LLM4SZZ: Enhancing szz algorithm with context-enhanced assessment on large language models. Proceedings of the ACM on Software Engineering 2, ISSTA (2025), 343–365. [49] Gladys Tyen, Hassan Mansoor, Victor Cărbune, Peter Chen, and Tony Mak. 2024. LLMs cannot find reasoning errors, but can correct them given the error location. arXiv:2311.08516 [cs.AI] https://arxiv.org/abs/2311.08516 [50] Seunghoon Woo, Eunjin Choi, Heejo Lee, and Hakjoo Oh. 2023. { V1SCAN } : Discovering 1-day vulnerabilities in reused { C/C++ } open-source software components using code classification techniques. In 32nd USENIX Security Symposium (USENIX Security 23). 6541–6556. [51] Seunghoon Woo, Hyunji Hong, Eunjin Choi, and Heejo Lee. 2022. { MOVERY } : A precise approach for modified vulnerable code clone discovery from modified { Open-Source } software components. In 31st USENIX Security Symposium (USENIX Security 22). 3037–3053. [52] Seunghoon Woo, Dongwook Lee, Sunghan Park, Heejo Lee, and Sven Dietrich. 2021. { V0Finder } : Discovering the correct origin of publicly reported software vulnerabilities. In 30th USENIX Security Symposium (USENIX Security 21). 3041– 3058. [53] Susheng Wu, Ruisi Wang, Kaifeng Huang, Yiheng Cao, Wenyan Song, Zhuotong Zhou, Yiheng Huang, Bihuan Chen, and Xin Peng. 2024. Vision: Identifying Affected Library Versions for Open Source Software Vulnerabilities. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (Sacramento, CA, USA) (ASE ’24). Association for Computing Machinery, New York, NY, USA, 1447–1459. doi:10.1145/3691620.3695516 [54] Yang Xiao, Bihuan Chen, Chendong Yu, Zhengzi Xu, Zimu Yuan, Feng Li, Binghong Liu, Yang Liu, Wei Huo, Wei Zou, et al. 2020. { MVP } : Detecting

Conference’17, July 2017, Washington, DC, USA

vulnerabilities using { Patch-Enhanced } vulnerability signatures. In 29th USENIX Security Symposium (USENIX Security 20). 1165–1182. [55] Zhipeng Xue, Zhipeng Gao, Shaohua Wang, Xing Hu, Xin Xia, and Shanping Li. 2024. SelfPiCo: Self-Guided Partial Code Execution with LLMs. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (Vienna, Austria) (ISSTA 2024). Association for Computing Machinery, New York, NY, USA, 1389–1401. doi:10.1145/3650212.3680368 [56] Zhipeng Xue, Xiaoting Zhang, Zhipeng Gao, Xing Hu, Shan Gao, Xin Xia, and Shanping Li. 2026. Clean Code, Better Models: Enhancing LLM Performance with Smell-Cleaned Dataset. ACM Trans. Softw. Eng. Methodol. (Feb. 2026). doi:10. 1145/3793252 Just Accepted. [57] Xiao Yu, Lei Liu, Xing Hu, Jin Liu, and Xin Xia. 2025. Where Is Self-admitted Code Generated by Large Language Models on GitHub? arXiv:2406.19544 [cs.SE] https://arxiv.org/abs/2406.19544 [58] Qi Zhan, Xing Hu, Zhiyang Li, Xin Xia, David Lo, and Shanping Li. 2024. PS3: Precise Patch Presence Test based on Semantic Symbolic Signature. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (Lisbon, Portugal) (ICSE ’24). Association for Computing Machinery, New York, NY, USA, Article 167, 12 pages. doi:10.1145/3597503.3639134 [59] Qi Zhan, Xing Hu, Xin Xia, and Shanping Li. 2024. REACT: IR-Level Patch Presence Test for Binary. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (Sacramento, CA, USA) (ASE ’24). Association for Computing Machinery, New York, NY, USA, 381–392. doi:10. 1145/3691620.3695012 [60] Jiayuan Zhou, Michael Pacheco, Jinfu Chen, Xing Hu, Xin Xia, David Lo, and Ahmed E. Hassan. 2023. CoLeFunDa: Explainable Silent Vulnerability Fix Identification. In Proceedings of the 45th International Conference on Software Engineering (Melbourne, Victoria, Australia) (ICSE ’23). IEEE Press, 2565–2577. doi:10.1109/ICSE48619.2023.00214 [61] Jiayuan Zhou, Michael Pacheco, Zhiyuan Wan, Xin Xia, David Lo, Yuan Wang, and Ahmed E. Hassan. 2022. Finding a needle in a haystack: automated mining of silent vulnerability fixes. In Proceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering (Melbourne, Australia) (ASE ’21). IEEE Press, 705–716. doi:10.1109/ASE51524.2021.9678720 [62] Zhuotong Zhou, Yongzhuo Yang, Susheng Wu, Yiheng Huang, Bihuan Chen, and Xin Peng. 2024. Magneto: A Step-Wise Approach to Exploit Vulnerabilities in Dependent Libraries via LLM-Empowered Directed Fuzzing. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (Sacramento, CA, USA) (ASE ’24). Association for Computing Machinery, New York, NY, USA, 1633–1644. doi:10.1145/3691620.3695531 [63] Markus Zimmermann, Cristian-Alexandru Staicu, Cam Tenny, and Michael Pradel. 2019. Small World with High Risks: A Study of Security Threats in the npm Ecosystem. In 28th USENIX Security Symposium (USENIX Security 19). USENIX Association, Santa Clara, CA, 995–1010. [64] Anni Zou, Wenhao Yu, Hongming Zhang, Kaixin Ma, Deng Cai, Zhuosheng Zhang, Hai Zhao, and Dong Yu. 2024. DOCBENCH: A Benchmark for Evaluating LLM-based Document Reading Systems. arXiv:2407.10701 [cs.CL] https://arxiv. org/abs/2407.10701 [65] Xiaochen Zou, Yu Hao, Zheng Zhang, Juefei Pu, Weiteng Chen, and Zhiyun Qian. 2024. SyzBridge: Bridging the Gap in Exploitability Assessment of Linux Kernel Bugs in the Linux Ecosystem. NDSS (2024). doi:10.14722/ndss.2024.24926

Related documents

• The Complexity Kink: A Prompt-Side Structural Complexity Index for Code-Generation Reliability#978496
• Semantic Layer Induction from Raw Telemetry via Hierarchical LLM and RAG Abstraction#978497
• The Output-Space Hypothesis: Enumerative Equivalence Checking for Tensor Programs#978498
• DeltaSelect: Affordable A/B Testing for Coding Agents#978499
• Code-as-Auditor: Executable Compliance Reasoning via Regulation-to-Code#978500