Beyond Predefined Sinks: Security-Aware Dependency Analysis for LLM Agents

Beyond Predefined Sinks: Security-Aware Dependency Analysis for LLM Agents

arXiv:2610.03014v1 [cs.CR] 2 Oct 2026

HANG CUI Large language model (LLM)-based agents increasingly connect model-generated decisions to security-sensitive software capabilities, including command execution, filesystem access, network communication, browser control, and external tools. Existing analyses often use predefined sensitive operations to anchor their analysis, but operation identity alone is insufficient to determine the security implications of an agent behavior. We present AgentSecGraph, a security-aware static analysis framework that constructs a candidatecentered Security-Aware Agent Dependency Graph (Security-ADG) for each security-sensitive operation. The representation augments operation identity with agent relevance, source and dependency evidence, trust-boundary context, guard evidence, and external-effect semantics. We further introduce AgentSecBench, a reproducible corpus of 67 real-world LLM-agent repositories spanning 11 agent ecosystems and 37,542 source files. Applying the current analyzer to all frozen repositories yields 23,866 static security-sensitive operation candidates across 65 repositories, for which AgentSecGraph emits 23,866 candidate-centered Security-ADG artifacts without reported construction failures. Corpus-wide analysis recovers source-to-operation dependency evidence for 9,821 candidates (41.15%) and potential guard evidence for 3,075 (12.88%). The complete analysis finishes in 50.8 minutes. Using a separate reproduction-backed evaluation layer, we establish a conservative reference set of 22 security-sensitive behaviors across 13 repositories: one confirmed vulnerability, one pending disclosure candidate, and 20 guarded behaviors that do not meet the vulnerability criteria. In a held-out representation study, Security-ADG preserves 91.1% of the reference context and all five observed guards, compared with 20.0% for a sink-only view and 40.0% for a simplified ADG. These results show that security-aware dependency and contextual evidence enable distinctions that cannot be recovered from sensitive-operation identity alone. CCS Concepts: • Security and privacy → Software security engineering; • Software and its engineering → Automated static analysis. Additional Key Words and Phrases: LLM agents, software security, program analysis, static analysis Preprint. This version has not been peer reviewed.

1 Introduction Large language model (LLM)-based agents are rapidly evolving from text-generating assistants into software systems capable of planning, invoking tools, maintaining state, and interacting with external environments. Early paradigms such as ReAct [36] and Toolformer [27] demonstrated how language models can interleave reasoning with external actions. Modern frameworks, including LangChain [5], LangGraph [18], AutoGen [33], and CrewAI [7], provide reusable abstractions for tool invocation, memory, workflow orchestration, and multi-agent coordination. The Model Context Protocol (MCP) [2] extends this ecosystem by standardizing access to external tools and resources. This transition fundamentally changes the security boundary of LLM applications. In a conventional chatbot, an unsafe model output may remain textual; in an agent system, the same output can influence command execution, filesystem operations, browser actions, database access, or remote services. Agent behavior may additionally depend on user instructions, retrieved documents, memory contents, model-generated plans, tool outputs, and messages from other agents. Untrusted information can therefore cross multiple software and trust boundaries before affecting an externally visible operation. Prior work has demonstrated these risks through indirect prompt Author’s Contact Information: Hang Cui, cuihang26@mails.ucas.ac.cn.

2

Hang Cui

Fig. 1. Motivation: operation identity alone does not determine security semantics. All three agent tools contain the same command-execution operation. Cases A and C preserve a dependency from the agentfacing input, whereas Case B removes that dependency through a constant overwrite. Case C additionally contains a candidate allowlist guard. Security-aware dependency analysis distinguishes these cases using dependency and security-context evidence.

injection [13, 37], realistic agent attack environments [9], unsafe tool interactions [26], agent security benchmarks [38], resource abuse [25], and MCP-specific attack surfaces [16, 40]. Static program analysis provides a natural foundation for reasoning about these behaviors. Source–sink and taint analyses determine whether values from selected sources may reach securitysensitive operations. Systems such as YASA [30] demonstrate scalable taint analysis across multiple languages, while recent work has adapted program analysis to LLM-integrated applications. AgentFuzz [22] dynamically searches for prompts that trigger taint-style vulnerabilities, TaintP2X [14] statically traces LLM-generated values toward sensitive operations, and Agent Audit [39] combines agent-aware data-flow reasoning with credential, configuration, and privilege analysis. However, operation identity alone is insufficient for security interpretation. The same securitysensitive operation may be directly influenced by an agent-facing input, disconnected from that input by program transformations, or constrained by validation and authorization logic. Conversely, security-relevant capabilities may be exposed through framework-specific abstractions rather than obvious low-level API calls. A useful analysis must therefore recover not only which operation is present, but also how agent-relevant information reaches it and the security context in which it executes. Figure 1 illustrates this distinction. Agent frameworks further complicate this problem because relevant relationships may be hidden behind decorators, tool wrappers, workflow nodes, dynamically registered functions, plugins, and remote MCP tools. AgentFlow [29], for example, recovers framework-induced Agent Dependency Graphs (ADGs) over prompts, models, capabilities, memory, and control policies. Recovering these structural dependencies is valuable, but security interpretation also requires evidence about source provenance, trust boundaries, guards, privilege context, and resulting external effects.

Beyond Predefined Sinks

3

This motivates the central research question of this work: How can static analysis augment security-sensitive operations with agent-specific dependency and security-context evidence to analyze security-critical behaviors in real-world LLM-agent programs? We present AgentSecGraph, a security-aware static analysis framework for LLM-agent programs. Rather than treating a matched sensitive operation as a vulnerability verdict, AgentSecGraph uses it as an analysis anchor and constructs a candidate-centered Security-ADG. The representation augments operation identity with agent relevance, source provenance, local dependency evidence, trust-boundary context, candidate guards, statically available privilege information, and external-effect semantics. This design separates three analytical layers: Operation → Dependency → Security Context.

(1)

This separation allows AgentSecGraph to distinguish behaviors that appear identical under an operation-only view while making the limitations of the underlying static evidence explicit. We further introduce AgentSecBench, a reproducible corpus of 67 real-world LLM-agent repositories spanning 11 agent ecosystems, 37,542 source files, and 291.2 MB of source code. Applying the current candidate extractor to the full corpus identifies 23,866 static security-sensitive operation candidates across 65 repositories and 7,214 candidate-bearing source files. AgentSecGraph emits one candidate-centered Security-ADG artifact for every candidate without reported construction failures. Across this population, AgentSecGraph recovers source-to-operation dependency evidence for 9,821 candidates (41.15%), explicit framework-specific evidence for 749 (3.14%), and potential guard evidence for 3,075 (12.88%). End-to-end analysis completes in 50.8 minutes. Separately, we retain a frozen pilot cohort and construct reproduction-backed evidence for 22 security-sensitive behaviors across 13 repositories. The resulting reference set contains one confirmed vulnerability, one pending disclosure candidate, and 20 guarded non-vulnerabilities under the evaluated conditions; neither model-generated judgments nor single-reviewer labels are used as ground truth. Across nine evidence-backed held-out cases, Security-ADG preserves 91.1% of the reference context and all five observed guards, compared with 20.0% context completeness and 0/5 guard visibility for a sink-only view, and 40.0% and 0/5, respectively, for a simplified ADG view. Our contributions are summarized as follows: • Security-analysis formulation. We formulate LLM-agent security analysis by distinguishing among sensitive-operation identity, security-critical behavior evidence, and vulnerability claims, avoiding the assumption that every matched sensitive operation constitutes a vulnerability. • Security-aware dependency analysis. We design AgentSecGraph and its candidatecentered Security-ADG representation to integrate agent relevance, source provenance, local dependency evidence, trust-boundary context, candidate guards, and external-effect semantics. • Benchmark and corpus-wide analysis. We construct AgentSecBench from 67 frozen real-world LLM-agent repositories spanning 11 ecosystems and 37,542 source files. A fullcorpus run of the current analyzer emits 23,866 candidate-centered Security-ADG artifacts across 65 repositories, enabling corpus-wide characterization and scalability analysis. • Reproduction-backed empirical evidence. We establish a reproduction-confirmed reference set of 22 security-sensitive behaviors across 13 repositories and quantify the securitycontext information preserved by sink-only, simplified ADG, and full Security-ADG representations.

4

2

Hang Cui

Motivation and Problem Formulation

LLM-agent software differs from conventional language-model applications: model-mediated decisions can directly trigger executable capabilities. Modern agents combine tools, memory, retrieval, workflows, and multi-agent communication [5, 7, 18, 27, 33, 36], while protocols such as MCP [2] standardize access to external tools and resources. These capabilities introduce security-relevant relationships that are often hidden behind framework abstractions such as tool decorators, workflow nodes, dynamically registered functions, plugins, and inter-agent handoffs. Consequently, security analysis must reason not only about host-language operations, but also about how agent-facing inputs can access and influence those operations. 2.1

Agent-Specific Security Semantics

Untrusted information in an agent application may originate from user instructions, retrieved documents, web content, memory, model outputs, or external tools. Such values may subsequently influence command execution, filesystem access, network communication, browser actions, database operations, or other externally visible effects. Static source–sink and taint analysis provides a natural foundation for recovering these relationships. Classical program representations such as Program Dependence Graphs (PDGs) [12] capture data and control dependencies, while systems such as FlowDroid [3] and YASA [30] demonstrate precise and scalable source–sink reasoning. Recent work extends these ideas to LLM-integrated software: TaintP2X [14] traces LLM-generated values toward sensitive operations, Agent Audit [39] incorporates agent-specific configuration and privilege information, and AgentFuzz [22] dynamically searches for inputs that trigger sensitive behavior. However, LLM-agent software introduces an additional semantic layer. A parameter exposed through an agent tool interface has different security implications from an unrelated internal parameter, even if both eventually reach the same low-level API. Framework-induced relationships must therefore be interpreted alongside ordinary program dependencies. 2.2

Why Sink Identity Is Insufficient

Figure 1 illustrates the central limitation of operation-only analysis. All three examples contain the same command-execution operation, but their security semantics differ: one preserves a direct dependency from an agent-facing input, one removes that dependency through a constant overwrite, and one retains the dependency while introducing an allowlist guard. This observation motivates a distinction among three levels of evidence. A security-sensitive operation candidate is a source-code operation that may produce or prepare a security-relevant external effect. A security-critical behavior combines such an operation with agent-specific dependency and contextual evidence. A vulnerability is a stronger conclusion that additionally depends on factors such as controllability, protection mechanisms, deployment exposure, authorization, privilege, impact, and exploitability. Accordingly, Sensitive Operation ≠ Security-Critical Behavior ≠ Vulnerability.

(2)

Predefined operation specifications are therefore useful for locating code that deserves further analysis, but operation identity should be treated as an analysis anchor, not as a vulnerability verdict.

Beyond Predefined Sinks

2.3

5

Security Context and Analysis Objective

Let O = {𝑜 1, 𝑜 2, . . . , 𝑜𝑛 } (3) denote the set of security-sensitive operation candidates in a frozen program snapshot. Candidate generation is intentionally permissive: a match does not imply that the operation is agent-controlled, reachable from an untrusted source, insufficiently guarded, or exploitable. For each candidate 𝑜 ∈ O, we characterize the statically available security context as 𝐶 (𝑜) = ⟨𝑅, 𝑆, 𝐷,𝑇 , 𝑃, 𝐺, 𝐸⟩, (4) where 𝑅 denotes agent relevance; 𝑆, source provenance; 𝐷, source-to-operation dependency evidence; 𝑇 , trust-boundary evidence; 𝑃, statically available privilege context; 𝐺, candidate guard evidence; and 𝐸, the external-effect category. These dimensions represent evidence rather than security verdicts. Dependency or trustboundary evidence does not establish exploitability, while the presence of a guard does not establish that the guard is complete or effective. Given a frozen LLM-agent program and candidate set O, our goal is therefore to recover the dependency and security-context evidence needed to interpret each candidate: Operation + Dependency + Security Context → Behavior Evidence. (5) The objective is deliberately narrower than automatic vulnerability verification. AgentSecGraph recovers auditable evidence about how an agent-relevant value may influence a sensitive capability and what security context surrounds that relationship. Any stronger vulnerability determination is made separately using reproduction and security evidence. Section 3 presents the resulting candidate-centered Security-ADG analysis. 3

AgentSecGraph

3.1

Overview

AgentSecGraph is a static analysis framework for recovering auditable security-relevant evidence associated with sensitive operations in LLM-agent software. Rather than interpreting a matched operation as a security violation, AgentSecGraph treats it as an analysis anchor and builds a candidate-centered Security-ADG for it. Given a candidate operation 𝑜 ∈ O, AgentSecGraph determines how the operation is exposed to agents, recovers relevant source-to-operation dependencies, extracts trust-boundary and guard evidence, and summarizes the resulting security context using 𝐶 (𝑜) from Equation 4. Figure 2 summarizes the end-to-end workflow. The framework deliberately separates static evidence recovery from vulnerability assessment. Its output records only what can be established from the frozen source snapshot and retains uncertainty when the conclusion depends on runtime, deployment, or threat-model evidence. 3.2

Candidate Anchoring and Framework Normalization

The analysis begins with security-sensitive operation candidates 𝑜 ∈ O. Each candidate is associated with its frozen repository snapshot, source location, operation category, detector provenance, and local code context. Candidate generation is intentionally permissive. A match establishes that a potentially securityrelevant capability exists, but not that the capability is agent-controlled, reachable from an untrusted source, insufficiently guarded, or exploitable.

6

Hang Cui

Fig. 2. Overview of AgentSecGraph. Starting from a security-sensitive operation candidate, AgentSecGraph normalizes framework-specific agent semantics, identifies relevant sources, recovers dependency and security-context evidence, and constructs a candidate-centered Security-ADG for subsequent analysis and review.

Agent frameworks further complicate source-level interpretation because tools and capabilities are exposed through heterogeneous abstractions, including decorators, registration APIs, factories, workflow nodes, and tool classes. AgentSecGraph therefore maps supported framework constructs into a common semantic record N = ⟨𝐹, 𝑅𝑠 , 𝑌 , 𝑃𝑠 , 𝑄⟩,

(6)

where 𝐹 identifies the framework; 𝑅𝑠 , the semantic role of the construct; 𝑌 , the associated program symbol; 𝑃𝑠 , its exposed parameters; and 𝑄, the match confidence. Normalization affects semantic interpretation rather than vulnerability assessment. For example, when a function is recognized as an agent-facing tool, its exposed parameters can be classified as agent-relevant sources rather than ordinary function parameters. The current implementation supports representative constructs from MCP/FastMCP, LangChain, CrewAI, AutoGen, OpenAI Agents SDK, Semantic Kernel, and LlamaIndex. The supplementary material documents the normalization contract, supported construct families, and matching rules. 3.3

Security-ADG Construction

For each candidate operation 𝑜, AgentSecGraph constructs a typed Security-Aware Agent Dependency Graph 𝐺𝑜 = (𝑉𝑜 , 𝐸𝑜 , Φ𝑜 ),

(7)

where 𝑉𝑜 contains evidence nodes associated with 𝑜, 𝐸𝑜 contains recovered evidence relationships, and Φ𝑜 stores graph-level provenance and security-context attributes. Unlike a whole-program dependency graph, 𝐺𝑜 is candidate-centered. Its purpose is not to reconstruct every program relationship, but to retain the evidence required to interpret one securitysensitive operation. Figure 3 shows a representative candidate-centered Security-ADG. Table 1 summarizes the core node and edge types.

Beyond Predefined Sinks

7

Fig. 3. Example candidate-centered Security-ADG. The representation augments a security-sensitive operation with agent-facing source provenance, dependency relationships, trust-boundary evidence, candidate guards, and the resulting external effect. Relations prefixed with may_ represent static evidence rather than proofs of reachability or exploitability.

Table 1. Core node and edge types in Security-ADG. Relations prefixed with may_ represent static evidence rather than reachability or exploitability proofs.

Type

Meaning

Agent/program symbol Sensitive operation Input source Trust boundary Guard candidate

Agent-facing or enclosing program construct associated with the candidate. Candidate operation that performs or prepares a security-relevant effect. Value originating from an agent-facing or external context. A crossing from a less-trusted or external context to a local capability. Validation, authorization, sandboxing, argument restriction, or related defensive evidence. An abstract effect involving a process, filesystem, network, browser, or external service, deterministically assigned from the operation category.

External effect contains may_cause may_data_depend_on may_cross may_guard

A program symbol contains the candidate operation. The candidate may produce the represented external effect. A source may influence values consumed by the candidate. Evidence suggests that a trust boundary is crossed. A guard candidate may constrain the operation.

The use of may_* relations is deliberate. These edges represent static evidence and possible relationships; they are not proofs of end-to-end reachability, attacker control, exploitability, or guard sufficiency.

8

3.4

Hang Cui

Dependency, Trust-Boundary, and Guard Analysis

Starting from values consumed by a candidate operation, AgentSecGraph performs backward dependency recovery to identify whether a recognized source may influence the candidate. Conceptually, the analysis seeks a local chain 𝑠 ⇝ 𝑣 1 ⇝ · · · ⇝ 𝑣𝑜 → 𝑜,

(8)

where 𝑠 is an agent-relevant or externally influenced source and 𝑣𝑜 is consumed by operation 𝑜. For Python, dependency recovery is AST-based and follows local parameters, assignments, definitions, and reassignments. For example, if an agent-facing parameter is overwritten by a constant before reaching the candidate, the corresponding dependency is terminated. For TypeScript and JavaScript, the current implementation performs syntax-aware lexical backward tracing over local parameters, definitions, assignments, and supported registration patterns. This analysis is more conservative than the Python engine and does not provide complete alias, type, or control-flow reasoning. Dependency evidence alone does not imply that a behavior is unsafe. AgentSecGraph therefore also records evidence of safeguards and trust boundaries. Representative guard evidence includes allowlists, fixed-argument or argument-vector command construction, schema or syntax validation, authentication checks, path restrictions, sandboxing, and resource limits. A may_guard edge indicates only that defensive evidence is visible; it does not prove that the guard is complete or effective. Similarly, may_cross records available evidence that an external, agent-facing, or otherwise less-trusted value may influence a local or more privileged capability, while deployment-dependent trust and privilege may remain unknown. The current implementation represents privilege context only when explicit evidence is recoverable from supported source-level constructs. It does not infer deployment identities, ambient permissions, or runtime authorization state. Likewise, every candidate is deterministically mapped to an abstract external-effect category. The resulting effect node encodes modeled semantics rather than a runtime-confirmed effect and is therefore a schema property, not an empirical recovery result. The recovered evidence is summarized using the security-context tuple 𝐶 (𝑜) defined in Equation 4, whereas the graph itself preserves source-level provenance and relationships for inspection. In particular, 𝐷 (𝑜) = true means that dependency evidence was recovered; it does not imply Vulnerability(𝑜) = true. 3.5

Implementation and Analysis Contract

AgentSecGraph is implemented as a modular source-analysis pipeline with language-specific analyzers, framework adapters, and deterministic graph generation. The Python engine uses the language AST to recover enclosing scopes, parameters, local definitions, assignments, decorators, and backward def–use relationships. The TypeScript/JavaScript engine performs syntax-aware lexical tracing and explicitly records the weaker guarantees provided by this approach. Framework adapters map framework-specific tool declarations and capability-registration mechanisms into the normalized representation from Equation 6. The adapter layer is independent of the graph schema, allowing new frameworks to be supported without redesigning Security-ADG. For each candidate, the implementation emits a candidate-centered Security-ADG together with source-level provenance and analysis metadata. We report graph construction as successful when the deterministic pipeline emits exactly one graph artifact linked to the original candidate identifier and source provenance. This measures one-to-one generation and linkage, not the semantic correctness

Beyond Predefined Sinks

9

or completeness of recovered relationships. It also derives three deterministic representation views from the same candidate: (1) sink-only, retaining only the sensitive operation; (2) simplified ADG, retaining structural operation and external-effect information; and (3) full Security-ADG, retaining source/dependency, trust-boundary, guard, and external-effect evidence. These projections allow representation-level comparisons without changing the underlying candidate population. All analyses operate on repositories frozen at recorded Git commits. Candidate-generation rules, framework adapters, dependency logic, and graph construction are independent of human annotations, model-generated security judgments, reproduction outcomes, and vulnerability dispositions. The current analysis is intentionally local. Python analysis is not a complete whole-program interprocedural, alias, points-to, or symbolic analysis, and TypeScript/JavaScript tracing does not provide sound, complete alias or control-flow analysis. Reflection, dynamic registration, metaprogramming, and deployment-dependent exposure or privilege may therefore produce incomplete evidence. Accordingly, Security-ADG should be interpreted as an auditable static evidence representation for a candidate operation, not as a proof of reachability, exploitability, guard sufficiency, or vulnerability. 4

Evaluation

We evaluate AgentSecGraph on frozen snapshots of real-world LLM-agent repositories. Our four questions concern corpus-wide coverage and scalability, conservative behavior-level ground truth, distinctions beyond sink identity, and the security context preserved by Security-ADG. Throughout the evaluation, we distinguish three levels of evidence: static candidates, reproductionconfirmed behaviors, and vulnerability dispositions. Static candidates are analysis inputs rather than vulnerability labels. 4.1

Evaluation Populations and Experimental Setup

Corpus construction. AgentSecBench is a corpus of publicly available LLM-agent repositories drawn from diverse ecosystems, including MCP-based systems, coding agents, framework-based applications, autonomous agents, and multi-agent platforms. We retain repositories containing substantive agent logic and analyzable source code while excluding duplicates, mirrors, trivial examples, and repositories unsuitable for reproducible source-level analysis. Repository inclusion is independent of AgentSecGraph output. Each retained repository is frozen at a recorded Git commit so that candidate extraction, graph construction, reproduction, and evaluation refer to the same source snapshot. The supplementary material documents the collection criteria and aggregate corpus composition. The public artifact repository provides the complete row-level repository inventory and immutable commit identifiers. Table 2 summarizes the corpus-wide analysis and the reproduction-confirmed reference set. Python is the dominant implementation language: 51 of 67 repositories (76.1%) primarily use Python, while 7 (10.4%) primarily use TypeScript. The remaining repositories are implemented mainly in C#, Rust, Go, PHP, and Shell. This imbalance is relevant because AgentSecGraph provides stronger AST-based dependency recovery for Python than the lexical analysis currently used for TypeScript/JavaScript. Evaluation populations. We distinguish three candidate populations throughout the evaluation. The frozen pilot cohort contains 531 candidates from five repositories and is retained for compatibility

10

Hang Cui

with earlier pilot and annotation studies. Running the current scanner on the same five repository snapshots yields 508 candidates because later language scoping and AST-based filtering changed candidate extraction. We call this the current five-repository scan and use it only to document scanner-version drift; we do not merge it with the frozen cohort. The current full-corpus run applies the current scanner to all 67 frozen repositories and yields 23,866 static security-sensitive operation candidates. This population is used for corpus-wide characterization and scalability measurements. None of these candidate counts denotes vulnerabilities. Full-corpus candidate extraction. The candidate taxonomy includes command/process execution, dynamic code execution, filesystem access, network and remote access, browser and desktop automation, database operations, credential or secret access, and external-tool invocation. Applying the current scanner to all 67 frozen repositories yields 23,866 candidates from 7,214 source files. Sixty-five repositories contain at least one candidate. All 67 repositories complete without skips or scanner failures. These candidates form the population for corpus-wide Security-ADG characterization; they may represent legitimate capabilities, disconnected operations, guarded behaviors, or potentially unsafe paths. Experimental environment. All experiments were conducted on an HP ProBook 440 14-inch G10 equipped with a 13th Gen Intel Core i5-1340P CPU (16 logical processors) and 32 GB of RAM, running Microsoft Windows 11 Home (build 26200). The implementation used CPython 3.13.12 from the Anaconda distribution. Candidate extraction and Security-ADG construction each used a single analysis process with no worker pool; repository Git reads were issued sequentially. The frozen repositories and generated artifacts were stored on a 1 TB WD Elements SE USB SSD. Wallclock measurements are from one completed full-corpus run. The archived aggregate records phase and per-repository timings, but not repeated-run measurements or a formal peak-RSS measurement; we therefore report the observed wall time rather than a mean or stable cross-system performance estimate. Freeze and leakage control. Each evaluation population is analyzed under a fixed configuration. The historical 531-candidate pilot manifest is preserved unchanged, whereas the current fullcorpus analysis uses the later scanner version frozen before corpus-wide characterization. Within the current full-corpus evaluation, repository snapshots, candidate-generation rules, framework adapters, dependency-recovery logic, the Security-ADG schema, representation projections, and evaluation scripts remain fixed. Graph construction uses neither human security labels nor modelgenerated judgments, reproduction outcomes, vulnerability dispositions, or disclosure status. Use of generative AI tools. We used OpenAI Codex with the GPT-5.6 model as a supervised research assistant during implementation and experimental execution. Its roles included generating and revising analysis code, test scaffolding, experiment orchestration commands, diagnostic suggestions, and figure-generation code. The authors reviewed AI-assisted changes before their use in reported experiments. We validated the resulting implementation through deterministic unit tests, frozeninput reruns, schema and invariant checks, and recomputation of reported aggregates from archived intermediate outputs. We additionally used a frozen five-model panel to independently pre-annotate 150 candidate tasks and support disagreement-based triage. These outputs were not used as ground-truth labels, were not consumed by Security-ADG construction, and did not determine reproduction or vulnerability dispositions. Reference-set inclusion instead required inspectable evidence from frozen source snapshots and controlled local reproduction reviewed by the authors. The exact model identifiers, endpoints, and inference settings are reported in the supplementary material.

Beyond Predefined Sinks

11

Table 2. Corpus-wide analysis and reproduction evidence. Candidate counts are not vulnerability findings.

Corpus-Wide Analysis Statistic Frozen repositories Agent ecosystems Source files Source size Candidate-bearing repositories Candidate-bearing source files Current full-corpus candidates Generated Security-ADG artifacts

4.2

Reproduction Evidence Value 67 11 37,542 291.2 MB 65 7,214 23,866 23,866

Outcome Reproduction-confirmed behaviors Repositories represented Vulnerability ground truth Pending disclosure candidate Guarded non-vulnerability

Count 22 13 1 1 20

Reproduction-Confirmed Ground Truth

Static candidates do not constitute behavior or vulnerability ground truth. We therefore establish a conservative reference set through targeted reproduction. Validation progresses from source inspection to controlled behavioral confirmation using, where appropriate, isolated execution, stubbing or monkeypatching of external effects, or paired attack–defense tests. A case enters the reference set only if the relevant behavior can be tied to the frozen source snapshot. Model assistance may support diagnostics or test scaffolding, but model judgments are never treated as ground-truth labels. The resulting set contains 22 reproduction-confirmed behaviors across 13 repositories. Each case retains its frozen source location, Security-ADG instance, and recovered dependency path. We conservatively assign each case one of three dispositions: • Vulnerability ground truth: reproduction and security evidence support a vulnerability conclusion under the evaluated threat model; • Pending disclosure candidate: the behavior warrants further coordination or validation but is not treated as confirmed vulnerability ground truth; and • Guarded non-vulnerability: the behavior is reproducible, but observed protections or usage constraints do not support a vulnerability conclusion under the tested conditions. Table 2 summarizes the reference set alongside the corpus-wide analysis population. The supplementary material documents the auxiliary 344-operation inventory and matching protocol. Incomplete independent adjudication precludes aggregate recovery metrics and exhaustive repository-wide ground-truth claims. 4.3

RQ1: Corpus-Wide Coverage and Scalability

Question. How does AgentSecGraph behave when applied to the full AgentSecBench corpus in terms of candidate scale, recovered security-context evidence, graph construction success, and analysis cost? Results. The current scanner completes all 67 frozen repositories without skips or failures and identifies 23,866 static security-sensitive operation candidates across 65 repositories and 7,214 candidate-bearing source files. AgentSecGraph emits one candidate-centered Security-ADG artifact for every candidate, with no reported construction failures. Following the operational criterion in Section 3.5, this measures one-to-one generation and provenance linkage rather than semantic correctness.

12

Hang Cui

Table 3. Evidence recovered in the current full-corpus run. Rates use all 23,866 candidates as the denominator.

Evidence or artifact

Count

Rate

Framework-specific evidence Source-to-operation dependency evidence Potential guard evidence Generated Security-ADG artifacts

749 9,821 3,075 23,866

3.14% 41.15% 12.88% 100.0%

Table 4. Scalability results from one completed full-corpus run.

Metric

Observed result

Repositories completed Candidates / generated graphs Candidate extraction Security-ADG construction End-to-end wall time Throughput Median repository time P90 / P95 repository time

67/67 23,866 / 23,866 1,508.5 s 1,538.2 s 3,046.6 s (50.8 min) 7.83 candidates/s 9.6 s 203.0 s / 236.2 s

Across this population, explicit framework-specific evidence is recovered for 749 candidates (3.14%), source-to-operation dependency evidence for 9,821 (41.15%), and potential guard evidence for 3,075 (12.88%). Table 3 summarizes these results. These frequencies characterize evidence exposed by the current static analysis and do not estimate vulnerability prevalence or repositorywide recall. The exported corpus-wide aggregates give identical counts for source, dependency, and trustcrossing fields. Because these aggregates do not demonstrate three independent recovery events, we conservatively report one source-to-operation dependency-evidence measure rather than triplecounting the same 9,821 candidates. Independent trust-boundary recovery accuracy is not measured by this experiment. The current extraction rules recover no explicit privilege-context evidence in this run. Every candidate is assigned an abstract external-effect node by its operation category; this is a graph schema property rather than a recovered or runtime-confirmed effect. Evidence recovery varies across operation categories. Source-to-operation dependency evidence is recovered for 84.14% of filesystem-write candidates and 65.28% of filesystem-read candidates, compared with 11.28% of external-tool invocations and 1.87% of browser-control candidates. Potential guard evidence ranges from 51.29% for filesystem deletion and 41.82% for dynamic-code execution to 1.37% for external-tool invocation. These differences characterize the current analyzer and corpus; they must not be interpreted as differences in vulnerability prevalence. The complete per-category table is provided in the supplementary material. Scalability. Candidate extraction requires 1,508.5 seconds and Security-ADG construction 1,538.2 seconds, for an end-to-end wall time of 3,046.6 seconds (50.8 minutes) and throughput of 7.83 candidates/s. Combined per-repository analysis time has a median of 9.6 seconds, with P90 and P95 values of 203.0 and 236.2 seconds, respectively. Table 4 reports the observed run. The latency distribution is strongly skewed toward larger repositories.

Beyond Predefined Sinks

13

Finding. AgentSecGraph processes the complete 67-repository corpus and emits candidatecentered graph artifacts for all 23,866 detected operations without reported construction failures. Security-context evidence is unevenly distributed: 41.15% of candidates contain recovered source-tooperation dependency evidence and 12.88% expose potential guard evidence. The observed runtime supports corpus-scale offline analysis, although large repositories produce a pronounced latency tail. 4.4

RQ2: Reproduction-Confirmed Evidence

Question. Can reproduction-backed validation establish conservative ground truth for securitysensitive agent behaviors without relying on model judgments or single-reviewer labels? Results. We reproduce 22 security-sensitive behaviors across 13 repositories. Only one case currently satisfies our criteria for vulnerability ground truth; one additional case remains pending disclosure, while 20 are guarded behaviors that do not meet the vulnerability criteria under the evaluated conditions. This distribution shows that reproducing a sensitive behavior does not by itself establish a vulnerability. Even when an agent-facing value reaches a powerful capability, deployment exposure, authorization, privilege, guards, and impact remain relevant to the vulnerability determination. Finding. Reproduction-backed validation provides auditable behavior-level evidence while preserving the distinction between security-sensitive behavior and vulnerability. Neither static candidates nor model-generated judgments are used as ground truth. 4.5

RQ3: Beyond Sink Identity

Question. What security-relevant distinctions does Security-ADG expose beyond sensitive-operation identity alone? Results. Across the reproduction-confirmed cases, superficially similar sensitive operations exhibit materially different security contexts. Recurring patterns include direct versus removed source dependencies, guarded versus unguarded execution, validation paths versus effectful execution paths, and identical low-level operations exposed under different agent or trust contexts. For example, an agent-facing parameter may directly influence command execution in one case but be overwritten by a constant in another. Similarly, an otherwise equivalent dependency path may be accompanied by an allowlist, authentication check, fixed argument construction, path restriction, or sandboxing mechanism. A sink-only representation cannot express these distinctions because it records only operation identity. Security-ADG instead retains source provenance, dependency, trust-boundary, guard, and effect evidence associated with the same candidate. Finding. Sensitive-operation identity is useful for candidate discovery but insufficient for security interpretation. The additional dependency and security-context evidence preserved by SecurityADG distinguishes behaviors that appear equivalent under an operation-only view. 4.6

RQ4: Context Preservation

Question. How much security-relevant context does Security-ADG preserve compared with sinkonly and simplified ADG representations? Setup. We derive three deterministic projections from the same underlying candidate: a sinkonly view retaining only the sensitive operation; a simplified ADG retaining structural operation

14

Hang Cui

and external-effect information; and the full Security-ADG retaining source/dependency, trustboundary, guard, and external-effect evidence. All three representations derive from the same candidate, so their differences reflect information lost by the representation rather than differences in candidate generation. We evaluate nine evidence-backed held-out cases: five validated by controlled local monkeypatch proofs and four by source-level construction proofs that did not execute the external effect. Five cases contain reference guard evidence. Before scoring, we freeze a case-level reference matrix from the corresponding evidence records. Let 𝐴𝑖 contain the independently supported, applicable context elements for case 𝑖, and let 𝑅𝑖 (𝑣) ⊆ 𝐴𝑖 be those retained by representation 𝑣. Absent or unknown reference elements are excluded. Context completeness is Í 𝑖 |𝑅𝑖 (𝑣)| CC(𝑣) = Í , (9) 𝑖 |𝐴𝑖 | An explicitly recorded no_guard_confirmed posture is treated as a supported negative reference element rather than an absent or unknown one; it therefore remains in the denominator, although a representation receives positive guard-visibility credit only for concrete guard evidence. The sums give a micro-average. Guard visibility counts cases in which both the reference and representation expose guard evidence. The supplementary matrix reports each case’s applicable elements, evidence provenance, projection-specific retention, and score, making the 91.1% aggregate auditable. Results. Table 5 reports the comparison. Table 5. Security-context preservation across representation variants. Higher values indicate that more reference context is preserved.

Representation

Context Completeness

Guard Visibility

Sink-only Simplified ADG

20.0% 40.0%

0/5 0/5

Security-ADG

91.1%

5/5

The sink-only view preserves 20.0% of the applicable reference context. Adding structural operation and effect information increases completeness to 40.0%, but does not expose any of the five observed guards. The full Security-ADG reaches 91.1% context completeness and preserves all five instances of guard evidence. The increase from 40.0% to 91.1% shows that structural dependency information alone does not account for the improvement; source provenance, security-relevant dependency, trust-boundary, and guard evidence contribute substantial additional context. Finding. Security-ADG preserves substantially more independently supported security context than either reduced representation. These figures measure context preservation, not vulnerabilitydetection accuracy, and guard visibility in all five cases does not imply that the observed guards are sufficient. 5 5.1

Discussion and Threats to Validity Interpreting “Beyond Predefined Sinks”

Our results do not imply that predefined security-sensitive operations should be discarded. Such specifications remain a practical and scalable mechanism for locating code that warrants further

Beyond Predefined Sinks

15

analysis. The limitation is that operation identity alone does not determine whether an operation is agent-influenced, whether it crosses a trust boundary, or what protection mechanisms constrain its use. Accordingly, “beyond predefined sinks” refers to augmenting sink identity rather than replacing sink specifications. AgentSecGraph uses predefined operations as candidate anchors and then recovers agent semantics, source provenance, dependency, trust-boundary, guard, and externaleffect evidence for each one. This distinction is reflected in the representation study: retaining only operation identity preserves substantially less independently supported security context than the full Security-ADG. The benefit of Security-ADG therefore lies not in eliminating sink specifications, but in providing the evidence needed to interpret superficially similar operations differently. 5.2

From Security Evidence to Vulnerability

A reproduced security-sensitive behavior is not necessarily a vulnerability. The reproduction study demonstrates this distinction directly: behaviors can be tied to frozen source snapshots and exercised under controlled conditions while still being constrained by validation, authorization, sandboxing, fixed argument construction, deployment assumptions, or intended usage. Static evidence should therefore be interpreted according to its strength. A recovered dependency indicates that the available source-level evidence supports a possible influence from an agentrelevant value to a sensitive operation. A recovered guard indicates that potentially relevant defensive evidence is visible. Neither observation alone proves exploitability, vulnerability, or safety. We therefore view Security-ADG as an evidence representation rather than a binary vulnerability classifier. Candidate-centered graphs support a staged workflow: inexpensive static analysis identifies and contextualizes behaviors, and costlier reproduction and security review examine cases that warrant it. This separation also supports security auditing, candidate prioritization, regression analysis, and responsible-disclosure triage, provided that these uses preserve the distinction between evidence and vulnerability conclusions. 5.3

Limitations and Threats to Validity

Construct and internal validity. Our evaluation relies on distinctions among sensitive operations, security-critical behaviors, and vulnerabilities, as well as on the context dimensions in SecurityADG. These abstractions omit some deployment-dependent factors. Context completeness measures represented evidence, not proven reachability or exploitability; guard visibility does not establish sufficiency. Errors in candidate rules, adapters, dependency recovery, or reproduction could affect the results. We mitigate this risk by freezing snapshots and analysis configurations before evaluation, separating graph construction from labels, and retaining source locations, graphs, and dependency evidence. The analysis is intentionally incomplete. Python recovery is local and AST-based, not a sound whole-program, alias, points-to, or symbolic analysis. TypeScript and JavaScript rely on weaker, syntax-aware lexical tracing. Reflection, dynamic registration, metaprogramming, or deploymentspecific exposure and privilege may therefore yield missing or imprecise evidence. Privilege context is represented only when explicit evidence is recoverable from supported source-level constructs; the current full-corpus extraction recovers none. In addition, the available corpus-wide aggregates report identical source, dependency, and trust-crossing counts. We therefore do not interpret them as three independent recovery measurements or claim an independently validated trust-boundary recovery rate.

16

Hang Cui

External validity. AgentSecBench contains 67 frozen repositories from 11 agent ecosystems but is not statistically representative of the entire ecosystem. Availability, selection criteria, and open-source access shape its composition. Python dominates the sample, and AgentSecGraph analyzes Python dependencies more strongly than TypeScript/JavaScript dependencies; aggregate results are therefore not language-independent performance estimates. Conclusion validity. The current full-corpus candidate population is generated by a later scanner version than the frozen five-repository pilot manifest. The 23,866-candidate full-corpus population and historical 531-candidate cohort are therefore not directly comparable as a corpus-size scaling experiment. Re-running the current scanner on the same five snapshots yields 508 candidates because language scoping and AST filtering changed; we preserve the historical manifest rather than merging the populations. Corpus-wide evidence frequencies are descriptive properties of the current analyzer and benchmark. They neither estimate vulnerability prevalence nor establish recall with respect to all securityrelevant behaviors in the repositories. The reported timing comes from one completed run rather than a repeated-run distribution, and the archived aggregate does not include peak memory usage; scalability conclusions are correspondingly limited to the observed run. The five-model panel was used only to prioritize candidate inspection and did not determine reference-set eligibility. Because this triage may nevertheless influence which cases receive earlier manual attention, we do not treat the reproduction set as an unbiased estimate of corpus-wide behavior prevalence. The reference set contains 22 behaviors across 13 repositories; the representation study contains nine held-out cases, including five with guard evidence. These samples support the reported caseand representation-level findings, but not exhaustive recall or general vulnerability-detection claims. We therefore do not use precision, recall, or F1 as primary metrics for the reference set. Incomplete adjudication likewise precludes matched counts or aggregate recovery metrics for the 344-operation inventory. Likewise, 91.1% context completeness and 5/5 guard visibility describe the evaluated cases rather than universal performance. 6 6.1

Related Work Security Evaluation of LLM Agents

The growing autonomy of LLM agents has motivated extensive work on their attack surfaces. Prior studies demonstrate indirect prompt injection [13, 24, 37], evaluate attacks and defenses in realistic agent environments [9], and study unsafe agent–tool interactions [26]. Broader benchmarks examine security failures across prompts, tools, and memory [38], harmful multi-step agent misuse [1], and real-world software-security tasks [19]. Recent work broadens the threat model beyond prompt injection to poisoned agent memory [6], web-agent compromise [11], resource abuse [25], and MCP-specific risks [16, 31, 32, 40]. A recent systematization organizes the design, attack, and defense landscape of agentic systems [17]. Largescale ecosystem evidence also shows that third-party agent skills can carry deliberately malicious instructions and helper code [23]. These studies primarily focus on attack generation, adversarial evaluation, or vulnerability benchmarking. AgentSecGraph addresses a complementary problem: recovering auditable dependency and security-context evidence through static analysis of sensitive behavior in real-world agent source code.

Beyond Predefined Sinks

6.2

17

Program Analysis for Agent Security

Traditional program analysis provides several foundations for security-oriented dependency reasoning. Program Dependence Graphs (PDGs) capture control and data dependencies [12], while Code Property Graphs integrate syntax, control flow, and dependence information into a unified representation for vulnerability discovery [35]. Declarative systems such as QL further demonstrate that expressive program queries can scale to large codebases [4]. Dynamic information-flow tracking in TaintDroid [10] and static source–sink analyses such as FlowDroid [3] and YASA [30] provide complementary taint-oriented reasoning. Recent work adapts these techniques to LLM-integrated software. AgentFuzz [22] dynamically searches for inputs that drive agents toward sensitive operations. TaintP2X [14] statically traces LLM-generated values toward sensitive sinks, while Agent Audit [39] combines agent-aware data flow with credential, configuration, and privilege analysis. AgentSecGraph differs in its analytical objective. Rather than treating a predefined source–sink relation as sufficient to characterize security risk, it uses sensitive operations as candidate anchors and augments them with agent semantics, source provenance, dependency, trust-boundary, guard, and external-effect evidence. 6.3

Security Enforcement for LLM Agents

A complementary line of work aims to prevent unsafe agent behavior through isolation, informationflow control, and least-privilege enforcement. IsolateGPT [34] isolates LLM-integrated applications from system resources, CaMeL [8] separates trusted control flow from untrusted data, Progent [28] introduces programmable least-privilege policies over tool calls, and ACE [21] separates planning from execution while enforcing information-flow and capability boundaries. AttriGuard instead performs runtime causal attribution of tool invocations to identify actions driven by untrusted observations [15]. These systems primarily enforce security policies at runtime. AgentSecGraph is complementary: it does not enforce a policy, but statically records evidence about trust boundaries, guards, statically available privilege context, and external effects so that existing agent software can be audited. 6.4

Agent Dependency Representations and Positioning

Table 6. Positioning against closely related approaches. “Context” denotes explicit security-relevant evidence beyond operation identity.

Approach

Setting

Agent Dep.

Context

Primary Focus

AgentFuzz [22] TaintP2X [14] Agent Audit [39] AgentFlow [29] Agent-BOM [20]

Grey-box Static Static Static Runtime

No Partial Partial Yes Yes

Partial Partial Yes Partial Yes

Vulnerability triggering Prompt-to-sink detection Agent security auditing Dependency recovery Runtime reconstruction

AgentSecGraph

Static

Yes

Yes

Contextual behavior evidence

Agent frameworks encode relationships that conventional host-language representations do not fully capture. AgentFlow [29] builds Agent Dependency Graphs over agents, prompts, models, capabilities, memory, and control policies. Agent-BOM [20] combines static capabilities with runtime state to reconstruct security-relevant execution and attack chains. AgentSecGraph also

18

Hang Cui

uses a graph representation, but at a different granularity: the candidate-centered Security-ADG augments one sensitive operation with the dependencies and context needed to interpret it. These systems are complementary: taint analyses recover source-to-operation paths, agent-aware analyzers add security semantics, AgentFlow recovers framework dependencies, and Agent-BOM reconstructs runtime context. AgentSecGraph combines candidate-centered static recovery with explicit security context. 7

Conclusion

LLM-agent software connects model-generated decisions to security-sensitive capabilities, making operation identity alone insufficient for interpreting security-relevant behavior. We presented AgentSecGraph, a static analysis framework that treats sensitive operations as candidate anchors and constructs candidate-centered Security-ADG representations. By preserving agent relevance, provenance, dependencies, trust boundaries, candidate guards, available privilege information, and external effects, AgentSecGraph separates operation detection from security interpretation. We also introduced AgentSecBench, a reproducible corpus of 67 frozen repositories spanning 11 ecosystems and 37,542 source files. Applying the current analyzer to the full corpus yields 23,866 static security-sensitive operation candidates across 65 repositories, for which AgentSecGraph emits 23,866 candidate-centered Security-ADG artifacts without reported construction failures. Corpus-wide analysis recovers source-to-operation dependency evidence for 41.15% of candidates and potential guard evidence for 12.88%, completing in 50.8 minutes. A separate reproduction-backed evaluation establishes a conservative reference set of 22 behaviors: one confirmed vulnerability, one pending disclosure candidate, and 20 guarded behaviors that did not meet the vulnerability criteria. In the held-out study, Security-ADG preserves 91.1% of the reference context and all five observed guards, compared with 20.0% for a sink-only view and 40.0% for a simplified ADG; neither reduced view exposes the guards. These results support the central conclusion of this work: predefined sensitive operations remain useful analysis anchors, but security-critical behavior in LLM-agent software is interpreted more faithfully when operation identity is augmented with dependency and security-context evidence. Data and Artifact Availability The implementation of AgentSecGraph, the analysis pipeline, experiment configuration, evaluation scripts, and documentation are publicly available at https://github.com/A1LinLin1/ AgentSecBench. The repository is released under the Apache License 2.0. Security-sensitive reproduction details associated with ongoing coordinated disclosures are intentionally withheld. They will be released, where appropriate, after remediation or coordinated public disclosure. References [1] Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, J. Zico Kolter, Matt Fredrikson, Yarin Gal, and Xander Davies. 2025. AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents. In The Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=AC5n7xHuR1 [2] Anthropic. 2024. Model Context Protocol. https://modelcontextprotocol.io. [3] Steven Arzt, Siegfried Rasthofer, Christian Fritz, Eric Bodden, Alexandre Bartel, Jacques Klein, Yves Le Traon, Damien Octeau, and Patrick McDaniel. 2014. FlowDroid: Precise Context, Flow, Field, Object-Sensitive and Lifecycle-Aware Taint Analysis for Android Apps. In Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation. Association for Computing Machinery, 259–269. doi:10.1145/2594291.2594299

Beyond Predefined Sinks

19

[4] Pavel Avgustinov, Oege de Moor, Michael Peyton Jones, and Max Schäfer. 2016. QL: Object-Oriented Queries on Relational Data. In 30th European Conference on Object-Oriented Programming (ECOOP 2016) (Leibniz International Proceedings in Informatics (LIPIcs), Vol. 56). Schloss Dagstuhl–Leibniz-Zentrum fuer Informatik, 2:1–2:25. doi:10.4230/ LIPIcs.ECOOP.2016.2 [5] Harrison Chase. 2023. LangChain: Building Applications with LLMs through Composability. https://github.com/ langchain-ai/langchain. [6] Zhaorun Chen, Zhen Xiang, Chaowei Xiao, Dawn Song, and Bo Li. 2024. AgentPoison: Red-Teaming LLM Agents via Poisoning Memory or Knowledge Bases. In Advances in Neural Information Processing Systems, Vol. 37. [7] CrewAI. 2024. CrewAI: Framework for Orchestrating Role-Playing Autonomous AI Agents. https://github.com/ crewAIInc/crewAI. Accessed: 2026-08-07. [8] Edoardo Debenedetti, Ilia Shumailov, Tianqi Fan, Jamie Hayes, Nicholas Carlini, Daniel Fabian, Christoph Kern, Chongyang Shi, Andreas Terzis, and Florian Tramèr. 2026. Defeating Prompt Injections by Design. In 4th IEEE Conference on Secure and Trustworthy Machine Learning. https://arxiv.org/abs/2503.18813 [9] Edoardo Debenedetti, Jie Zhang, Mislav Balunović, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. 2024. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. In Advances in Neural Information Processing Systems, Vol. 37. doi:10.52202/079017-2636 [10] William Enck, Peter Gilbert, Byung-Gon Chun, Landon P. Cox, Jaeyeon Jung, Patrick McDaniel, and Anmol N. Sheth. 2010. TaintDroid: An Information-Flow Tracking System for Realtime Privacy Monitoring on Smartphones. In 9th USENIX Symposium on Operating Systems Design and Implementation (OSDI 10). USENIX Association. [11] Ivan Evtimov, Arman Zharmagambetov, Aaron Grattafiori, Chuan Guo, and Kamalika Chaudhuri. 2025. WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks. In Advances in Neural Information Processing Systems, Vol. 38. doi:10.52202/085713-0666 [12] Jeanne Ferrante, Karl J. Ottenstein, and Joe D. Warren. 1987. The Program Dependence Graph and Its Use in Optimization. ACM Transactions on Programming Languages and Systems 9, 3 (1987), 319–349. doi:10.1145/24039. 24041 [13] Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. Not What You’ve Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. arXiv preprint arXiv:2302.12173 (2023). [14] Junjie He, Shenao Wang, Yanjie Zhao, Xinyi Hou, Zhao Liu, Quanchen Zou, and Haoyu Wang. 2026. TaintP2X: Detecting Taint-Style Prompt-to-Anything Injection Vulnerabilities in LLM-Integrated Applications. In Proceedings of the IEEE/ACM 48th International Conference on Software Engineering. doi:10.1145/3744916.3773199 [15] Yu He, Haozhe Zhu, Yiming Li, Shuo Shao, Hongwei Yao, Zhihao Liu, and Zhan Qin. 2026. AttriGuard: Defeating Indirect Prompt Injection in LLM Agents via Causal Attribution of Tool Invocations. In 35th USENIX Security Symposium (USENIX Security 26). USENIX Association, 1547–1566. https://www.usenix.org/conference/usenixsecurity26/ presentation/he-yu [16] Xinyi Hou, Yanjie Zhao, Shenao Wang, and Haoyu Wang. 2026. Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directions. ACM Transactions on Software Engineering and Methodology 35, 10, Article 302 (2026), 37 pages. doi:10.1145/3796519 [17] Juhee Kim, Wenbo Guo, and Dawn Song. 2026. SoK: Attack and Defense Landscape of Agentic AI Systems. In 35th USENIX Security Symposium (USENIX Security 26). USENIX Association, 6047–6066. https://www.usenix.org/ conference/usenixsecurity26/presentation/kim-juhee-agentic [18] LangChain Inc. 2024. LangGraph: Build Resilient Language Agents as Graphs. https://github.com/langchainai/langgraph. Accessed: 2026-08-07. [19] Hwiwon Lee, Ziqi Zhang, Hanxiao Lu, and Lingming Zhang. 2025. SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks. In Advances in Neural Information Processing Systems, Vol. 38. doi:10. 52202/085713-3878 [20] Chaofan Li, Lyuye Zhang, Jintao Zhai, Siyue Feng, Xichun Yang, Huahao Wang, Shihan Dou, Yu Ji, Yutao Hu, Yueming Wu, Yang Liu, and Deqing Zou. 2026. Towards Security-Auditable LLM Agents: A Unified Graph Representation. arXiv preprint arXiv:2605.06812 (2026). doi:10.48550/arXiv.2605.06812 [21] Evan Li, Tushin Mallick, Evan Rose, William Robertson, Alina Oprea, and Cristina Nita-Rotaru. 2026. ACE: A Security Architecture for LLM-Integrated App Systems. In Proceedings of the 2026 Network and Distributed System Security Symposium. Internet Society. doi:10.14722/ndss.2026.230352 [22] Fengyu Liu, Yuan Zhang, Jiaqi Luo, Jiarun Dai, Tian Chen, Letian Yuan, Zhengmin Yu, Youkun Shi, Ke Li, Chengyuan Zhou, Hao Chen, and Min Yang. 2025. Make Agent Defeat Agent: Automatic Detection of Taint-Style Vulnerabilities in LLM-Based Agents. In 34th USENIX Security Symposium (USENIX Security 25). USENIX Association, 3767–3786. https://www.usenix.org/conference/usenixsecurity25/presentation/liu-fengyu

20

Hang Cui

[23] Yi Liu, Zhihao Chen, Yanjun Zhang, Gelei Deng, Yuekang Li, Jianting Ning, and Leo Yu Zhang. 2026. “Do Not Mention This to the User”: Detecting and Understanding Malicious Agent Skills in the Wild. In 35th USENIX Security Symposium (USENIX Security 26). USENIX Association, 1727–1746. https://www.usenix.org/conference/usenixsecurity26/ presentation/liu-yi [24] Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. 2024. Formalizing and Benchmarking Prompt Injection Attacks and Defenses. In 33rd USENIX Security Symposium (USENIX Security 24). USENIX Association, 1831–1847. https://www.usenix.org/conference/usenixsecurity24/presentation/liu-yupei [25] Jiaqi Luo, Jiarun Dai, Fengyu Liu, Songyang Peng, Youkun Shi, Tong Bu, Geng Hong, Xudong Pan, and Yuan Zhang. 2026. Autonomy Comes with Costs: Detecting Denial-of-Service Vulnerabilities Caused by Resource Abusing in LLM-Based Agents. In 35th USENIX Security Symposium (USENIX Security 26). USENIX Association, 3991–4010. https://www.usenix.org/conference/usenixsecurity26/presentation/luo [26] Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J. Maddison, and Tatsunori Hashimoto. 2024. Identifying the Risks of LM Agents with an LM-Emulated Sandbox. In International Conference on Learning Representations. [27] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools. In Advances in Neural Information Processing Systems, Vol. 36. [28] Tianneng Shi, Jingxuan He, Zhun Wang, Hongwei Li, Linyu Wu, Wenbo Guo, and Dawn Song. 2025. Progent: Securing AI Agents with Privilege Control. arXiv preprint arXiv:2504.11703 (2025). https://arxiv.org/abs/2504.11703 [29] Shenao Wang, Xinyi Hou, Yanjie Zhao, Xiao Cheng, and Haoyu Wang. 2026. AgentFlow: Building Agent Dependency Graphs for Static Analysis of Agent Programs. arXiv preprint arXiv:2607.01640 (2026). https://arxiv.org/abs/ 2607.01640 [30] Yayi Wang, Shenao Wang, Jian Zhao, Shaosen Shi, Ting Li, Yan Cheng, Lizhong Bian, Kan Yu, Yanjie Zhao, and Haoyu Wang. 2026. YASA: Scalable Multi-Language Taint Analysis on the Unified AST at Ant Group. In Companion Proceedings of the 34th ACM International Conference on the Foundations of Software Engineering. 524–535. doi:10. 1145/3803437.3805225 [31] Zhiqiang Wang, Yichao Gao, Yanting Wang, Suyuan Liu, Haifeng Sun, Haoran Cheng, Guanquan Shi, Haohua Du, and Xiangyang Li. 2026. MCPTox: A Benchmark for Tool Poisoning on Real-World MCP Servers. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40. 35811–35819. doi:10.1609/aaai.v40i42.40895 [32] Zihan Wang, Rui Zhang, Yu Liu, Wenshu Fan, Wenbo Jiang, Qingchuan Zhao, Hongwei Li, and Guowen Xu. 2026. MPMA: Preference Manipulation Attack Against Model Context Protocol. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40. 35838–35846. doi:10.1609/aaai.v40i42.40898 [33] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W. White, Doug Burger, and Chi Wang. 2024. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation. In Conference on Language Modeling. [34] Yuhao Wu, Franziska Roesner, Tadayoshi Kohno, Ning Zhang, and Umar Iqbal. 2025. IsolateGPT: An Execution Isolation Architecture for LLM-Based Agentic Systems. In Proceedings of the 2025 Network and Distributed System Security Symposium. Internet Society. [35] Fabian Yamaguchi, Nico Golde, Daniel Arp, and Konrad Rieck. 2014. Modeling and Discovering Vulnerabilities with Code Property Graphs. In 2014 IEEE Symposium on Security and Privacy. 590–604. doi:10.1109/SP.2014.44 [36] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In International Conference on Learning Representations. [37] Qiusi Zhan, Zhixiang Liang, Zifan Ying, and Daniel Kang. 2024. InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents. In Findings of the Association for Computational Linguistics: ACL 2024. Association for Computational Linguistics, 10471–10506. doi:10.18653/v1/2024.findings-acl.624 [38] Hanrong Zhang, Jingyuan Huang, Kai Mei, Yifei Yao, Zhenting Wang, Chenlu Zhan, Hongwei Wang, and Yongfeng Zhang. 2025. Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-Based Agents. In International Conference on Learning Representations. [39] Haiyue Zhang, Yi Nian, Aojie Yuan, and Yue Zhao. 2026. Agent Audit: A Security Analysis System for LLM Agent Applications. In Proceedings of the ACM Conference on AI and Agentic Systems (CAIS 2026). https://viterbiweb.usc.edu/~yzhao010/files/agent_audit_2026.pdf [40] Xuanjun Zong, Zhiqi Shen, Lei Wang, Yunshi Lan, and Chao Yang. 2026. MCP-SafetyBench: A Benchmark for Safety Evaluation of Large Language Models with Real-World MCP Servers. In International Conference on Learning Representations.

Beyond Predefined Sinks

21

Table 7. Aggregate composition of the frozen AgentSecBench corpus. Candidate counts characterize the current analyzer and are not vulnerability counts.

Corpus property Frozen repositories Agent ecosystems Source files Source size Primarily Python repositories Primarily TypeScript repositories Repositories in other primary languages Repositories with current-scan candidates Current full-corpus candidates

A

Value 67 11 37,542 291.2 MB 51 7 9 65 23,866

AgentSecBench Repository Inventory and Collection Criteria

AgentSecBench contains 67 publicly available LLM-agent repositories spanning 11 agent ecosystems. Corpus construction was performed independently of AgentSecGraph output. We retained repositories with substantive agent logic and analyzable source code, including MCP-based systems, coding agents, framework-based applications, autonomous agents, and multi-agent platforms. We excluded duplicates, mirrors, trivial examples, and repositories unsuitable for reproducible source-level analysis. Each retained repository is identified by its repository coordinate and an immutable Git commit. Candidate extraction, graph construction, reproduction, and evaluation use that same frozen snapshot. The machine-readable frozen corpus manifest in the public artifact repository provides the complete row-level repository inventory and commit identifiers; the PDF reports the aggregate composition in Table 7 rather than duplicating the long manifest. The nine repositories in the remaining-language group primarily use C#, Rust, Go, PHP, or Shell. Repository inclusion is not conditioned on candidate counts, graph construction outcomes, reproduction outcomes, vulnerability dispositions, or model-generated judgments. The frozen manifest is therefore an input to the evaluation rather than an output selected by the analyzer. B

Framework Normalization and Matching Rules

Agent frameworks expose tools and capabilities through heterogeneous source constructs. The adapter layer maps a matched construct to the normalized record N = ⟨𝐹, 𝑅𝑠 , 𝑌 , 𝑃𝑠 , 𝑄⟩,

(10)

where 𝐹 is the framework family, 𝑅𝑠 is the semantic role, 𝑌 is the associated callable or program symbol, 𝑃𝑠 is the set of exposed parameters, and 𝑄 records the matching evidence and confidence. Normalization determines how source constructs are represented; it does not assign vulnerability labels. Adapters apply frozen framework-specific selectors to decorators, calls, registrations, wrappers, workflow nodes, or tool classes. When available, module qualification and imported aliases are normalized before matching. A successful match records the construct form, associated symbol, source location, exposed parameters, and framework provenance. Ambiguous or unsupported constructs remain available to the language-level analysis but are not promoted to frameworkspecific evidence solely from a security label, model judgment, or reproduction outcome.

22

Hang Cui

Table 8. Framework construct families represented by the frozen adapter registry. Concrete selectors are versioned with the implementation.

Framework family Construct family

Normalized interpretation

MCP / FastMCP

Agent-facing callable and exposed parameters Tool capability associated with its callable

Tool decorators and server-side tool registration LangChain Tool decorators, callable wrappers, and toolkit exports CrewAI Tool declarations and tool collections attached to agents or tasks AutoGen Function or tool registration between conversational agents and executors OpenAI Agents SDK Function-tool wrappers and tools attached to an agent Semantic Kernel Kernel-function declarations and plugin methods LlamaIndex Function/query tool wrappers and tool collections

Agent-available capability Registered callable capability Agent-facing callable capability Kernel-exposed callable capability Agent- or query-facing callable capability

Framework normalization is independent of candidate-generation and graph schema decisions. Adding an adapter extends semantic recognition without changing the candidate-centered SecurityADG representation. The frozen adapter registry and its tests are part of the complete method implementation available in the public artifact repository. C

Generative-AI-Assisted Candidate Triage

We evaluated a frozen panel of five model raters over 150 candidate tasks. Each rater processed every task independently in a fresh single-turn request. Raters did not receive scanner confidence, detector identity, sampling weight, human annotations, or another model’s response. Tools, browsing, retrieval, memory, and code execution were disabled. Responses were required to follow a fixed JSON schema and were validated locally. Transport and schema failures were recorded rather than silently discarded. Four raters were accessed through OpenRouter using the exact routing slugs openai/gpt5-mini, anthropic/claude-sonnet-4.6, google/gemini-3.5-flash-lite, and qwen/qwen3max. Provider fallbacks were disabled, parameter support was required, and the run recorded the returned model identifier, generation identifier, upstream provider, token usage, cost, and system fingerprint. The fifth rater, deepseek-v4-pro, was accessed through the official DeepSeek API rather than OpenRouter. The completed panel contains 750 valid task–rater responses (150 tasks times five raters). The panel was used only for pre-annotation and disagreement-based triage. Inter-model agreement was not interpreted as human inter-annotator agreement or as evidence of semantic correctness. Panel responses were excluded from Security-ADG construction and did not establish behavior or vulnerability ground truth. For M1–M4, requests used https://openrouter.ai/api/v1/chat/completions with allow_fallbacks=false, require_parameters=true, and data_collection=deny. Zerodata-retention endpoints were preferred, but the recorded run explicitly allowed non-ZDR endpoints under the data-collection-denial policy. The recorded upstream providers were OpenAI, Anthropic, Google AI Studio, and Alibaba, respectively, and provider consistency checks passed.

Beyond Predefined Sinks

23

Table 9. Frozen five-model pre-annotation panel.

Rater Exact identifier and interface

Inference and output settings

M1

openai/gpt-5-mini; OpenRouter

M2

anthropic/claude-sonnet-4.6; OpenRouter

M3

google/gemini-3.5-flash-lite; OpenRouter

M4 M5

qwen/qwen3-max; OpenRouter deepseek-v4-pro; DeepSeek official API

Medium reasoning; reasoning excluded; JSON Schema Medium reasoning; reasoning excluded; JSON Schema Medium reasoning; reasoning excluded; JSON Schema Provider-default reasoning; JSON Schema Thinking enabled, high; reasoning not stored; JSON object with local schema validation

For M5, requests used https://api.deepseek.com/chat/completions with a separate API credential. Because DeepSeek’s JSON mode did not enforce the supplied JSON Schema, its outputs underwent the same local schema validation as the other responses. No API keys, account identifiers, billing records, or author-identifying generation URLs are included in the artifact. All model-generated outputs used in the study were validated by the authors or by deterministic validation scripts. Model judgments were not treated as vulnerability ground truth. The authors remain fully responsible for the study design, implementation, analysis, and claims. D

Held-Out Context-Preservation Study

We evaluate three representation projections over nine held-out evidence-backed cases: five validated by controlled local monkeypatch proofs and four by source-level construction proofs that did not execute the external effect. Each case-level reference contains independently supported context dimensions for operation, external effect, dependency path, trust boundary, and guard posture. The four cases recorded as no_guard_confirmed contain an explicit negative guard-posture observation rather than an absent or unknown field; the guard-posture dimension therefore remains in the denominator. Positive guard-visibility credit is awarded only when concrete guard evidence is exposed by the representation. These measurements concern representation retention, not vulnerability-detection accuracy. The micro totals reproduce the values reported in the main paper: 9/45 = 20.0%, 18/45 = 40.0%, and 41/45 = 91.1%. Security-ADG exposes concrete guard evidence in all five guarded cases. The four remaining reference records explicitly report no_guard_confirmed; because the graph does not emit a positive guard node for these cases, they contribute no guard-retention credit. E

Corpus-Wide Characterization and Scalability

This section provides the detailed corpus-wide characterization underlying RQ1 in the main paper. All measurements in this section use the current full-corpus run: the current candidate extractor applied to all 67 frozen repositories in AgentSecBench. This population is distinct from the historical 531-candidate frozen pilot cohort and the 508 candidates obtained by re-running the current scanner on the same five pilot repositories. The current full-corpus run identifies 23,866 static security-sensitive operation candidates from 7,214 source files. Sixty-five of the 67 repositories contain at least one candidate. All 67 repositories complete candidate extraction and graph construction without reported skips or failures. Candidate counts in this section are analysis populations and must not be interpreted as vulnerability counts.

24

Hang Cui

Table 10. Evidence provenance for the held-out context-preservation cases. Repository identifiers are retained to make the frozen evidence auditable; no row is a vulnerability claim. Case

Repository

HSQ-0014 dddabtc/winremotemcp HSQ-0087 dddabtc/winremotemcp HSQ-0089 dddabtc/winremotemcp HSQ-0091 dddabtc/winremotemcp HSQ-0103 dddabtc/winremotemcp HSQ-0211 nexu-io/htmlanything HSQ-0298 nexu-io/htmlanything HSQ-0300 nexu-io/htmlanything HSQ-0302 nexu-io/htmlanything

Behavior

Evidence mode

Guard posture

shell command execution application launch

local monkeypatch proof local monkeypatch proof local monkeypatch proof local monkeypatch proof local monkeypatch proof source-level construction proof source-level construction proof source-level construction proof source-level construction proof

no_guard_ confirmed no_guard_ confirmed guarded

desktop notification script network diagnostic process service control agent CLI discovery process external content file write external tarball extraction validated skill file write

no_guard_ confirmed guarded no_guard_ confirmed guarded guarded guarded

Table 11. Case-level representation retention. Each case has five applicable reference dimensions: operation, effect, dependency, trust boundary, and guard posture. Case

Guard posture

Applicable

Sink-only

Simplified ADG

Security-ADG

HSQ-0014 HSQ-0087 HSQ-0089 HSQ-0091 HSQ-0103 HSQ-0211 HSQ-0298 HSQ-0300 HSQ-0302

no_guard_confirmed no_guard_confirmed guarded no_guard_confirmed guarded no_guard_confirmed guarded guarded guarded

5 5 5 5 5 5 5 5 5

1/5 1/5 1/5 1/5 1/5 1/5 1/5 1/5 1/5

2/5 2/5 2/5 2/5 2/5 2/5 2/5 2/5 2/5

4/5 4/5 5/5 4/5 5/5 4/5 5/5 5/5 5/5

Micro total

–

45

9/45

18/45

41/45

Table 12. Micro-averaged context completeness and positive guard visibility. Representation

Retained

Completeness

Guard visibility

Sink-only Simplified ADG Security-ADG

9/45 18/45 41/45

20.0% 40.0% 91.1%

0/5 0/5 5/5

Beyond Predefined Sinks

25

Table 13. Aggregate evidence recovered in the current full-corpus run. Rates use all 23,866 static candidates as the denominator. Candidate counts are not vulnerability findings.

Evidence or artifact

Count

Rate

Framework-specific evidence Source-to-operation dependency evidence Potential guard evidence Generated Security-ADG artifacts

749 9,821 3,075 23,866

3.14% 41.15% 12.88% 100.00%

Table 14. Corpus-wide evidence by security-sensitive operation category. Percentages are relative to the number of candidates in each row. Category browser_control command_execution credential_access database_access dynamic_code_execution external_tool_invocation filesystem_delete filesystem_read filesystem_write message_or_email_send network_access permission_or_auth_change Total

E.1

Candidates

Dependency

Guard

Trust-crossing

588 2,722 2,548 1,304 55 5,344 583 4,182 5,013 164 1,157 206

11 (1.87%) 933 (34.28%) 251 (9.85%) 106 (8.13%) 24 (43.64%) 603 (11.28%) 235 (40.31%) 2,730 (65.28%) 4,218 (84.14%) 70 (42.68%) 553 (47.80%) 87 (42.23%)

13 (2.21%) 315 (11.57%) 454 (17.82%) 36 (2.76%) 23 (41.82%) 73 (1.37%) 299 (51.29%) 1,062 (25.39%) 632 (12.61%) 32 (19.51%) 121 (10.46%) 15 (7.28%)

11 (1.87%) 933 (34.28%) 251 (9.85%) 106 (8.13%) 24 (43.64%) 603 (11.28%) 235 (40.31%) 2,730 (65.28%) 4,218 (84.14%) 70 (42.68%) 553 (47.80%) 87 (42.23%)

23,866

9,821 (41.15%)

3,075 (12.88%)

9,821 (41.15%)

Aggregate Security-ADG Evidence

Table 13 summarizes the evidence exposed by the current full-corpus analysis. For every candidate, AgentSecGraph emits one candidate-centered Security-ADG artifact linked to its candidate identifier and source provenance. The exported corpus-wide aggregates contain identical candidate counts for source provenance, dependency paths, and trust-crossing fields: 9,821 candidates in each case. These exported counts do not establish that the three fields were recovered by independent analytical events. We therefore report the consolidated source-to-operation dependency evidence measure in Table 13 rather than treating the three identical counts as independent results. The current extraction rules recover no explicit privilege-context evidence in this run. This does not imply that the analyzed programs execute without privilege; rather, AgentSecGraph records privilege context only when it can be derived from supported source-level constructs. Likewise, every candidate receives an abstract external-effect node based on its operation category. The resulting 100% external-effect-node coverage is therefore a schema property rather than evidence that the modeled effect occurred at runtime. E.2

Evidence by Operation Category

Table 14 reports candidate counts and the frequency of recovered dependency, guard, and trustcrossing fields for each operation category. The category-level percentages characterize the current analyzer and corpus only. They do not estimate vulnerability prevalence or category-specific security risk.

26

Hang Cui

Table 15. Scalability measurements from one completed full-corpus run. The measurements characterize the observed execution environment and are not presented as cross-system performance estimates.

Metric Frozen repositories processed Repositories with candidates Candidate-bearing source files Static candidates Generated Security-ADG artifacts Candidate extraction time Candidate extraction throughput Graph construction time Graph construction throughput End-to-end wall time End-to-end throughput Median repository time P90 repository time P95 repository time

Observed result 67 / 67 65 7,214 23,866 23,866 1,508.456 s 15.821 candidates/s 1,538.164 s 15.516 candidates/s 3,046.620 s (50.78 min) 7.834 candidates/s 9.562 s 203.008 s 236.191 s

Dependency evidence is strongly category-dependent. It is recovered for 84.14% of filesystemwrite candidates and 65.28% of filesystem-read candidates, compared with 11.28% of external-tool invocations and 1.87% of browser-control candidates. Guard evidence is likewise uneven: 51.29% of filesystem-delete candidates and 41.82% of dynamic-code-execution candidates contain potential guard evidence, whereas only 1.37% of external-tool invocations do. These differences should not be interpreted as evidence that one operation category is more vulnerable than another. They reflect the interaction among the current source-analysis rules, the coding patterns in the frozen corpus, and the evidence that can be recovered statically. As noted above, dependency and trust-crossing counts are identical in the exported corpus-wide aggregate. The trust-crossing column is retained in Table 14 for auditability of the archived results, but we do not interpret it as an independently validated trust-boundary recovery rate. E.3

Scalability

The full-corpus timing experiment was executed using one analysis process with no worker pool. Candidate extraction and graph construction were timed without tracemalloc so that profiler overhead did not inflate the reported wall-clock measurements. Table 15 summarizes the completed run. The per-repository timing distribution is strongly skewed. Although the median combined analysis time is 9.562 seconds, P90 and P95 increase to 203.008 and 236.191 seconds, respectively. Reporting these quantiles is important because an arithmetic mean alone would obscure the large-repository tail. The completed timing run predates formal process-level peak-RSS collection. We therefore do not report a peak-memory result for this run. A separate earlier tracemalloc experiment on the frozen 531-candidate pilot cohort measured Python allocation behavior, but its profiler-inflated wall time is not comparable with the ordinary full-corpus timing experiment and is not used in the main scalability result.

Beyond Predefined Sinks

27

Table 16. Category distribution of the frozen 344-operation held-out inventory. The inventory is finite and is not treated as exhaustive repository-wide ground truth.

Category

E.4

Count

Command execution Filesystem write External tool invocation Network access Filesystem read Browser control Dynamic code execution Filesystem delete

76 62 58 46 43 27 17 15

Total

344

Population and Reproducibility Notes

The full-corpus measurements above must not be compared directly with the historical 531candidate frozen pilot cohort as though repository count were the only changing variable. The pilot manifest was generated by an earlier scanner configuration. Applying the current scanner to the same five frozen repository snapshots yields 508 candidates because subsequent language scoping and AST filtering changed candidate extraction. We therefore preserve the three populations separately: • Frozen pilot cohort: 531 candidates from five repositories, retained for compatibility with historical pilot and annotation studies; • Current five-repository scan: 508 candidates on the same repository snapshots, used only to document scanner-version drift; and • Current full-corpus run: 23,866 candidates from all 67 frozen repositories, used for the corpus-wide characterization and scalability results reported above. None of these candidate populations constitutes a vulnerability count, and this descriptive corpus-wide experiment does not establish repository-wide recall. F

Inventory-Bounded Evaluation

The main evaluation uses corpus-wide descriptive measurements, reproduction-confirmed behavior evidence, and a held-out representation-preservation study. We additionally retain an auxiliary source-level operation inventory to provide bounded evidence about operation recovery on code that was separated from development-time method tuning. This evaluation is deliberately described as inventory-bounded. The finite inventory does not establish that every relevant operation in either repository has been enumerated and therefore cannot support claims of repository-wide exhaustive recall. F.1

Held-Out Operation Inventory

The frozen inventory contains 344 source-level operation records from two repositories: • dddabtc/winremote-mcp: 148 operation records; and • nexu-io/html-anything: 196 operation records. The two repositories were held separate from development-time method tuning when this inventory was constructed. The inventory covers eight security-sensitive operation categories.

28

Hang Cui

The category distribution is not intended to estimate the prevalence of security-sensitive operations in the broader LLM-agent ecosystem. It describes only the frozen auxiliary inventory used for this bounded evaluation. F.2

Reference Evidence

Inventory entries are tied to frozen source locations and are reviewed using inspectable source-level evidence. Where stronger validation is available for a corresponding behavior, the evaluation distinguishes among the following evidence levels: • Source-level evidence, establishing that a relevant program construction or dependency exists in the frozen source snapshot; • Stub or monkeypatch evidence, establishing which call or argument is produced under a controlled input; • Isolated execution evidence, establishing that the corresponding external effect occurs in a controlled environment; and • Attack–defense paired evidence, evaluating whether a candidate protection mechanism constrains the corresponding behavior. These evidence levels are not interchangeable. In particular, the 344-operation inventory primarily serves as a source-level operation inventory and is not itself a set of confirmed vulnerabilities. Model-generated judgments are not treated as reference labels. F.3

Prediction-to-Reference Matching

Method outputs are matched to inventory entries using the frozen matching rule match = repository + commit + file + operation category + line overlap.

(11)

No label-dependent line tolerance or manual per-item matching is introduced after evaluation. A reference inventory record is considered recovered only when a method output satisfies the frozen matching rule. Let 𝐼 denote the 344-operation reference inventory and let 𝑀 denote the set of method outputs matched to records in 𝐼 . We define inventory-bounded recall as 𝑅inv =

|𝑀 | . |𝐼 |

(12)

Because 𝐼 is a finite reviewed inventory rather than an exhaustive enumeration of all possible security-relevant operations in the two repositories, 𝑅inv must not be interpreted as repository-wide recall. F.4

Evaluation Status and Claim Boundary

The frozen inventory defines the finite population and the coordinate-level matching protocol used by the evaluator. At the time of this study, however, the independently adjudicated label file does not cover all 344 inventory items. The evaluator therefore refuses to emit aggregate or category-level recovery metrics, and we do not report matched counts, recall, precision, or F1 for this population. Results from the smaller reviewed sample are not substituted for full-inventory measurements. The inventory is retained here to document the evaluation population and protocol; completing the independent adjudication remains future work.

Beyond Predefined Sinks

F.5

29

Interpretation and Limitations

This auxiliary evaluation serves a narrower purpose than the corpus-wide characterization in Section E. The latter measures how the current analyzer behaves over all 67 frozen repositories, whereas the 344-operation inventory provides a finite source-level denominator on two held-out repositories. The inventory therefore supports statements about recovery relative to its reviewed records, but not claims of exhaustive repository-wide recall. Likewise, an operation matched in this evaluation is not necessarily agent-controlled, exploitable, or vulnerable. Behavior-level and vulnerability conclusions remain governed by the separate reproduction-backed evidence layer described in the main paper.

Related documents

• Beyond Predefined Sinks: Security-Aware Dependency Analysis for LLM Agents#1187692
• Towards a Systems Foundation for Agentic Skills: Architecture, Lifecycle, and Security#798236
• Towards a Systems Foundation for Agentic Skills: Architecture, Lifecycle, and Security#796823
• Beyond the Model: Demystifying Harness Effects in Software Engineering Agents#1116790
• Evaluating LLM Agents on Automated Software Analysis Tasks#10421