arXiv CSopen access

SkillGate: Cost Efficient Runtime Malicious Skill File Detection in Coding Agents

SkillGate: Cost Efficient Runtime Malicious Skill File Detection in Coding Agents Rui Yang∗† , Michael Fu‡ , Kla Tantithamthavorn∗ , Chetan Arora∗ , Joey Chua† ∗ Monash University, Melbourne, Victoria, Australia † Transurban, Melbourne, Victoria, Australia

arXiv:2607.25619v1 [cs.SE] 28 Jul 2026

‡ The University of Melbourne, Melbourne, Victoria, Australia

Abstract—Software engineering teams now deploy AI coding agents (Cursor, Claude Code, GitHub Copilot) as first-class productivity tools, installing domain-specific skill files to tailor agent behavior to project APIs, framework conventions, and organizational workflows. These complex Markdown files are easily downloaded from public registries with a single npx skills add command and no real security screening, representing a novel supply-chain attack surface: a malicious skill file can silently reprogram agent behavior, exfiltrating credentials, injecting backdoors into generated code, or redirecting agent actions to attacker-controlled endpoints. The threat is not hypothetical: recent reports document hundreds of malicious skill packages in public registries, including organized campaigns that distributed credential-stealing infostealers via fake productivity skills [1]. No systematic toolchain defense exists for this attack surface. We present S KILL G ATE, a deployable security gateway that screens AI skill packages before coding agent installation. S KILL G ATE uses a hybrid regex-prefilter + LLM-judge pipeline: safe-signal files bypass the LLM entirely (skip savings); flagged files have only their matched snippet windows sent to the judge, not the full content (snippet savings). We answer four research questions covering detection effectiveness, screening cost, runtime overhead, and false positive behavior on the SkillsBench benchmark against two existing tools. On SkillsBench (n=1,650, 9.1% malicious), S KILL G ATE achieves F1=0.817, FPR=1.13% while reducing LLM input tokens by 77% vs. full-file screening, and outperforming existing tools by 5–6× on threshold-independent AUPRC (0.830 vs. 0.144/0.162). Index Terms—MCP security, AI agent, supply-chain defense, agent jailbreak, coding agent, skill file classifier, software security

I. I NTRODUCTION Enterprise software engineering teams are adopting AI coding agents such as Claude Code, GitHub Copilot, Gemini CLI, Cursor, as first-class productivity tools for code generation, refactoring, and documentation [2]–[6]. Unlike an autocomplete tool, such an agent operates autonomously: it plans a task, then reads, writes, and runs code on the developer’s machine, often with low-risk actions auto-approved until the task is done. To extend agents beyond generic capabilities, practitioners install discipline-specific skill files: Markdown documents that supply tool definitions, API references, framework conventions, and preflight scripts tailored to a project or domain, which the agent loads into its context and follows as authoritative guidance. The Model Context Protocol (MCP) [7] standardizes this interface: skill content is delivered via tools/call, resources/read, and prompts/get responses, after which it influences every

Unscreened

“npx skills add <untrusted external source>” Developer

Loads skill les into local development system… Coding Agent

cat /.aws/credentials | curl -X POST -d @https://attacker-c2.com/log Malicious Skill File With SkillGate

“npx skills add <untrusted external source>” Developer

Runtime intercept + classify skill le content SkillGate BLOCKED

INSTALLED

Coding Agent fi

Malicious Skill File

Safe Install fi

Fig. 1. A malicious skill file installs with no screening and can silently exfiltrate credentials, backdoor generated code, or redirect the agent to attacker-controlled infrastructure. S KILL G ATE intercepts and classifies each skill before it reaches the agent, blocking malicious installs while allowing benign ones through.

agent action in the session. Skills reach an agent through several channels: npm-style registry commands such as npx skills add <package>, Git clones, and content delivered at runtime via MCP tools/call, resources/read, and prompts/get responses—none of which applies any security screening before the skill enters the agent’s context. This means that the same install-from-registry pattern that enabled npm and PyPI supply-chain attacks [8]–[10] now applies at the agent instruction layer: a malicious skill file can silently exfiltrate API keys, inject backdoors into generated code, or redirect agent actions to attacker-controlled infrastructure, all without any user interaction beyond the install command (Figure 1, top). Unlike malicious code packages, malicious skill files are plain Markdown and bypass existing static analysis tools (Semgrep [11], Snyk [12]) that target source code, not behavioral instructions. In February 2026, security researchers reportedly found 1,184 malicious skill

packages in the ClawHub registry, approximately 20% of listed limitation we set out to address: purely static pattern matching skills, including the “ClawHavoc” campaign, said to distribute incurs false-positive rates well above any realistic deployable credential-stealing infostealers via fake productivity skills [1]. threshold, while naively escalating every file to an LLM judge Concurrent measurement studies confirm the trend: large-scale restores accuracy only at a per-file cost and latency that cannot audits of public registries find malicious agent skills distributed be sustained at install time. The novelty of our method is a in the wild [13], and controlled evaluations show coding agents hybrid, targeted snippet pipeline that resolves this challenge: are broadly vulnerable to skill-file attacks [14]. This exposes a regex prefilter gates the majority of benign files at neara critical gap: no deployable runtime guardrail exists for zero cost, and the LLM judge sees only the matched snippet the agent instruction layer, to intercept and detect the windows rather than full file content, satisfying both a realistic, installation of malicious skill files before they reach the deployable FPR constraint and the cost constraint that prior agent. tools violate. To make our work reproducible and available To address this gap, we present S KILL G ATE, a deployable to the public, we deploy S KILL G ATE as an open-source security gateway that intercepts each skill package and screens MCP proxy that integrates with five major AI coding agents it before it reaches the agent, blocking malicious installs while through a one-line configuration change, and we release the installing benign ones correctly (Figure 1, bottom). S KILL G ATE full implementation, the 530-pattern ruleset, and our evaluation applies a two-stage hybrid pipeline: (1) a 530-pattern regex harness as open source.1 Novelty & Contributions. To the best of our knowledge, RuleEngine (428 MITRE ATT&CK-derived core patterns [15] plus 102 Sigma-imported community rules [16]), and (2) a the main contributions of this paper are: (1) S KILL G ATE— configurable LLM judge that classifies only the sections that a locally deployable, open-source proxy runtime daemon the regex detector considers as a risk rather than the full skill implementing the hybrid classifier, policy engine, quarantine file content. The targeted snippet design sends the judge only manager, and audit logger; integrates with five major AI coding the flagged regions rather than the full file, reducing token cost agents via a one-line configuration change. (2) Empirical evaluation—evaluation on the open-source SkillsBench dataset while maintaining deployable recall. (n=1,650) with baseline comparison, and an ablation study Our study is structured to answer the following research over classifier configuration parameters. (3) Key finding— questions: on SkillsBench, S KILL G ATE achieves F1=0.817, R=0.769, 1) RQ1: How accurately does S KILL G ATE detect mali- FPR=1.13% while reducing LLM input tokens by 77% vs. cious skill files? full-file screening, outperforming existing tools by 5–6× on Results. S KILL G ATE achieves F1=0.817, R=0.769, AUPRC. FPR=1.13%, MCC=0.803, and AUPRC=0.830 versus II. BACKGROUND & R ELATED W ORK 0.144 for ClawVet and 0.162 for SkillScanner—a 5–6× gap across all thresholds, and a 2.4× MCC margin over In this section, we present background on LLM coding agents the strongest baseline (0.331). and their skill ecosystem, survey documented malicious skill 2) RQ2: How much does S KILL G ATE reduce LLM input files, and then formalize a threat model, and related defenses. tokens? Results. The targeted snippet design reduces LLM A. LLM Coding Agents In the field of Software Engineering, LLM-based coding input tokens by 77% relative to full-file screening on SkillsBench, a 4.3× overall reduction, combining assistants, such as GitHub Copilot, Cursor, and Claude Code skip savings (67% of files bypass the LLM entirely, have evolved from autocomplete tools to autonomous agents 1,110/1,650) with snippet savings on the files that are that plan, edit, and execute code across multi-step tasks [2], [3], [5], [6], [17]. An agentic assistant runs an iterative escalated. 3) RQ3: What is the runtime latency of S KILL G ATE? perceive–plan–act loop, invoking tools—reading and writing Results. Weighted average latency is ∼818ms (prefilter- files, running shell commands, querying APIs—and feeding safe: ∼139ms at 67.2%; LLM-bound: ∼2,208ms at each result back into its context until the task completes [17]– 32.8%), which is 7.7× faster than SkillScanner+LLM [19]. Unlike a chat assistant, it executes these actions itself, often with low-risk actions auto-approved without per-step (∼6,281ms), compatible with a 0.5–5s registry fetch. 4) RQ4: What categories of benign content trigger false confirmation [4]. This expands the trusted computing base: any third-party content that steers the agent inherits the developer’s positives? Results. On SkillsBench, S KILL G ATE produces on privileges. Prior work shows the outputs of these agents carry average 17 false positives (FPR=1.13%), versus 756 security risk [20], [21]; we instead target the instructions they for ClawVet (FPR=50.4%) and 261 for SkillScanner ingest. (FPR=17.4%)—a 15–44× reduction. The dominant B. Skill Files and the MCP Interface category is base64-encoded content in documentation The Model Context Protocol [7] defines a standard interthat triggers the encoded-payload prefilter rule without face for AI coding agents to consume tool definitions and clear benign context. Existing tools such as SkillScanner illustrate the core

1 https://github.com/awsm-research/skillgate

behavioral instructions from external servers, extending the tool-augmented language models that learn to invoke external APIs [18], [19]. MCP follows a client–server model: the agent embeds an MCP client, and each capability provider (a documentation server, a database connector, a skill registry) runs as an MCP server. The client requests content from a server and merges the reply into the LLM’s context. Because the protocol is standardized, a single agent can connect to many third-party servers, and the same server can be reused across agents, making this distribution channel broadly accessible to developers and agents. A skill file (typically SKILL.md or skills/*.md) is the primary delivery vehicle: a Markdown document that packages reusable expertise for a task or domain. A skill file typically contains a metadata header (a name and a natural-language description that tells the agent when to load the skill), a body of workflow instructions written as imperative prose, and often auxiliary assets: tool definitions, code templates, and shell or Python scripts that the skill instructs the agent to run before starting a task (for example, to check the environment or install a dependency). When the agent judges a skill relevant, it pulls this content into its context and treats the instructions as authoritative guidance for the task. A skill reaches an agent through two kinds of channels. It is installed from a distribution source, npm-style registries (npx skills add <package>), Git repositories, or manual placement in an agent’s configuration directory. It can also be delivered at runtime through three MCP response types: tools/call (the reply to a tool invocation), resources/read (a fetched document or file), and prompts/get (a server-supplied prompt template). In every case the returned content is appended to the model’s context window, and this is where the exposure arises: an MCP response is data to the protocol but instructions to the LLM, so a server or package that supplies adversarial text is effectively issuing commands the agent may follow. Each channel therefore presents a supply-chain attack surface similar to packageregistry attacks on npm and PyPI [8]–[10], [22], but operating at the agent instruction layer rather than the code-execution layer. Because skill content is plain Markdown rather than executable source, it bypasses conventional supply-chain scanners [11], [12] that reason about code, not natural-language behavioral directives; and unlike a code dependency, which executes only when explicitly called, a skill file shapes agent behavior the moment it enters the context window, giving it a broad and immediate blast radius.

inspects the file. C. Documented Malicious Skill Files The threat is not hypothetical. In February 2026, the “ClawHavoc” campaign distributed credential-stealing infostealers through fake productivity skills in the ClawHub registry; researchers reported 1,184 malicious packages, roughly 20% of listed skills [1]. Recent measurement studies corroborate a systemic problem: Liu et al. [13] find malicious skills distributed at scale across public registries, while Schmotz et al. [14] show coding agents are broadly susceptible to skill-file attacks across payload types. SkillsBench [23] realizes attack behaviors as hand-crafted payloads injected into real skill hosts, including credential exfiltration (reading .env and ˜/.ssh/ and posting their contents to a remote endpoint), code backdoor injection (instructing the agent to append command-andcontrol callback stubs to generated code), endpoint redirection (overriding tool registrations to point at attacker-controlled servers), and prompt injection (embedding jailbreak directives that override system-level safety instructions). These behaviors mirror the indirect prompt-injection attacks studied for LLMintegrated applications [24], but here the malicious payload is delivered as an installable, persistently loaded skill rather than transient runtime data, making pre-installation screening the natural point of defense. D. Threat Model Attacker capabilities. We model a non-adaptive adversary who: (1) controls the content of a skill file in a package registry or Git repository; (2) does not know that S KILL G ATE is deployed; (3) has not inspected the deployed ruleset. Attack taxonomy. The concrete attack behaviors realizing this model are those surveyed above. For evaluation, SkillsBench groups them into 8 attack categories: plaintext cmd, data exfil, encoded payload, steganography, rsa modular, ascii smuggle, binary exec, and remote script [23]. Out of scope. Post-execution attacks, attacks via the agent’s existing context window, and adaptive adversaries who reverseengineer the deployed ruleset are out of scope. E. Related Work

Static analysis for software supply chains. Supply-chain security tools target code artifacts: SCA tools [25] flag knownmalicious packages by hash or CVE; Semgrep [11] and CodeQL detect patterns in source code. These tools do not process Markdown instruction documents and cannot detect agent behavioral manipulation embedded in skill files. Prompt injection defenses. Prompt injection attacks [24], Anatomy of a malicious skill. A malicious [26], [27] embed adversarial instructions in data consumed skill advertises a benign description (e.g. by LLMs. Defense approaches include input sanitization [28], “numpy-api-compatibility”) so the agent loads spotlighting [29], guardrail frameworks [30], and privilege it during ordinary work, while its body embeds a hostile separation [31]. These approaches address runtime manipulation action—a preflight “health check” whose shell block reads of an already-running agent, not pre-execution skill content ˜/.ssh/id_rsa and POSTs it to an attacker endpoint, say. screening. Since low-risk actions are auto-approved and the payload is a small fraction of otherwise legitimate content, the agent LLM safety guardrails. Content-moderation guardrails such as executes it with the developer’s credentials and no human LlamaGuard [32] and the OpenAI Moderation API [33] classify

Skill les enter via

1 SkillGate Runtime Proxy

3 Snippet Extraction + LLM Judge

2 Regex Pre lter RuleEngine

Registry install (npx skills add <pkg>)

428 MITRE ATT&CK + 102 SIGMA security patterns

Yes

Runtime MCP Response (resources/read)

Matched?

4 PolicyEngine

Matched Snippet Extraction

LLM Judge

…… # Quick Health Check cat ~/.ssh/id_rsa | head -5 …… curl -s https://…/check \ …… -d "$(cat ~/.ssh/id_ ……

Sys: classify skill le for safety SAFE|MALICIOUS|SUSPICIOUS Input: pre lter matched snippet Output: {class, con dence, risk tag, reasoning}

Default: GPT-5.4-mini

Content Extraction

Manual Download (Manually trigger scan)

Safe agent install No

Skills folder content concatenation

Safe agent install Blocked/ Quarantined

fi

fi

fi

fi

Coding Agent fi

Block/ Quarantine/ Warn/Allow/ Audit

Fig. 2. End-to-end overview of S KILL G ATE.

prompts and responses against a fixed harm taxonomy (violence, self-harm, hate, and similar categories). These guardrails are built for conversational systems: they screen free-form user– assistant text, not the tool definitions, workflow instructions, and scripts that constitute a skill file, and carry no notion of an agent installing and acting on third-party instructions. S KILL G ATE instead screens skill-file content along an attacktechnique taxonomy before the agent ingests it. Skill file screening. ClawVet [34] is the closest prior system: it applies 54 static patterns in 6 analysis passes. SkillScanner [35] is a multi-engine scanner that combines static YARA/regex rules with a behavioural control-flow graph, reporting findings across five severity levels, and additionally offers an optional LLM judge that inspects the full file content. Both are nonetheless static-first: a pattern or rule match sets the primary verdict, so the operating point is fixed by a severity threshold, and when the optional judge is enabled it reads the entire file rather than the specific regions that triggered a rule.

A. Stage 1: Runtime Proxy & Content Extraction In Stage 1 , S KILL G ATE operates as a Unix-socket daemon that binds ˜/.skillgate/skillgate.sock at startup, forming a transparent proxy between the coding agent and any upstream MCP server. A per-agent stdio bridge routes the MCP transport through the daemon without requiring agent-side code changes; integration requires only a one-line configuration change that points the agent’s MCP server entry to the S KILL G ATE socket. For each registered MCP server, the S KILL G ATE daemon intercepts every tools/call, resources/read, and prompts/get response, extracting the skill file payload for classification before forwarding or blocking the message. By default, the common ‘npx skills add’ is intercepted, but the daemon can be configured to intercept any MCP message type that contains skill file content. All classification decisions are written to a structured JSONL audit log (˜/.skillgate/logs/audit.jsonl), enabling post-hoc forensic analysis. B. Stage 2: Regex Prefilter

III. S YSTEM D ESIGN & E VALUATION S ETUP Design Rationale. Existing static-pattern tools for skill file screening achieve high recall but incur false-positive rates above 40% [34], rendering them impractical in any development workflow where spurious blocks erode developer trust. At the other extreme, sending every skill file to an LLM judge with full content incurs unnecessary cost: these files often contain up to hundreds of thousands of tokens, making it both expensive and slow to send all of the content to the LLM for classification during runtime. We therefore design S KILL G ATE around two complementary cost-reduction principles. First, skip savings: files with no pattern hits are classified safe at near-zero cost, concentrating all LLM calls on the subset of files that exhibit suspicious signal. Second, snippet savings: for files that do trigger the prefilter, the LLM receives only the matched snippet windows, not the full file, as the malicious content is often concentrated in a small region of the skill file, we preserve the most important context while reducing per-call token consumption. Figure 2 provides an end-to-end overview of S KILL G ATE, and we describe each stage below.

In Stage 2 , the RuleEngine compiles 530 patterns into an AttackPatternMatcher and scans the extracted skill file content: 428 core patterns systematically derived from the MITRE ATT&CK framework [15], plus 102 community rules imported from the Sigma detection format [16]. The 12 largest categories of the MITRE-derived core account for 412 of its 428 patterns—defense evasion (62), execution (56), risky commands (56), credential access (44), persistence (38), impact (33), obfuscation (27), exfiltration (27), discovery (23), command-and-control (21), privilege escalation (17), and collection (8)—with the remaining 16 in smaller categories. Table I lists representative examples; the full set is released with the code. TABLE I R EPRESENTATIVE PREFILTER PATTERNS DERIVED FROM MITRE ATT&CK [15]. ID

Technique

Category

Pattern (simplified)

Sev.

RCE010 CRED010 EXFIL002 PERS002 OBFUS017

T1140 T1552.001 T1048 T1546.004 T1027

Def. Evasion Cred. Access Exfiltration Persistence Obfuscation

base64 -d | sh .aws/credentials curl --data @/<path> >> .bashrc zero-width Unicode cluster

CRIT CRIT HIGH HIGH HIGH

The RuleEngine returns all hit positions as tion command. The gate is activated by installing a PATH shim (pattern id, start, end) tuples. By default, files with (skillgate install-skills-shim), which prepends zero hits are immediately classified SAFE with no LLM ˜/.skillgate/bin/ to the shell PATH so that any call is made, this is configurable to allow LLM review of npx skills add <package> invocation is transparently all files if desired. Critically, the prefilter is a gate, not a routed through S KILL G ATE before the package is installed. classifier: a matched pattern hit does not produce a final label. Once triggered, the gate fetches the candidate skill package, All labelling decisions are deferred to the LLM judge in assembles a unified SKILL.md from all Markdown files in the Stage 3 , preventing the high false-positive rates characteristic package, and runs the classification pipeline on the assembled of pure-regex approaches. content. If the classifier returns QUARANTINE or BLOCK, the install is aborted and the agent never loads the package. C. Stage 3: Snippet Extraction & LLM Judge In Stage 3 , for files with at least one prefilter matched F. Datasets hit, the ContentExtractor assembles a compact snippet We evaluate on SkillsBench [23], an open-source benchmark representation for the LLM. Each matched hit generates a char- of skill files. SkillsBench comprises 1,500 real-world benign acter window [match.start−cw, match.end+cw] where default skills sourced from public repositories and 150 hand-crafted cw=500 characters. Windows within cw/4 characters of each malicious skills spanning 8 attack categories, giving a total other are de-duplicated to avoid repeating overlapping evidence. evaluation set of n=1,650 files with 9.1% malicious prevalence. The first default ms=20 unique windows are concatenated with --- separators, and a fixed preamble of approximately 700 G. Experiment Setup characters, including the skill package name, file path, and 1) Baselines: We compare S KILL G ATE against the current metadata, is prepended to provide context to the judge. The existing methods in skill file scanning. ClawVet [34]: a staticdefault configuration (cw=500, ms=20) caps snippet content analysis tool applying 54 hand-written patterns in six sequential at 8,000 characters (the payload builder’s hard limit), versus the passes, evaluated at the oracle-optimal threshold on SkillsBench. full file length; combined with skipping the 67% of files that SkillScanner [35]: the Cisco AI Defense multi-engine scanner raise no prefilter hit, this produces the token savings quantified (v2.0.11) combining static YARA/regex rules with a behavioral in RQ2. A smaller budget (ms=5) saves further tokens at a control-flow graph, evaluated at the oracle-optimal severity small accuracy cost (Section V). threshold across five output levels (SAFE / INFO / LOW / The assembled snippet is then submitted to the LLM backend MEDIUM / HIGH / CRITICAL). Oracle-optimal thresholds (default: gpt-5.4-mini) together with a structured system prompt. for ClawVet and SkillScanner favor those baselines; reported The prompt instructs the model to act as a security analyst comparisons are therefore conservative. and return a classification label SAFE, SUSPICIOUS, or 2) Metrics: We report Precision (P), Recall (R), F1, and MALICIOUS, along with a numeric confidence score in [0, 1], False Positive Rate (FPR). We additionally report the Matthews following the LLM-as-a-judge paradigm [36]. By sending the Correlation Coefficient (MCC), defined as flagged regions rather than the full file, we bound per-call token TP · TN − FP · FN cost while giving the judge the specific evidence that triggered MCC = p , (T P +F P )(T P +F N )(T N +F P )(T N +F N ) the prefilter; our ablation (Section V) shows that a moderate window budget (ms=20) supplies enough surrounding context which ranges over [−1, 1] (1 = perfect, 0 = no better than for accurate adjudication, outperforming a tighter budget. chance, < 0 = anti-correlated). Unlike F1, MCC incorporates D. Stage 4: Policy Engine and Audit Log In Stage 4 , the PolicyEngine maps the LLM verdict and confidence score to a configurable concrete action: by default MALICIOUS (confidence ≥ 0.7) → BLOCK; SUSPICIOUS (confidence ≥ 0.5) → QUARANTINE; SUSPICIOUS (confidence < 0.5) → WARN; SAFE → ALLOW. Blocked skill files are rejected with a structured error response; quarantined files are written to ˜/.skillgate/quarantine/ for human review. Every decision, including the classification label, confidence score, pattern hits, snippet, and elapsed time, is appended to the audit log (˜/.skillgate/logs/audit.jsonl), enabling engineers to audit the screening history and calibrate the confidence threshold to their organization’s risk tolerance. E. Skills CLI Gate In addition to runtime MCP interception, S KILL G ATE ships a guarded wrapper for the common npx skills add installa-

all four confusion-matrix cells so it is not inflated by a classifier that simply flags most inputs. This property matters under SkillsBench’s heavy 9.1% class imbalance, where the benign class dominates: a tool with a high FPR can still post a deceptively reasonable F1, whereas MCC penalizes the resulting false positives and gives a single imbalancerobust score for ranking methods at their operating points [37]. Token counts are computed using the tiktoken o200k_base tokenizer. Latency is measured as wall-clock time from file receipt at the daemon to final policy decision. AUPRC (Area Under the Precision-Recall Curve) serves as the primary threshold-independent metric, computed from a per-sample malicious-likelihood score. For S KILL G ATE, prefilter-safe files (no LLM call) receive score 0; for files sent to the judge, the score is the model’s confidence when it labels the file malicious and 1 − confidence otherwise, so that higher scores rank moresuspicious files first. Baseline scores use each tool’s native ranking signal (ClawVet’s numeric risk score; SkillScanner’s

1.0 FPR≤5% zone

Precision

0.8 0.6

prefilter ceiling

0.4 0.2 random (0.09)

0.0 0.0

0.2

0.4

0.6

0.8

1.0

Recall

SkillGate (ours) (AP=0.830) ClawVet (AP=0.144) SkillScanner (AP=0.161)

Fig. 3. Precision-Recall curves on SkillsBench. Filled circles mark current operating points.

severity rank). We compute AUPRC as the average precision over the resulting ranking. All results were computed over a three run average to account for LLM nondeterminism, with standard deviations reported. IV. R ESULTS RQ1: How accurately does S KILL G ATE detect malicious skill files? Approach: We evaluate detection on SkillsBench (n=1,650, 9.1% malicious) and compare S KILL G ATE against existing methods ClawVet and SkillScanner. SkillScanner is reported at all five static severity thresholds to expose its precision– recall tradeoff, and as SkillScanner+LLM, which escalates every file to the same LLM judge S KILL G ATE uses (gpt-5.4mini). We report Precision (P), Recall (R), F1, the Matthews Correlation Coefficient (MCC), and False Positive Rate (FPR), with AUPRC as the primary threshold-independent metric computed from per-sample confidence scores. ClawVet and SkillScanner+LLM are evaluated at their oracle-optimal thresholds on the test set. Results: Table II presents the full method comparison and Figure 3 the precision–recall curves. S KILL G ATE achieves the best F1 and AUPRC on SkillsBench, outperforming ClawVet (F1=0.258), all SkillScanner static thresholds (best F1=0.295), and SkillScanner+LLM (F1=0.287) under oracle-optimal settings. S KILL G ATE achieves F1=0.817, R=0.769, FPR=1.13%, and AUPRC=0.830 versus 0.144 for ClawVet and 0.162 for SkillScanner, resulting in a 5–6× gap that holds across all operating thresholds as shown in Figure 3. The same ordering holds under MCC, which accounts for all four confusionmatrix cells under the benchmark’s 9.1% class imbalance: S KILL G ATE scores MCC=0.803 against a best-baseline 0.331 (SkillScanner+LLM), 0.225 (ClawVet), and 0.215 (SkillScanner ≥MEDIUM), a 2.4× margin over the strongest competitor. SkillScanner ≥INFO attains F1=0.157 yet MCC= −0.037, i.e.

slightly worse than random, exposing a degenerate operating point that F1 alone masks. SkillScanner+LLM underperforms despite 3× more LLM calls. Using the same LLM backbone (gpt-5.4-mini), SkillScanner’s LLM mode applies the judge to every file (1,650 calls) yet achieves only F1=0.287, R=0.180, and AUPRC=0.246, roughly 0.59 lower in recall and 0.58 lower in AUPRC than S KILL G ATE, despite using 3× more LLM calls (1,650 vs. 540). The gap is not explained by full-file input per se: our own fullfile ablation (B2, Section V) reaches R=0.804 with the identical judge, so a purpose-built prompt over whole files can score well. Rather, the shortfall stems from how a general-purpose scanner integrates the LLM, coupling its verdict to a conservative static severity gate rather than deferring the decision to the judge. The practical takeaway is that S KILL G ATE’s targeted-snippet design matches or exceeds this full-escalation baseline while sending a fraction of the tokens (RQ2), i.e. it recovers the accuracy of an LLM pass without its cost. ClawVet and SkillScanner static are undeployable. At oracleoptimal thresholds, ClawVet reaches F1=0.258 at FPR=50.4% (756/1,500 false positives) and SkillScanner’s best static operating point (≥MEDIUM) reaches F1=0.295 at FPR=17.4% (261/1,500 false positives). The structural FPR floor arises from pattern-matching on real-world benign content that incidentally matches security-related vocabulary; neither tool can be deployed in any workflow where false positives disrupt developer productivity. We further analyze the false positives in RQ4 (Section IV). RQ2: How much does S KILL G ATE reduce LLM input tokens? Approach: We measure LLM input tokens with the tiktoken o200k_base tokenizer (gpt-5.4-mini/gpt-4o family) and compare S KILL G ATE’s hybrid pipeline against full-file screening, in which every file is sent to the judge with its full content. We decompose the saving into two sources: skip savings (files with zero prefilter hits bypass the LLM entirely) and snippet savings (flagged files send only their matched windows rather than the whole file). Results: Table III presents token counts and cost comparisons. S KILL G ATE reduces LLM input tokens by 76.9% compared to full file scan on SkillsBench. Token savings arise from two sources: (1) Skip savings: 67.2% of SB files have zero prefilter hits and bypass the LLM entirely. (2) Snippet savings: LLM-bound files send only their matched windows (up to ms=20 × 500-char snippets, capped at 8k chars) rather than up to 12k-char full-file truncation. The saving therefore scales with file size, for SB’s ∼7k-char average files the flagged windows are a fraction of the full payload, whereas corpora of very short skills benefit less as the fixed preamble dominates each request. Lowering the cap to ms=5 raises the saving to 80.0% at a small accuracy cost (Section V). RQ3: What is the runtime latency of S KILL G ATE? Approach: We measure wall-clock latency from file receipt at the daemon to the final policy decision, separating the

TABLE II D ETECTION RESULTS : S KILL G ATE VS . EXISTING TOOLS ON THE S KILLS B ENCH BENCHMARK . † C LAW V ET AND S KILL S CANNER +LLM THRESHOLDS ARE ORACLE - OPTIMAL ON EACH TEST SET. S KILL G ATE AND S KILL S CANNER +LLM ARE THE MEAN OF THREE RUNS (LLM NONDETERMINISM ); THE STATIC TOOLS (C LAW V ET, S KILL S CANNER ) ARE DETERMINISTIC . S KILL G ATE PER - RUN STD : F1 0.013, AUPRC 0.016, MCC 0.019, FPR 0.59 PP. Dataset

Method

AUPRC

S KILL G ATE (ours) ClawVet† SkillScanner ≥INFO SB (n=1,650) SkillScanner ≥LOW SkillScanner ≥MEDIUM SkillScanner ≥HIGH SkillScanner ≥CRITICAL SkillScanner+LLM†

Prec

Recall

F1

0.830 0.875 0.144 0.151 0.162 0.087 0.205 0.214 0.345 0.438 0.246 0.710

0.769 0.893 0.820 0.480 0.473 0.067 0.047 0.180

0.817 0.258 0.157 0.287 0.295 0.112 0.084 0.287

TABLE III S KILL G ATE ’ S HYBRID LLM INPUT TOKENS SENT TO THE JUDGE VS FULL FILE DETECTION . T OKEN COUNTS ARE MEASURED ON ONE RUN ; PER - RUN PAYLOAD SIZES VARY BY <0.1%. Dataset SkillsBench (n=1650)

Total Tokens

Hybrid tokens

Savings

Ratio

3,022,409

699,085

76.9%

4.3×

TABLE IV AVERAGE PER - FILE SCREENING LATENCY ON S KILLS B ENCH (n=1,650), AGAINST A SINGLE HOSTED G P T -5.4- M I N I ENDPOINT. S KILL S CANNER +LLM LATENCY IS THE MEAN OVER SUCCESSFUL CALLS ; INCLUDING ERRORED CALLS IT IS ∼6,124 MS (7.5×). Method

Avg latency

LLM calls

Fraction

S KILL G ATE

∼139ms ∼2,208ms

0 1

67.2% (prefilter-safe) 32.8% (LLM path)

S KILL G ATE (weighted)

∼818ms

540

—

SkillScanner+LLM

∼6,281ms

1,650

100%

prefilter-safe path (regex only) from the LLM path, and report the weighted average over the SkillsBench operating mix (67.2% prefilter-safe, 32.8% LLM-bound). As a comparison point we measure SkillScanner+LLM, which calls the LLM on every file. Results: On average (weighted), S KILL G ATE adds only ∼818ms overhead per skill scanned: ∼139ms when the LLM is skipped and ∼2,208ms when an LLM call is required—7.7× faster than SkillScanner+LLM’s ∼6,281ms per file. This ∼87% reduction in overhead comes with substantially better detection (F1=0.817 vs. 0.287, Table II). A local LLM backend (Ollama) would eliminate the network round-trip on the LLM path; we leave local backend evaluation to future work. RQ4: What categories of benign content trigger false positives? Approach: We manually categorize the union of the 26 benign SkillsBench files that S KILL G ATE flags across the three runs by the benign content that triggered them, and contrast both the volume and the severity of false alarms against the baselines at their representative operating points (S KILL G ATE: default ms=20; ClawVet: oracle-optimal threshold; SkillScanner: best static severity, ≥MEDIUM; SkillScanner+LLM: oracle-optimal

MCC

FPR

LLM calls

0.803 1.13% 0.225 50.40% −0.037 86.47% 0.206 18.67% 0.215 17.40% 0.118 1.27% 0.119 0.60% 0.331 0.73%

540 0 0 0 0 0 0 1650

17 False Positives on SkillsBench (3-run mean) Other 18% (3) 12% (2)

.env reference

71% (12)

Base64 in docs

Fig. 4. False positive categorization of S KILL G ATE on SkillsBench

threshold). We additionally report the block-grade false-positive rate—the share of benign files each tool flags at HIGH or CRITICAL severity, which a severity-gated policy would hardblock. Results: S KILL G ATE produces on average 17 false positives on SkillsBench (FPR=1.13%, mean of three runs). These are dominated by MEDIUM-severity flags (∼13 of 17): base64-encoded content in code examples and API documentation that triggers the encoded-payload prefilter rule, which the judge confirms as suspicious without a clearly benign context. A stable core of about seven files is flagged on every run, while the remaining ∼10 are borderline MEDIUM cases that flip run-to-run under judge nondeterminism. Only about 3 of the 17 reach block-grade severity, so a tiered policy (quarantine ≥MEDIUM, block ≥HIGH only) would surface roughly three user-visible blocks. Baseline methods flag legitimate skills at scale. Table V contrasts the false positives of S KILL G ATE with those of the two static baselines on the same 1,500 benign SkillsBench files. At their representative settings ClawVet reaches FPR=50.4% (756 false alarms) and SkillScanner FPR=17.4% (261), 44× and 15× the false-positive rate of S KILL G ATE’s 1.13% (17). Both sit far away from practical deployable scenarios; in practice, a developer adopting ClawVet would see roughly one in two benign skills blocked. The gap widens at the severities that matter operationally: under a policy that hard-blocks HIGH and CRITICAL findings, ClawVet would block 296 legitimate

deployable at acceptable FPR, and trading 389 fewer false positives for 20 additional missed malicious files is strongly favorable under the benchmark’s heavy class imbalance. Full-file vs. snippet (B2 vs. S KILL G ATE). B2 attains 3.5pp higher recall than the default (R=0.804 vs. 0.769) but at higher Method FPR FP Crit High Med Low Block-grade FPR (1.76% vs. 1.13%), lower precision (0.822 vs. 0.875), and S KILL G ATE (ours) 1.13% 17 3 0 13 1 3 by screening every file in full (1,650 LLM calls vs. 540). The † ClawVet 50.40% 756 39 257 307 153 296 extra recall is consistent with the long-file effect as SB files SkillScanner ≥MED 17.40% 261 9 10 242 0 19 SkillScanner+LLM† 0.73% 11 11 0 0 0 11 average ∼7k characters, so bounded snippet windows can miss malicious content outside the captured regions, but B2 pays for it with more false positives and full-file cost. The default skills (block-grade FPR=19.7%) and SkillScanner 19 (1.3%), snippet configuration lands at the better F1/FPR operating point: whereas S KILL G ATE contributes only about 3 (0.20%). Because concentrating the flagged evidence in a short context sharpens both baselines pattern-match security-related vocabulary in the judge’s precision. benign documentation without a disambiguating judge, their Snippet budget (ms=20 default vs. ms=5). The snippet cap false positives are not only far more numerous but frequently sets how many matched windows each escalated file sends; it graded severe enough to interrupt the developer, confirming does not change which files are escalated, so both configurations that neither is deployable under realistic constraints. issue the same 540 LLM calls. Lowering the cap from the A low false-positive rate alone is not sufficient. SkillScan- default 20 to 5 reduces accuracy, specifically F1 0.806 vs. ner+LLM is the one baseline whose raw false-positive rate 0.817, FPR 1.31% vs. 1.13%, recall 0.762 vs. 0.769, because is comparable to ours: at its oracle-optimal threshold it the judge sees less of the flagged evidence and makes slightly produces 11 false positives (FPR=0.73%), slightly fewer than worse calls. The gain from the wider budget is modest but S KILL G ATE’s 17 (1.13%). Two facts make this raw comparison consistent, and it costs only marginally more tokens: ms=20 misleading in SkillScanner+LLM’s favour. First, it reaches this reduces LLM input tokens by 76.9% versus full-file screening, low FPR only by flagging almost nothing: the same conservative against 80.0% for ms=5 (RQ2). We therefore adopt ms=20 threshold (CRITICAL-only) that suppresses false alarms also as the default, since it maximizes F1 and minimizes FPR at drops its recall to 0.180 (Table II), missing more than four in no additional call count, and expose ms=5 as a lower-cost five malicious files. Its few false positives are thus a by-product operating point (a further 3pp token saving) for cost-sensitive of a detector that rarely fires, not evidence of broad coverage. It deployments. flags 38 files in total, of which 27 are true positives (precision Threshold sensitivity. Figure 5 plots F1, Recall, and Precision as 0.710) and 11 are benign. Second, every one of those 11 false the confidence threshold θ varies from 0 to 1. The pre-specified positives is graded CRITICAL, so a severity-gated policy would default θ=0.70 yields F1=0.817, R=0.769, FPR=1.13%. The hard-block all 11 legitimate skills, roughly 3.7× S KILL G ATE’s F1-optimal threshold (θ≈0.14) achieves F1=0.842, R=0.824, block-grade count of 3, even though S KILL G ATE also catches FPR=1.33%, i.e. an operator prioritizing recall can lift deover four times as many malicious files (R=0.769 vs. 0.180). tection by 2.5pp while keeping FPR below 1.5%. A realisitc What matters for deployment is the block-grade false-positive deployabile FPR (¡5%) holds across essentially the whole rate paired with recall, and on that axis S KILL G ATE dominates: range θ ∈ [0.06, 0.99], so the operating point does not require it is the only method that keeps hard-blocks of legitimate skills per-deployment tuning. low while retaining deployable recall. TABLE V FALSE - POSITIVE BREAKDOWN ON S KILLS B ENCH (1,500 BENIGN FILES ). B LOCK - GRADE COUNTS BENIGN FILES FLAGGED AT HIGH OR CRITICAL . S KILL G ATE COUNTS ARE THE MEAN OF THREE RUNS ; STATIC TOOLS ARE DETERMINISTIC . † O RACLE - OPTIMAL THRESHOLDS ON THE TEST SET.

V. A BLATION S TUDY

VI. T HREATS TO VALIDITY

We ablate the HybridClassifier design against the deployed Threats to construct validity relate to our evaluation metrics and default (ms=20 snippet windows). B1 (pure regex, no LLM) measurement pipeline. We report Precision, Recall, F1, FPR, and B2 (full-file LLM, no prefilter) are the architectural the Matthews Correlation Coefficient (MCC), and AUPRC, and extremes; the ms=5 variant tests a lower snippet budget than treat recall as the primary operational metric because a missed the default. Table VI presents results on SkillsBench. malicious skill file (false negative) carries a higher security cost Prefilter contribution. Comparing B1 (prefilter only) to S KILL - than a blocked benign file, while FPR captures the deployability G ATE (default) isolates the value of the LLM judge on top constraint; F1 is reported only for conventional comparison of the regex prefilter. The prefilter alone flags any file with to prior work and is not the optimization target. Because a pattern hit, catching 135/150 malicious files (R=0.900) but SkillsBench is heavily imbalanced (9.1% malicious), replicating also 406 benign ones (FPR=27.1%), which is far above any a real world scenario where the vast majority of skill files are deployable constraints. Adding the LLM judge cuts FPR from benign, we report MCC and AUPRC as our imbalance-robust 27.1% to 1.13% (a 26.0pp reduction) by reclassifying the vast summary metrics, MCC accounts for all four confusion-matrix majority of those false positives, at the cost of some recall cells and AUPRC is threshold-independent, so that the headline (0.900→0.769) where the judge overrides a genuine prefilter comparison does not hinge on F1, which can be inflated under hit. The judge is thus irreplaceable: the prefilter alone is not class imbalance. Token savings are measured with tiktoken

TABLE VI A BLATION OF CLASSIFIER DESIGN PARAMETERS ON S KILLS B ENCH (n=1,650). LLM- BASED ROWS (B2, ms=5, ms=20) ARE REPORTED AS MEAN ± STD OVER THREE RUNS (FPR STD IN PP ); B1 ( REGEX ONLY ) IS DETERMINISTIC . Config B1 (pure regex, no LLM) B2 (full-file LLM, no prefilter) S KILL G ATE ms=5 snippets S KILL G ATE (default, ms=20)

Prec

Recall

F1

MCC

FPR

LLM calls

0.250 0.822 ±.031 0.856 ±.054 0.875 ±.058

0.900 0.804 ±.010 0.762 ±.020 0.769 ±.020

0.391 0.813 ±.011 0.806 ±.015 0.817 ±.013

0.385 0.794 ±.013 0.789 ±.019 0.803 ±.019

27.1% 1.76% ±.38 1.31% ±.57 1.13% ±.59

0 1650 540 540

1.0 prefilter recall ceiling (0.90)

F1-optimal default θ=0.70 θ=0.14, F1=0.842 F1=0.817, R=0.769

Metric value

0.8

0.6

0.4

0.2

F1 Recall Precision

0.0 0.0

0.2

0.4

0.6

0.8

1.0

Confidence threshold θ

Fig. 5. F1, Recall, and Precision of S KILL G ATE on SkillsBench as the confidence threshold θ varies.

o200k_base over the exact payloads the deployed system sends (preamble, skill header, and snippet windows) rather than simplified estimates, so the reported reductions match production behavior; nonetheless, our reliance on an LLM-asa-judge for the final label could introduce measurement error relative to a purely manual review.

Threats to external validity concern the generalizability of our findings. The prefilter patterns are derived from the MITRE ATT&CK taxonomy which is the most comprehensive pattern collection we could find, but cannot guarantee to cover all known security patterns. The benchmark’s attack vocabulary may also not represent all real-world payloads or advanced obfuscation tactics, as evidenced by the residual encoding-based false negatives (RSA modular arithmetic, zero-width characters) that expose a concrete coverage gap. A further limitation is that SkillsBench’s 150 malicious files are hand-crafted attack exemplars rather than captured in-the-wild samples, so measured recall may not transfer to organically evolving campaigns. All results use gpt-5.4-mini (OpenAI); recall on the LLM path may differ for other backends (e.g., claude-haiku-4-5 or local Ollama models), which we do not evaluate. Finally, the ClawHavoc campaign [1] and other registry-scale incidents were not available as benchmark datasets, so performance on live adversarial skill packages, particularly novel techniques crafted to evade the current ruleset, may differ from our benchmark results. VII. C ONCLUSION

In this paper, we present S KILL G ATE, a low-cost, deployable security proxy that screens MCP skill files before they reach an AI coding agent, closing the gap of the agent skills security Threats to internal validity concern factors within the study challenge. S KILL G ATE pairs a 530-pattern regex prefilter with that could influence the reported outcomes. The gpt-5.4-mini an LLM judge that sees only the flagged snippet windows: judge is nondeterministic, and results may shift across API benign files bypass the model entirely, and flagged files are versions or sampling settings. To account for this, every classified from their risky regions rather than their full content. SkillsBench detection number we report for S KILL G ATE and On the SkillsBench benchmark (1,650 skills, 9.1% malicious; SkillScanner+LLM is the mean of three independent runs (per- 150 hand-crafted attack exemplars), S KILL G ATE achieves run std for the ms=20 default: F1 0.013, AUPRC 0.016, MCC F1=0.817, recall=0.769, and FPR=1.13% (MCC=0.803, 0.019, FPR 0.59pp); because the regex prefilter is deterministic AUPRC=0.830; mean of three runs), outperforming ClawVet (540 escalations per run), this variance is confined to the and SkillScanner by 5–6× on threshold-independent AUPRC LLM judge, and the static baselines (ClawVet, SkillScanner) while reducing LLM input tokens by 77% and screening each are invariant across runs. The three runs span a three-week skill in ∼818ms, 7.7× faster than escalating every file to window and may straddle a provider-side model update (per-run the same judge. These results show that targeted, snippetfalse positives 7/24/20); we report their mean, not any single level screening makes inline malicious-skill detection both run. The context window (cw=500) and confidence threshold accurate and inexpensive enough to run at install time. We (θ=0.70) were pre-specified on a held-out development set. hope S KILL G ATE serves as a practical first line of defense as The snippet budget was set to ms=20 because it dominates the agent skill ecosystem, and its attack surface, continues to the smaller ms=5 budget on that same development set (recall grow. 0.972 vs. 0.928 at precision 1.0), so the choice is justified R EFERENCES without consulting benchmark labels; nonetheless, we cannot [1] C. Press, “Clawhavoc poisons openclaw’s clawhub with 1,184 malicious rule out that a still different configuration would shift the skills,” https://www.linkedin.com/pulse/clawhavoc-poisons-openclawsaccuracy–cost trade-off. clawhub-1184-malicious-skills-f5xqc, Feb. 2026.

[2] R. Sapkota, K. I. Roumeliotis, and M. Karkee, “Vibe coding vs. agentic [20] H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri, “Asleep coding: Fundamentals and practical implications of agentic ai,” arXiv at the keyboard? assessing the security of github copilot’s code contributions,” Communications of the ACM, vol. 68, no. 2, pp. 96–105, preprint arXiv:2505.19443, 2025. 2025. [3] M. Vukovic, R. Pan, T. K. Ho, R. Krishna, R. Pavuluri, and M. Merler, “Usage, effects and requirements for ai coding assistants in the enterprise: [21] N. Perry, M. Srivastava, D. Kumar, and D. Boneh, “Do users write more insecure code with ai assistants?” in Proceedings of the 2023 ACM An empirical study,” arXiv preprint arXiv:2601.20112, 2026. SIGSAC conference on computer and communications security, 2023, [4] Stack Overflow, “Stack overflow developer survey 2024,” pp. 2785–2799. https://survey.stackoverflow.co/2024/, 2024, 76% of respondents use or plan to use AI coding tools in their workflow. Accessed 2026-06. [22] A. Sejfia and M. Schäfer, “Practical automated detection of malicious npm packages,” in 44th IEEE/ACM International Conference on Software [5] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25–27, 2022. ACM, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, 2022, pp. 1681–1692. G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, [23] X. Chen, “SkillsBench-1650: A benchmark for AI agent skill safety detection,” https://huggingface.co/datasets/zenith6888/SkillsBench-1650, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, 2026, hugging Face dataset, CC-BY-4.0. E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, [24] K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, “Not what you’ve signed up for: Compromising real-world llm-integrated A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, applications with indirect prompt injection,” in Proceedings of the 16th M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, ACM workshop on artificial intelligence and security, 2023, pp. 79–90. D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba, “Evaluating large language models trained on code,” CoRR, vol. abs/2107.03374, [25] Open Source Security Foundation (OpenSSF), “Open Source Security Foundation: Dependency review and SCA tooling,” 2021. [Online]. Available: https://arxiv.org/abs/2107.03374 https://owasp.org/www-project-dependency-check/, 2022. [6] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, [26] F. Perez and I. Ribeiro, “Ignore previous prompt: Attack techniques for and K. R. Narasimhan, “Swe-bench: Can language models language models,” arXiv preprint arXiv:2211.09527, 2022. resolve real-world github issues?” in The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, [27] Y. Liu, Y. Jia, R. Geng, J. Jia, and N. Z. Gong, “Formalizing and benchmarking prompt injection attacks and defenses,” in May 7-11, 2024. OpenReview.net, 2024. [Online]. Available: 33rd USENIX Security Symposium, USENIX Security 2024, https://openreview.net/forum?id=VTF8yNQM66 Philadelphia, PA, USA, August 14-16, 2024, D. Balzarotti and [7] Anthropic, “Model Context Protocol specification,” W. Xu, Eds. USENIX Association, 2024. [Online]. Available: https://modelcontextprotocol.io/, 2024, accessed 2026-06. https://www.usenix.org/conference/usenixsecurity24/presentation/liu[8] M. Ohm, H. Plate, A. Sykosch, and M. Meier, “Backstabber’s knife yupei collection: A review of open source software supply chain attacks,” [28] J. Yi, Y. Xie, B. Zhu, E. Kiciman, G. Sun, X. Xie, and F. Wu, in Proceedings of the 17th International Conference on Detection of “Benchmarking and defending against indirect prompt injection attacks Intrusions and Malware, and Vulnerability Assessment (DIMVA), 2020. on large language models,” in Proceedings of the 31st ACM SIGKDD [9] M. Zimmermann, C. Staicu, C. Tenny, and M. Pradel, “Small world Conference on Knowledge Discovery and Data Mining V. 1, 2025, pp. with high risks: A study of security threats in the npm ecosystem,” 1809–1820. in 28th USENIX Security Symposium, USENIX Security 2019, Santa Clara, CA, USA, August 14-16, 2019, N. Heninger and P. Traynor, [29] K. Hines, G. Lopez, M. Hall, F. Zarfati, Y. Zunger, and E. Kiciman, “Defending against indirect prompt injection attacks with spotlighting,” Eds. USENIX Association, 2019, pp. 995–1010. [Online]. Available: in Proceedings of the Conference on Applied Machine Learning in https://www.usenix.org/conference/usenixsecurity19/presentation/zimmerman Information Security (CAMLIS 2024), Arlington, Virginia, USA, October [10] P. Ladisa, H. Plate, M. Martinez, and O. Barais, “Sok: Taxonomy of 24-25, 2024, ser. CEUR Workshop Proceedings, R. Allen, S. Samtani, attacks on open-source software supply chains,” in 2023 IEEE Symposium E. Raff, and E. M. Rudd, Eds., vol. 3920. CEUR-WS.org, 2024, pp. on Security and Privacy (SP). IEEE, May 2023, p. 1509–1526. 48–62. [Online]. Available: https://ceur-ws.org/Vol-3920/paper03.pdf [Online]. Available: http://dx.doi.org/10.1109/sp46215.2023.10179304 [11] Semgrep, Inc., “Semgrep: Static analysis at ludicrous speed,” [30] T. Rebedea, R. Dinu, M. N. Sreedhar, C. Parisien, and J. Cohen, “Nemo guardrails: A toolkit for controllable and safe llm applications with https://semgrep.dev/, 2024, accessed 2026-06. programmable rails,” in Proceedings of the 2023 Conference on Empirical [12] Snyk Ltd., “Snyk: Security platform for developers,” https://snyk.io/, Methods in Natural Language Processing: System Demonstrations. 2024, accessed 2026-06. Association for Computational Linguistics, 2023, p. 431–445. [Online]. [13] Y. Liu, Z. Chen, Y. Zhang, G. Deng, Y. Li, J. Ning, Y. Zhang, and Available: http://dx.doi.org/10.18653/v1/2023.emnlp-demo.40 L. Y. Zhang, “Malicious agent skills in the wild: A large-scale security [31] S. Willison, “Prompt injection and privilege separation in LLM applicaempirical study,” arXiv preprint arXiv:2602.06547, 2026. tions,” https://simonwillison.net/2023/Apr/25/dual-llm-pattern/, 2023. [14] D. Schmotz, L. Beurer-Kellner, S. Abdelnabi, and M. Andriushchenko, [32] H. Inan, K. Upasani, J. Chi, R. Rungta, K. Iyer, Y. Mao, M. Tontchev, “Skill-inject: Measuring agent vulnerability to skill file attacks,” arXiv Q. Hu, B. Fuller, D. Testuggine, and M. Khabsa, “Llama guard: LLMpreprint arXiv:2602.20156, 2026. based input-output safeguard for human-AI conversations,” arXiv preprint [15] MITRE Corporation, “MITRE ATT&CK: Adversarial tactics, techniques, arXiv:2312.06674, 2023. and common knowledge,” https://attack.mitre.org, 2024, accessed: 2024. “OpenAI moderation API,” [16] SigmaHQ, “Sigma: Generic signature format for SIEM systems,” [33] OpenAI, https://platform.openai.com/docs/guides/moderation, 2022, accessed https://github.com/SigmaHQ/sigma, 2024, accessed 2026-06. 2026-06. [17] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao, “React: Synergizing reasoning and acting in language models,” [34] M. Shaikh et al., “ClawVet: Static analysis for MCP skill security,” https://github.com/MohibShaikh/clawvet, 2024. in The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. [35] Cisco AI Defense, “skill-scanner: MCP skill security scanner,” https://github.com/cisco-ai-defense/skill-scanner, 2024, v2.0.11. Accessed [Online]. Available: https://openreview.net/forum?id=WE vluYUL-X 2026-06. [18] T. Schick, J. Dwivedi-Yu, R. Dessı̀, R. Raileanu, M. Lomeli, [36] L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: Language Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, models can teach themselves to use tools,” CoRR, vol. abs/2302.04761, and I. Stoica, “Judging llm-as-a-judge with mt-bench and chatbot 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2302.04761 arena,” CoRR, vol. abs/2306.05685, 2023. [Online]. Available: [19] Y. Qin, S. Liang, Y. Ye, K. Zhu, L. Yan, Y. Lu, Y. Lin, X. Cong, https://doi.org/10.48550/arXiv.2306.05685 X. Tang, B. Qian, S. Zhao, R. Tian, R. Xie, J. Zhou, M. Gerstein, [37] D. Chicco and G. Jurman, “The advantages of the matthews correlation D. Li, Z. Liu, and M. Sun, “Toolllm: Facilitating large language models coefficient (mcc) over f1 score and accuracy in binary classification to master 16000+ real-world apis,” CoRR, vol. abs/2307.16789, 2023. evaluation,” BMC Genomics, vol. 21, no. 1, p. 6, 2020. [Online]. Available: https://doi.org/10.48550/arXiv.2307.16789