arXiv:2606.15549v1 [cs.CR] 14 Jun 2026
C MD N EEDLE: Measuring the Incompleteness of Command Denylists for AI Agents Chuyang Chen
Zhiqiang Lin
The Ohio State University [email protected]
The Ohio State University [email protected]
Abstract—The adoption of AI agents is increasing rapidly. Terminal AI agents, i.e., AI agents that run in terminal environments, are a widely used type of AI agents. Terminal AI agents rely heavily on shell command execution to interact with the host systems. They adopt a three-list commandgating mechanism to mitigate security risks introduced by command execution, with denylists serving as the load-bearing component. However, modern operating systems often ship a large, ever-expanding set of shell commands with complex functionalities. Our observation is that even a built-in denylist of Claude Code, well-maintained by its developers, can overlook bypass commands that invalidate its effectiveness. Such negligence leads to fragile command denylists that cannot even block operations that practitioners expect them to block. This paper presents the first systematic characterization of command denylist fragility in terminal AI agents. The paper formalizes the command denylist fragility problem and proposes an LLM-driven pipeline, C MD N EEDLE, to detect such fragility. It prompts the LLM to propose possible bypasses and iteratively repairs them using feedback from a validator that executes them in a sandbox. In the evaluation, we applied C MD N EEDLE to 1,709 real-world command denylists (containing 13,332 denylist rules) collected from GitHub. The evaluation shows several key findings, including that 69.0–98.6% of the denylists are fragile, that this fragility occurs consistently across projects and agents, and the validity of several possible root causes for this fragility. Our pipeline and findings will hopefully facilitate future research and practice regarding the command denylists used by AI agents.
1. Introduction AI agents running in terminal environments, such as Claude Code and Codex, have recently emerged as versatile personal assistants, handling tasks ranging from software development to tax filing on behalf of human users [1]. One of the most important and yet most dangerous capabilities of these terminal AI agents is their ability to dynamically synthesize and execute shell commands to fulfill contextdependent needs of the tasks [2]. Such an ability creates a dilemma between flexibility and security: the agents are useful because they can choose and use various commands on the host system with broad autonomy and minimal human intervention, while security may be compromised by the extensive permissions required by such “freedom.”
Currently, the common practice to mitigate the dilemma is to gate commands using three lists: a denylist to filter out obviously dangerous operations, an allowlist to permit obviously benign operations, and an implicitly defined (though sometimes customizable) ask-list that contains everything else and sending them to an evaluator (users themselves for manual audit, or an LLM-based judge in some designs) to get a final decision. Two facts make the denylist the load-bearing component in this three-list command-gating mechanism: First, commands in the denylist are inherently dangerous and should not be executed under any circumstances (according to the semantics of “denylists”). The aftermath of these commands should be deterministic and severe. Second, approval fatigue causes the users to approve 93% of command execution requests [3]. Thus, the ask-list largely regresses to an allowlist, significantly compromising its effectiveness. The problem. However, denylists can be fragile, as they are often hurriedly compiled and may overlook commands that bypass them. This situation worsens further, given that modern operating systems are essentially “open worlds” that often ship a large, ever-expanding set of shell commands to meet diverse development needs. The functionality of these commands is often beyond the knowledge of ordinary practitioners. Our observations (detailed in §3) on the builtin denylist in Claude Code’s auto-mode and an arbitrarily sampled command denylist from GitHub confirm this intuition: even the denylist shipped with a widely used agent by default and well-maintained by its developers can overlook bypass commands that compromise its protection, let alone the denylists compiled by regular users of these agents. Therefore, a critical problem requiring urgent attention is to evaluate and characterize this fragility to facilitate future research and practice. Challenges and methodology. Two obstacles stand in the way of the characterization. First, it is hard to propose candidate operations that a denylist aims to block, and to identify bypass commands that can perform them. Second, even with candidate operations and bypass commands, it is hard to validate them, since their execution is ephemeral. To address the challenges, we propose C MD N EEDLE, an LLM-driven pipeline with two targeted designs for characterization. First, C MD N EEDLE leverages an LLM to propose candidate operations and bypass commands. LLMs’
expert-level capability to synthesize shell commands [4] enables them to produce thousands of diverse, high-quality candidates that no traditional technique can propose. Second, C MD N EEDLE defines and validates candidate operations and bypass commands based on the side effects they leave in a sandbox environment after execution. Thus, it avoids ambiguity and false positives in the characterization. Findings. We evaluate 1,731 real-world AI agent denylists (containing 13,332 denylist rules) collected from GitHub with C MD N EEDLE and reveal several key findings: • First, the denylist fragility problem is severe. 69.0–98.6% denylists cannot fully block operations they aim to block. • Second, the severity of the fragile denylist problem is consistently high across projects and agents. The proportion of denylists that overlook bypass commands is consistently above 44.4% across projects that receive varying levels of attention. Similarly, the proportion is always higher than 46.2% for projects that use different agents. • Third, we statistically confirmed two root causes of the denylist fragility problem. One is that the authors of denylists ignore less-known commands, creating bypasses. Another is that multi-purpose commands are deliberately unlisted from the denylists for their benign use. Still, they may become bypasses when used in ways that perform the operations the denylists aim to block. • Fourth, fixing fragile denylists will require adding numerous commands (averaging 217 if they aim to block specific operations) to complete them, which is a huge burden for anyone who wants to resolve the problem. Contributions Overall, we make three contributions: • We identify and formalize the command denylist fragility problem for terminal AI agents. We argue that despite their load-bearing role in the command gating mechanism of AI agents, command denylists can be fragile and overlook bypasses that compromise their effectiveness. • We design and implement C MD N EEDLE, an LLM-driven pipeline, to automatically discover bypass commands that can perform operations a denylist aims to block. It enables large-scale characterization of the problem across corpora comprising thousands of denylists. • We collect and construct a dataset of 1,709 denylists (containing 13,332 denylist rules) and evaluate them using C MD N EEDLE. The evaluation reveals several key findings that may bring insights and directions for future research and practice. The code and data of this paper will be released later.
Figure 1: A three-list command-gating model. Implementations may vary in some details. 1 2 3 4 5 6 7 8
"deny": [ "cat *.env*", "head *.env*", "tail *.env*", "less *.env*", "more *.env*", "grep *.env*" ]
Listing 1: Running example, a Claude Code command denylist where “*” is a wildcard that matches any string, including the empty string. based general-purpose assistant from Anthropic [3], Codex, a coding agent from OpenAI [6], and OpenCode, an opensource, community-driven agent compatible with different model providers [7]. All these agents share the same architecture: an LLM driver, a harness layer that orchestrates requests to and responses from the LLM, and various tools, including the shell command execution tool mentioned above. However, the ability to execute shell commands introduces a dilemma between flexibility and security. The agents are useful because they can run whatever commands a task requires. Yet this same freedom allows an adversary who hijacks it to perform arbitrary, harmful operations on the host. Three-list command gating. The common practice to mitigate the dilemma is to gate the shell commands before execution using three lists, as illustrated in Figure 1: • Denylist. Denylists block obviously dangerous operations that the agent must never run. Commands in denylists typically conduct destructive operations (such as file deletion) or pose an obvious threat (such as secret leaks). Listing 1 gives an example denylist in a GitHub project that uses Claude Code. The example will be revisited in §3.2. • Allowlist. Allowlists permit benign operations that the agent may run without any intervention, such as reading a non-confidential file. • Ask-list. Everything not matched by the allowlist or the denylist goes into the ask-list and is delegated to an authoritative evaluator, typically the user, for a final decision [8], [9], [10], [11]. Recently, more agents are exploring using an LLM-based judge as an evaluator [3].
2. Background Terminal AI agents. AI agents are software driven by AI models that interact with host systems to perform various tasks on behalf of human users. Terminal AI agents are a type of AI agent that run in terminal environments and use shell commands to interact with the system and have been deployed and used in the real world [1], [5]. Some wellknown terminal AI agents include Claude Code, a terminal-
2
This basic three-list command-gating architecture is observed in various terminal AI agents. Several notes should accompany the model. First, the denylist takes the highest precedence (thus it is at the top in Figure 1), meaning that any commands it contains will be rejected directly, neither matched by the allowlist nor checked by the evaluator. Second, the denylist and allowlist are typically customizable by users to meet their specific needs and can sometimes be updated while the agent is running to take commands from the ask-list, avoiding tedious, repeated approval requests. Third, the architecture described in this section is an abstract model, and implementations across different agents may vary in some details, e.g., Claude Code includes an extra built-in denylist in its auto-mode. Approval fatigue. The ask-list is advertised as a finegrained safety net, neither too permissive nor too restrictive, whereas the promise is largely compromised by approval fatigue. Users should, by expectation, carefully audit commands proposed to run by the ask-list. However, real-world data shows that the expected approval process breaks: users accept 93% of approval requests [3]. The overwhelming majority of decisions means users tend to approve anything [3], [12] unthinkingly. The ask-list largely regresses into an allowlist, and the boundary it should enforce thus loosens. An LLM judge, as the evaluator, will not suffer from approval fatigue. However, it may encounter LLM-specific issues such as hallucinations [13] and adversarial suffices [14], [15] though. Therefore, denylists become the load-bearing component in the three-list command-gating mechanism.
Category
Commands
Interpreters
python, python3, python2, node, deno, tsx, ruby, perl, php, lua npx, bunx, npm run, yarn run, pnpm run, bun run bash, sh, ssh
Package managers Local and remote shells
Table 1: The code execution commands in the built-in denylist of Claude Code’s auto-mode. use of AWK is to process texts, the domain-specific language (DSL) used by AWK is Turing-complete, and more importantly, AWK can invoke Bash and pass arbitrary Bash scripts in the invocation. The fact that AWK is not in the denylist essentially bypasses the rule that covers Bash. Experiments confirmed that AWK can bypass the denylist in Claude Code’s auto-mode, and a disclosure has been sent to Anthropic, the company that develops Claude Code. The observation. This bypass confirms the observation that denylists can be fragile, even when well-maintained by the agent developers, as they may overlook bypass commands.
3.2. Bypasses of a Customized Denylist The above denylist is a single special example. It indicates how fragile command denylists can be, though. Now, we demonstrate a real denylist for an open-source project on GitHub, compiled by the project’s developers, as an example of fragility in user-customized command denylists deployed more widely across projects that use these agents. Purpose of the denylist. Listing 1 shows the denylist. It is written in Claude Code’s denylist format, where the wildcard * matches any string, including the empty string. Clearly, all commands blocked the denylist read files that define environment variables and print their values to stdout, which is exactly the operation that the denylist targets. The bypasses. Many commands can perform the same operation, and those blocked by this denylist are only a small proportion among them. Examples include cp .env /dev/stdout and diff .env /dev/null. The denylist considers only the most common commands for this purpose, thereby overlooking others. The observation. These bypasses confirm the observation that agent users adopt fragile denylists in the wild.
3. Observations This section presents our observations on two denylists to demonstrate the prevalence of fragile command denylists. In §3.1, we run through the built-in denylist in Claude Code’s auto-mode (mentioned in §2) to show that even a denylist shipped with a widely used agent by default and well maintained by its developers can overlook bypasses. In §3.2, we examine the denylist of an open-source project on GitHub to show that the same issue occurs in denylists compiled by regular users.
3.1. Bypass of the Claude Code Built-in Denylist Purpose of the denylist. The built-in denylist in Claude Code’s auto-mode enumerates code-execution commands, including language interpreters, package managers’ script running subcommands, local and remote shells, and forces them through the LLM judge, forbidding the customizable allowlist from silently approving them, as these commands can conduct arbitrary operations on the host system, and thus shouldn’t be mindlessly waved through [3]. Table 1 lists the commands in this denylist, confirmed by experiments. The bypass. However, the enumeration is still incomplete. It overlooks commands that can also execute arbitrary code and are shipped in all commonly used Linux distributions (which are host systems that Claude Code often lives in). One of them is AWK, a POSIX-mandated utility present on every Unix and Unix-like host. Though the primary
4. Threat Model Attack scenarios. We consider an adversary whose objective is to cause the agent to execute a command that produces an attacker-chosen, security-relevant effect on the host, such as exfiltrating credentials or irreversibly destroying data. The adversary, however, does not hold a shell on the host directly, as the gating mechanism would be moot in that case. We further assume the adversary knows the gating policy in full, including the denylist. It is a realistic Kerckhoffs-style assumption, given that these lists are often managed in a per-project setting together with the source code in a Git repository [8], [9], [10]. The adversary should be able to dictate the agent’s proposed
3
commands. This premise reflects the fact that AI agents routinely ingest untrusted content that may carry adversarial payloads to conduct indirect prompt injection as part of normal operation [16], [17], [18]. Our analysis does not depend on any single injection technique. It requires only the now-standard observation that the commands reaching the gate cannot be assumed benign [2]. Trust boundary. The trust boundary in our model lies precisely at the command-gating layer. Everything upstream, including the LLM driver, the harness, and any content they have consumed, is untrusted, since a compromised alignment places the proposed command entirely under adversarial control. Everything downstream, including the shell, the binaries it can reach, and the host on which they run, is trusted only in the operational sense that it faithfully executes whatever the gate admits, performing no further check of its own. The gate with denylists like those shown in §2 and 3 is thus the sole point that can stop the adversarial operations. Out-of-scope problems. Some adjacent threats fall outside our scope, including how an adversary compromises the model’s alignment in the first place, which is taken as given, and implementation-level defects in the gating layer itself, as our argument is strictly stronger without them: we show that the denylist fails even when it is implemented and matched flawlessly. Attacks that bypass the shellexecution tool entirely, such as reverse-shell attacks [19], are not considered either, as they are orthogonal to the command-gating mechanism explained above.
A denylist L is a set of deny-rules: L ∈ P(R)
where R is the set of all possible deny-rules, and P(·) denotes power sets. A command c with argument list a is blocked by L, denoted as c ⊳a L, if _ r (c, a) = ⊤ r∈L
i.e., a command is blocked by a denylist if any deny-rule in the list blocks it. The internal form of denylists and deny-rules is deliberately abstracted, so that the definitions that follow are implementation-agnostic and apply uniformly across agents. The only capability a deny-rule must expose is a decidable match verdict on an invocation. How that verdict is computed (e.g., either by using prefix matching [6] or glob patterns [7]) is immaterial to our analysis. In Listing 1, each line between the brackets is a deny-rule. The pattern cat *.env* matches the command cat project.env. Therefore, the command should be blocked if it is the one undergoing command gating. The six deny-rules in this listing form a denylist together. Several notes about the definitions are worth mentioning: First, matching is a property of the surface form of a command invocation, not of its effect on the host. A denyrule fires on what a command and its arguments look like, whereas the operations an invocation achieves are settled only when it runs (§5.2). Second, a denylist is a purely negative specification. It enumerates invocations to reject, so any command invocation outside the union of its rules is admitted by default, and its protection is bounded by whatever its authors thought to enumerate.
5. Problem Fomalization Building on the previous observations, this section formalizes the problem of fragile command denylists. As mentioned in §1, the formalization should enable us to identify and observe what a denylist aims to block. Definitions in this section fix the objective for the rest of this paper, which will refer to these formal definitions as guidance when implementing our system.
5.2. Blocked Operations Before defining what a denylist aims to block, we first define what a denylist actually blocks, which should, intuitively, be the operations that the blocked commands perform. Take Listing 1 for example. All six deny-rules block commands that print the contents of files with env-suffixed names. Thus, this denylist should aim to block “printing the contents of files with env-suffixed names.” Definition 2 defines operations and blocked operations. Definition 2 (Operations and blocked operations). Given an aspect (e.g., the file system or network layers) of a host system h, an operation o is a function over the states of that aspect:
5.1. Denylists and deny-rules We begin by providing, in Definition 1, the formal definitions of denylists and denylist rules (or deny-rules for short), the elements that form them. Later explanations use Listing 1 instead of Table 1 as the running example because the auto-mode denylist is hardcoded in the private code of Claude Code (the table was discovered via experiments), and no detail is provided to validate our definitions. Definition 1 (Deny-rules and denylists). A deny-rule r is a predicate over a command with its argument list:
o:S→S
n
r : C × A → {⊤, ⊥}
where C is the set of all available commands, A is the set of all possible argument strings, and n is the number of command arguments. A command c with argument list a is blocked by r, denoted as c ⊳a r, if and only if
Here, S is the set of all possible states of that aspect, and p maps an old state to a new state. p is an operation performed by a command c with argument list a on h, denoted as c ⊢h,a p, if and only if the states before and after execution s and s′ satisfy
r (c, a) = ⊤
s′ = o(s)
4
The set of all operations performed by c on h is denoted as Oh (c). In turn, o is an operation blocked by denylist L on h, denoted as o ⊳h L, iff. ∃c ∈ Ch , a ∈ An .c ⊢h,a o ∧ c ⊳a L
where Ch is the set of all available commands on h, and A is the set of all possible argument strings. The set of all operations blocked by L on h is denoted as Oh (L).
Operation ID
Arguments
op_read op_write op_create op_delete op_copy op_chmode op_chowgr
path path, content path path path1, path2 path, mode path, owner_and_group
Table 2: Blocked operations investigated in this paper.
Instead of ephemeral behaviors, the definition focuses on the side effects that a command leaves in the host system after execution, and scopes the characterization of commands to the essential “aftermath” they cause. As an example, cat project.env and cp project.env /dev/stdout invoke different syscalls. The former writes the file content from a buffer directly to the special file descriptor the system maintains for stdout after reading it, while the latter creates a new file descriptor for /dev/stdout and writes to it. However, they leave the system in the same post-execution state where the inode of stdout holds the content read from project.env. These two commands perform the same operation. Specifically, this paper restricts the blocked operations under investigation into seven kinds of file system operations, each represented by an ID and an argument list, as shown in Table 2. File system operations are a major type of operation that malware performs [20] and are easy to record and analyze. Other operations, such as network accesses, will require complex simulations, e.g., fake HTTP servers, in the experiments and are thus not considered in this paper. Future work may further investigate the other operations. The seven kinds of operations are explained as follows: • op_read. The content of the file at path is read and printed to stdout or stderr. • op_write. content is written to the file at path. • op_create. A new file is created at path. • op_delete. The file at path is deleted. • op_copy. The file at path1 is copied to path2. • op_chmode. The mode of the file/directory at path is changed to mode. This is typically conducted by the chmod command on Linux. • op_chowgr. The owner/group of the file/directory at path is changed to owner_and_group. This is typically conducted by the chown command on Linux. As an example, all seven rules in Listing 1 correspond to a set of blocked operations that read the .env files: op_read(".env"), op_read("project.env"), ... In our implementation, arguments of blocked operations have placeholders to denote an infinite set of operations following the same pattern (see §6.1). The seven kinds cover three components of a host’s file system that can be modified: file contents, file paths, and file metadata. Several notes are worth mentioning here: First, the list does not include a kind for “code execution”, because “execution” is not a single side effect that can be
observed in the host’s file system. Instead, “execution” can indirectly leave other side effects, such as file creation or deletion. Thus, “execution” is reasonably excluded from being considered as one single operation kind. Denylists that aim to block code execution, like the one in Table 1, will be categorized as targeting multiple concrete file system operations, such as file deletion or creation. Second, the list is designed to be as minimal as possible. Thus, it does not include “moving a file,” which can be decomposed into a copy and a deletion (of the file at the original path). Third, the list does not include some less-used metainformation, such as xattr [21], to focus on the most common operations first. Then, the blocked operations B(L) are used as an approximation of the operations a denylist L aims to block, considering that a precise depiction, essentially equivalent to the intents of the authors of the denylist, would require an in-depth interview to investigate. We argue that this is a good enough approximation based on two intuitions: • First, though the authors of a denylist sometimes forget to include some commands, they should not be expected to leave an operation they want to block completely untouched, which means that there should be at least one command that performs the operation in the denylist. For example, the authors of the denylist in Table 2 forget that cp and diff can also print the content of the .env files, but they do block some obvious commands (cat, head, ...) for this purpose. • Second, authors tend to minimize the denylist, as an unnecessarily over-general denylist will block too many benign operations and hinder the normal development process, violating the design purpose of denylists in the three-list gating mechanism.
5.3. Incompletely Blocked Operations The previous subsection defines blocked operations to characterize what a denylist aims to block. This section will investigate whether a denylist can fully block an operation. Specifically, Definition 3 defines full blocked and incompletely blocked operations. Definition 3 (Fully and incompletely blocked operations). Given a denylist L and an operation o on a host system h, o is fully blocked by L on h, denoted as o ⊳ + h L, if and only if o ⊳h L ∧ ∀c ∈ Ch , a ∈ An .c ⊢h,a o =⇒ ¬c ⊳a L
where Ch is the set of all commands provided by h, and A is the set of all possible argument strings. p ⊳ + h L
5
essentially means that every command on h that may perform o has been blocked by L. In turn, o is incompletely blocked by L on h, denoted as o ⊳ − h L, if and only if,
Command with arguments
Operations
cat $N.env cp $P1 $P2 cp $P /dev/stdout
op_read($N.env) op_copy($P1, $P2) op_read($P)
Table 3: Examples of the commands (input), arguments (output), and operations (output) consumed and produced by the pipeline. The arguments and operations can contain placeholders, including $N for an arbitrary file name string, and $P, $P1, $P2 for arbitrary file paths.
o ⊳h L ∧ ∃c ∈ Ch , a ∈ An .c ⊢h,a o ∧ ¬c ⊳a L
The set of all operations fully/incompletely blocked by L is denoted as Oh+ (L)/Oh− (L). Note that a blocked operation, as defined in Definition 2 (o ⊳h L), is either fully − blocked (o ⊳+ h L) or incompletely blocked (o ⊳h L), + − + − and thus Oh (L) ∩ Oh (L) = ∅ and Oh (L) ∪ Oh (L) = Oh (L). Any command c not in L that perform an operation o ∈ Oh− (L) is called a bypass command or bypass in short, denoted as c ≻h,o L. The set of all bypasses of L that perform o on h is denoted as Bh,o (L).
6.1. Overview Input and output. The inputs of our system are a denylist L and a command c provided by the host system h that is not blocked by L (¬c ⊳a L for some argument list a). Given an operation o, our system should decide whether c ≻h,o L. If so, the command is a bypass, i.e., c ∈ Bh,o (L). For example, taking the denylist in Listing 1 and the command cp provided by the system, C MD N EEDLE will output true if cp (the c) can read the content of a file project.env (reading file as the o) because it is not blocked by the denylist (the L) and thus a bypass. This is achieved by running a shared pipeline in parallel on L and c to identify the operations they block/perform, i.e., Oh (L) and Oh (c), respectively. Then, the two sets of operations are compared, and if Oh (L) ∩ Oh (c) 6= ∅, c is a bypass of L respecting the operations in Oh (L) ∩ Oh (c). For example, as shown in Table 3, the pipeline will identify that Oh (L) is {op_read("$N.env")} for L being the denylist in Listing 1, and Oh (c) is {op_read("$P"), op_copy($P1, $P2)} for c being the command cp. Then, c ≻h,o L for o being op_read("$N.env") in Oh (L) ∩ Oh (c). Note that our implementation allows the arguments and operations to contain placeholders to represent infinite elements. Figure 2 shows an overview of the shared pipeline. The pipeline takes a command c and outputs the operations it performs with the proper arguments. The operations are then aggregated over all arguments to compute Oh (c). The commands are retrieved from entries of a denylist L (e.g., cat from cat *.env* in Listing 1) or the executable paths (e.g., cp in /usr/bin) and go through the same three stages to produce the output, validated operations and arguments: • Candidate enumeration (§6.2). Given the command, an LLM is prompted to propose a candidate operation within Table 2 that the command may perform with proper arguments. Candidate operations and arguments that have obvious errors (e.g., wrong arguments) are rejected. Overlapping or identical operations are deduplicated. The LLM is re-prompted until there are enough candidates after rejection and deduplication. • Execution-based validation (§6.3). Each command is executed in a sandbox with all placeholders in the arguments substituted by flag files and canary tokens, and the side effects it leaves are recorded. Specific oracles of the can-
A denylist that incompletely blocks some operations provides no effective protection, as an attacker can easily perform them using bypass commands. Take Listing 1 as an example. The operation op_read(".env") is incompletely blocked, as the command diff .env /dev/null can also read and print the content of .env while not being blocked. Thus, the diff command is a bypass. Several notes are worth mentioning. First, the relations o ⊳ + h L and o ⊳ − h L are not symmetric. o ⊳ − h L is not a negation of o ⊳ + h L. It has an extra precondition o ⊳ L, which excludes from consideration those operations completely ignored by the denylist, because the authors likely keep these operations untouched on purpose, for reasons such as that the operations are trivial for their projects. Accounting for operations that do not meet the precondition will result in a scope that is too large and contains excessive noise. For example, Listing 1 does not cover the operation op_read("README") at all, not because they forget it, but because reading this file does not imply any security issues. Second, bypass commands are not novel exploits and do not require vulnerabilities. They can be commands leveraged in their design usage, but for an attacker’s purpose (see the diff command). Third, a bypass command does not have to be precisely equivalent to a command blocked by the denylist and only has to perform the same blocked operation. As explained before, the cp command has semantics completely different from cat, but it can perform the same operation respecting reading and printing file contents, which makes it essentially a bypass of the denylist for the blocked op_read operations.
6. The C MD N EEDLE Pipeline Based on the formalization in the previous section, the core problem of characterizing denylist fragility is, given a denylist L and an operation o blocked by L on a host system h (o ⊳h L), to find whether bypasses exist (i.e., whether Bh,o (L) 6= ∅) and what they are if they exist (i.e., the elements of Bh,o (L)). This section explains the pipeline we use to compute Bh,o (L), The next subsection gives an overview (§6.1), followed by subsections explained the stages in the pipeline (§6.2–6.5), one for each.
6
didate operations check whether these side effects indicate that the operations are truly performed by the command. • Iterative repair (§6.4). Candidate operations that fail the validation are fed back to the LLM with error messages to get a repair. The repaired candidates then go through the validation process again. Ultimately, the operations performed by every command are compared with those blocked by the denylist to decide whether bypasses exist for that denylist (§6.5). C MD N EEDLE relies on an LLM for candidate enumeration and iterative repair, the two stages that require understanding and synthesizing shell commands. This is a deliberate design choice. The inputs to these stages (man pages, --help output, and other documentation) are unstructured text, and the command-line interfaces they describe follow widely divergent conventions. Absent an LLM, one would have to assemble a pipeline from conventional techniques to recover the relationship between a command and its arguments: natural-language processing to mine these relations from documentation [22], [23], static analysis to supply structural information when source code is available [24], and a constraint solver such as an SMT solver [25] to find argument assignments that satisfy them. Such a pipeline is unattractive on several fronts: documentation- and specification-mining techniques are known to suffer high false-positive and false-negative rates [26], [27] and demand substantial engineering effort, while the static-analysis components face well-documented scalability limits [28]. An LLM, by contrast, synthesizes shell invocations cheaply and at scale with little bespoke engineering [4]. Its principal weakness for our purpose is hallucination [29], [30], but this is precisely what the second stage neutralizes: every candidate is executed in a sandbox and checked against a predefined oracle, so unsound proposals are discarded rather than trusted.
sets of operations defined in §2. Say, two operations with placeholders represent the sets O1 and O2 . O1 duplicate O2 if O1 ⊆ O2 or vice versa. In this case, we keep only O2 , as it is more general. An extra constraint is imposed for commands retrieved from the denylist entries: the command with the proposed arguments must be blocked by the denylist. For example, cat Cargo.toml will be excluded from the denylist in Listing 1 because the denylist do not block it at all. As we argued previously, such a command is not what the denylist aims to block. The LLM will be reprompted for candidates who do not meet this requirement. On the example cp present in Table 3, the LLM may proposed a candidate argument list $P /dev/stdout with three candidate operations: op_copy("$P", "/tmp/f1"), op_read("$P"), and op_read("project.env"). op_read("$P") which contains the placeholder $P is strictly more general than op_read("project.env"), so the latter is discarded. After this stage, two candidate operations, op_copy("$P", "/tmp/f1") and op_read("$P"), with the candidate argument list $P /dev/stdout are proposed.
6.3. Execution-based Validation The validator decides, for each candidate operation, whether the command truly performs it when running on the host. Each of the seven scopes of Table 2 is equipped with an oracle. The command with arguments and the operation are instantiated by substituting the placeholders with freshly generated concrete values that serve as flag files or canary tokens (e.g., $P to ./f.txt). The command is then executed in a sandbox that records its side effects. If the operation is performed, the side effects must satisfy the oracles on the flag files and canary tokens. Figure 4 illustrates the oracles for three representative operations: • op_copy. The command is supplied with the source and target paths. The oracle passes if and only if the target exists after execution and it contains the same content as the source path. • op_write. The command is supplied with a flag file path already present in the host environment and a canary token to be written to the file. The oracle passes if and only if the canary appears in the file after execution. • op_read. A flag file is seeded with a canary token and used to substitute for the command’s file path placeholder(s). The oracle passes if and only if the canary appears in stdout or stderr. The remaining four operations in Table 2 are validated by oracles of the same form: a freshly minted canary makes the command’s effect observable, and an operation-specific oracle reports the presence or absence of that effect. Let’s continue the example input cp $P /dev/stdout and candidate scopes op_copy("$P", "/tmp/f1") and op_read("$P"). First, $P is substituted by a freshly created flag file ./f.txt containing the canary token 42. Then, the resulting side effects are checked against the oracles of the two candidate operations. The oracle for op_copy(path1, path2) requires that path2 exists and
6.2. Candidate Enumeration During the candidate enumeration stage, an LLM proposes candidate operations and arguments. The command may perform the operations with the arguments. This is done by prompting the LLM with the command’s documentation, including its help message, manual pages, and the corresponding entries on GTFOBins (a community-curated living-off-the-land attack database) [31] if any, to produce a structural list of candidate operations and arguments. The full prompt is shown in Figure 3. A caveat of the prompt is worth noting: we explicitly request that the reported operations be performed solely by this command, without chaining or pipes, to rule out cases in which the command itself does not meet the requirement but is concatenated with other commands that do. Obviously unsatisfactory candidates, such as those containing syntax errors, are discarded, and the LLM is re-prompted until it proposes a satisfactory one. Operations that are identical or overlap are deduplicated. Duplication may happen because the “operations” in the implementation could contain placeholders to denote infinite
7
Figure 2: End-to-end C MD N EEDLE pipeline. Each input (a command with arguments) flows through three stages. The candidate enumerator, an LLM conditioned on man pages, --help outputs, and GTFOBins entries, proposes candidate operations that the command may perform. The validator then executes the command against operation-specific oracles to decide whether these candidates are correct. Failures are routed through the repair loop back to the LLM with diagnostic messages, for at most T rounds. The confirmed operations are the pipeline’s output. example cat $N.env in Table 3, say, a candidate operation op_read($N.var) fails because of the wrong suffix. In the repair loop, the LLM may fix it to a correct version op_read($N.env) and succeed the re-run validation pass.
holds the same content as path1. It fails in the current case because /tmp/f1 was not created. The oracle for op_read(path) requires that the content of path is present in stdout or stderr, which is true in our case. Therefore, the candidate operation op_read("$P") is validated. Two design choices govern the validator: • Execution over LLM judgment. Validation runs the command and observes its effect rather than asking a second LLM to adjudicate. Because the candidate is itself generated by an LLM, it can exhibit the full range of LLM failure modes, such as hallucinations or outdated usage [32], and an LLM judge would share these failure modes rather than correct them. • Deliberately loose oracles. Each oracle checks only the essence of its operation. For scope_read, the essence is whether the seeded token reaches stdout/stderr. Everything orthogonal to it, such as exit codes, warnings on stderr, or incidental output, should be ignored. A command that emits a warning or returns a non-zero status while still leaking the file’s contents genuinely performs the operation, and a stricter oracle would wrongly reject it.
6.5. Matching Denylists with Bypasses The shared pipeline yields, for each command c, the operations it performs (Oh (c)), and for each denylist, the operations it blocks ( Oh (L)). For example, for c being the command cp, it computes Oh (c) being {op_read("$P"), op_copy($P1, $P2)}, and for L being the denylist in Listing 1, it computes Oh (L) is {op_read("$N.env")}. Then, the two sets are simply intersected as Oh (c) ∩ Oh (L), and if the intersection is not empty, c is a bypass of L. For each o ∈ Oh (c) ∩ Oh (L), c ≻h,L L. In our example, Oh (c) ∩ Oh (L) = {op_read("$N.env")}, and thus cp is a bypass of Listing 1 that can read .env files. Note that in the intersection, the most concrete values that contain placeholders are preserved (e.g., op_read("$N.env") instead of op_read("$P") in the intersection).
7. Evaluation
6.4. Repair Loop
We collected a dataset of real-world command denylists used by terminal AI agents from GitHub. We applied C MD N EE DLE on it to characterize the fragility of command denylists caused by incompletely blocked operations at scale and to draw insights for future practice and defenses.
A pair of a command and a candidate operation that fails validation, either because it raises errors or does not satisfy the operation’s oracle, is returned to the LLM together with the captured stdout, stderr, the return code, and any validator-side diagnostics such as parsing errors. The LLM is asked to emit a single revised candidate that addresses the observed failure while still targeting the same operation, and the revision is sent through execution-based validation again. This loop repeats for at most M (a configurable constant) rounds per original candidate and exits early as soon as a revision passes the oracle or once the LLM decides the candidate is not fixable. The repair loop’s purpose is to recover near-misses, such as unbalanced quotes or incorrectly used flags, which a validate-once pipeline would discard. For the
7.1. Research Questions The evaluation is framed against the formalization in §5 and targets the following research questions: • RQ1 (Problem severity). How severe is the command denylist fragility problem among the collected dataset? • RQ2 (Severity in different cases). Do factors such as how famous the project of a denylist is or what the agent is affect the severity of the problem?
8
Figure 4: Execution-based validation. Each candidate is instantiated with a freshly generated flag file (in blue)/canary token (in red). The command is executed in a sandbox. A per-scope oracle inspects the pre- and post-execution states to determine whether the operation was actually performed. Stars 0–99 100–999 ≥1000
Figure 3: Abbreviated prompt template used in candidate enumeration. The LLM receives the command’s documentation and is asked to propose a structured list of operation and argument candidates. Double braces mark the inserted contents.
Total
Claude Code
Codex
OpenCode
Total
936 57 7
130 4 2
563 8 2
1629 69 11
1,000
136
573
1709
Table 4: Stars of the GitHub repositories where the denylists are collected from.
• RQ3 (Root causes). What are the root causes of incompletely blocked operations? • RQ4 (Burden of fixes). How difficult is it to repair a fragile denylist? Specifically, how many bypass commands must be added to fully block an operation?
among repositories with different numbers of GitHub stars and the agent the repositories use. • RQ3. We propose three hypotheses for the root causes of incompletely blocked operations, and compute the relations between the likelihood of a command becoming a bypass and the proxies of the hypothetical causes. • RQ4. The repair burden is proxied by the number of commands that the denylist must block in addition to cover all incompletely blocked operations.
7.2. Metrics We answer each RQ using metrics computed from the operations and bypasses identified by C MD N EEDLE according to the formalization in §5: • RQ1. Severity is measured by incompletely-blocked rate: for each operation o, the fraction of denylists L that incompletely block it among all that block it, i.e., L o ⊳− h L k{L|o ⊳h L}k
7.3. Dataset Construction The evaluation targets denylists for three widely used AI agents: Claude Code, Codex, and OpenCode. We collect the denylists from GitHub in their per-project setting files. Other AI agents, such as Copilot CLI [33] and Antigravity [34], also follow the three-list gating architecture. However, their denylists are either set globally in the user’s home directory rather than the project directory (and thus not uploaded to remote repositories) or specified on the command line rather than in a configuration file. It is worth noting that we
where k · k means the size of a set. • RQ2. We compute the incompletely-blocking rate for each repository, and measure whether and how it varies
9
Figure 6: For each operation, the number of denylists that fully block it and incompletely block it. Figure 5: Number of denylists that block each operation, grouped by agent. The op_ prefix in the operation IDs is omitted.
laptop would have. C MD N EEDLE instantiates the command universe Ch of Definition 3 by walking the executable search directories. During execution-based validation (§6.3), each candidate is instantiated with freshly minted flag files and canary tokens in a per-run temporary directory within a sandbox using Bubblewrap [36] and OverlayFS [37].
did not include OpenClaw, as it was being developed at an extremely rapid pace (one release per day) at the time this paper was written, and its architecture and configuration schema are subject to drastic changes [35].
Models, costs, and time consumption. The candidate enumeration and the iterative repair stages are driven by Claude Sonnet 4.6 with default settings. The repair loop is capped at 5 iterations per candidate, exiting early on the first revision that passes the oracle or when the model reports the candidate as unfixable. We access the model through AWS Bedrock and run the pipeline with 16 threads in parallel. The full evaluation consumed 3 wall-clock hours and $25 in inference.
Representativeness The collected denylists well reflect how users use AI agents in the real world. In total, 24,453 repositories we find on GitHub contain denylists for Claude Code, 142 for Codex, and 1,253 for OpenCode. The number of repositories that contain Claude Code denylists is capped at 1000 via random sampling to limit the time required for the experiments. It is worth noting that the shell-command denylist is a rather new feature of Codex, marked as experimental [9] at the time this paper is being written, so the number of collected denylists of Codex is relatively small. Before the feature was shipped, Codex lacked a gating mechanism for shell commands and relied solely on the AI model’s decisions and a sandbox that restricted access by file path. Table 4 shows the distributions of the numbers of GitHub stars these repositories receive, which typically have long tails: most have only a few stars, while a few receive many (≥1,000). The repositories use 152 programming languages, and cover 1,643 topics (tagged by their maintainers). Figure 5 shows, for each of the seven operations in Table 2, the number of denylists that cover it (i.e., p ∈ Bh (L)). All operations are well represented, though with an imbalance across kinds that we attribute to users’ preferences and expectations: op_delete, for instance, is covered by the most denylists, which is intuitively explained by the destructive nature of deletion. Users tend to be more alert to such operations.
7.5. Results and Findings 7.5.1. RQ1 (Problem severity) RQ1 studies how often a blocked operation is left incompletely blocked. For each operation, we compute the incompletely blocked rate for each kind of operation. Figure 6 shows the number of denylists that incompletely or fully block an operation. Among the seven types of operations, denylists that incompletely block an operation account for the largest share (69.0–98.6%). The result means that most denylists overlooked at least one bypass for the operation they target, invalidating their protection against it. Finding 1. The problem of fragile command denylists caused by incompletely blocked operations is severe. 69.0– 98.6% of the denylists that target to block an operation overlook at least one bypass command for that operation. 7.5.2. RQ2 (Severity in different cases) RQ2 studies whether fragility is concentrated in particular kinds of projects or agents. Figure 7 reports, for each operation, the incompletely blocked rate, broken down by repository popularity (Figure 7a) and by agent (Figure 7b). The incompletely-blocked rate stays high across all three star number tiers in Figure 7a, though in 4 out of
7.4. Experiment Settings and Costs Host environments All measurements are taken with respect to a reference host h: a Debian 13.5 Docker image with the default GNU userland, which provides the executables a developer running a terminal AI agent on a typical Linux
10
initial insights and directions for future research. The hypotheses are as follows: • H1 (Ignorance). The denylist authors do not know that the bypass commands exist. • H2 (Versatility). A command that performs several operations is deliberately left unlisted so that it remains usable for the operations the authors do not wish to block, even though it may perform one that is blocked. • H3 (Semantic gap). The command’s ability to perform the operation is hidden from its apparent purpose, so the author never associates it with the operation in the first place. H1 (Ignorance). Intuitively, if the denylists overlook bypasses because their authors do not know that the bypass commands exist, then less popular (and thus less known) commands should be easier to become a bypass. To validate this hypothesis, we collect the rank of a command in the Debian Popularity Contest [38] as a proxy for the popularity of the commands. Less popular commands correspond to larger rank values. Then, we apply equal-frequency binning [39] to split the commands into six bins based on their ranks and use a logistic regression model to estimate the frequency of bypass commands within each bin. The fitted model will predict, for a command at a given popularity rank, the probability that it is a bypass command for the operation. Figure 8 shows the result. The lines are the fitted probability that a command bypasses the operation’s denylist as a function of its popularity rank (on a log scale). The shadows are the corresponding 95% confidence bands of these fitted curves. The points are the empirical bypass rates for the six equal-frequency bins, each plotted at its mean rank, and serve as a calibration reference for the fitted curve. The figure shows that for every operation except op_delete the fitted probability rises with rank (the curves are especially steep for op_read, op_write, and op_create), meaning that less popular (greater rank values) commands are markedly more likely to bypass the denylist, whereas under Delete popularity carries essentially no signal. This supports H1: the fitted probability of being a bypass rises with rank for all seven operations, and the logistic regression across operations gives highly significant positive slopes (lowest p being 1.02 × 10−6). Thus, less popular commands are markedly more likely to become bypasses.
(a) Incompletely blocked operations with respect to stars.
(b) Incompletely blocked operations with respect to agents.
Figure 7: Breakdown of incompletely blocked rate. the seven kinds of operations, projects with the most stars (≥ 1, 000) have the lowest incompletely-blocked rate. However, even the lowest incomplete-blocked rate for the operations is still higher than 44.4%. Similarly, the incomplete-blocked rate remains uniformly high across all three agents, consistently exceeding 46.2%, and none shows a tendency to lower values.
H2 (Versatility). Intuitively, if denylists overlook bypasses because multi-purpose commands are deliberately left unlisted to keep them usable for the operations the authors do wish to permit, then a command that spans more operations should be more likely a bypass across the denylists. To validate this hypothesis, for each command c, we count the number of denylists it bypasses (k{L|c ≻h,o L}k), aggregated across all seven operations, and group the commands according to how many operations they perform: those performing more than one operation (multi-op) versus a single operation (single-op). We compare the two groups with a Mann-Whitney U test [40] and report two statistics. The pvalue indicates whether the two groups’ bypass counts differ,
Finding 2. The fragile denylist problem is constantly severe across different projects and agents, as reflected by the uniformly high incomplete-blocked rates (> 44.4% across projects and > 46.2% across agents). Denylists from projects with more stars sometimes are less fragile, though. 7.5.3. RQ3 (Root causes) To find the root causes of incompletely blocked operations, we first propose several explanatory hypotheses and then validate them statistically. Note that the hypotheses are not meant to be a complete list but rather to provide
11
and Cliff’s δ is an effect size in [−1, 1] that captures how large the difference is and in which direction, with a positive value meaning multi-op commands bypass more denylists than single-op ones. Typically, a p-value below 0.05 indicates a statistically significant difference, and the magnitude of Cliff’s δ is conventionally read as negligible below 0.147, small below 0.33, medium below 0.474, and large otherwise. Table 5 shows the result. Multi-op commands bypass a median of 664 denylists, more than twice the 281 bypassed by single-op commands, and the gap is statistically significant (Mann-Whitney U =2482, p=0.006) with a non-negligible effect size (Cliff’s δ=0.30). This supports H2: a command that performs several operations remains available as a bypass for any of those operations that a given denylist does not cover, so the more operations a command spans, the more denylists it slips through. H3 (Semantic gap). Intuitively, if denylists overlook bypasses because a command’s ability to perform an operation is hidden behind its apparent purpose, then the overlooked bypasses should concentrate among commands that do not advertise the operation in their everyday documentation. To validate this hypothesis, we use the community-maintained TLDR pages [41], which list each command’s most common usages, as a proxy for a command’s apparent purpose. We rerun the C MD N EEDLE pipeline using only the TL;DR documentation in the prompts to identify the obvious operations each command performs. Then, for each operation, we split the host commands into those whose TLDR page states the operation and those whose page does not, and compute the percentage of validated bypasses in each group. If a semantic gap were the dominant cause, bypasses should be more frequent among commands whose TLDRs do not mention the operation. Table 6 shows the result, which runs opposite to what H3 predicts. For every operation, validated bypasses are far more common among commands whose TLDR states the operation than among commands whose TLDR does not, at 40.1% versus 5.7% overall and, for instance, 45.0% versus 15.3% for op_read. The overlooked bypasses are therefore mostly commands whose TL;DR already states the operation, so their capabilities are advertised rather than hidden, and the semantic gap is at best a secondary cause. This rejects H3: the commands authors miss are documented in plain sight.
Command group
N
Median [IQR]
Multi-op (> 1) Single-op (= 1)
35 109
664 [280, 1013] 281 [281, 497]
Mann–Whitney U p-value Cliff’s δ
2482 0.006 0.30
Table 5: Denylists that a command can bypass, grouped by whether the command performs multiple operations or a single operation. N is the number of commands in each group, and IRQ means interquartile range. Re ad
TLDR Op Other
45.0 15.3
W
rit e
Cr eat e
23.7 7.9
44.6 9.1
De le
te
35.1 3.7
Ch Co Ch py ow mo gr de
75.0 4.0
40.0 1.3
40.0 1.3
Table 6: Percentage of host commands that are validated bypasses, split by whether the operation appears in the command’s TLDR documentation. TLDR Op covers commands whose TLDR page states the operation. Other covers those whose does not. repair burden may therefore be proxied by the number of bypass commands a defender must add to the denylist. Table 7 reports the minimum, mean, and maximum number of distinct validated bypasses that must be added per operation. On the reference host, fully blocking op_read takes, on average, 217 additional commands (up to 222), and closing every operation that a denylist blocks at once takes, on average, 180 commands (up to 603). Even the cheaper operations are far from a one-line fix: op_write and op_create average 97 and 98 bypasses each. The numbers are substantial for anyone who intends to fix the denylists. Finding 4. Repairing a fragile denylist by enumerating bypasses requires adding numerous commands (217 on average for op_read).
8. Related Work
Finding 3. Two causes of the fragility can be statistically confirmed. First, ignorance (H1): the authors of the denylists do not know about the less popular commands that become bypasses. Second, command versatility (H2): commands that perform several operations in bypass are deliberately unlisted for their benign uses, while these commands may be used as bypasses to perform the operations the denylists aim to block.
Agent safety and tool-use sandboxing. A growing line of work studies how to make AI agents safe when they are granted access to real tools. Proposals include sandboxed execution environments for agent actions [42], [43], capability-scoped tool registries [44], [45], [46], and human-in-the-loop approval workflows [47], [48], [49]. A parallel line of work replaces or augments the human approver with an LLM-based judge that inspects each proposed command in context [50]. Such judges sidestep approval fatigue, but they reason over the same command-string surface and inherit the hallucination and adversarial-suffix failure modes noted in §2.
7.5.4. RQ4 (Burden of fixes) RQ4 examines the cost of repairing a fragile denylist. Fully blocking an operation (o ⊳+ h L) requires the denylist to match every host command that performs o (c ⊢h,a o. The
LLM red-teaming and jailbreaks. Red-team studies of LLMs have primarily targeted the model’s own output, including prompt injection [51], [52], jailbreaks that elicit prohibited content [53], [15], and tool-misuse via indirect prompting [54], [17].
12
Figure 8: Fitted probability that a host command is a bypass for each operation as a function of its popularity rank (log scale), with 95% CIs and empirical rates. E is the event that a command c becomes a bypass for any denylists (c ≻h,o L for any L), and Pr[·] is the probability of an event.
Living-off-the-land attacks. The community databases GTFOBins [31] and LOLBAS [55] catalogue standard binaries that can be abused for living-off-the-land attacks. Prior work has used these databases to study real intrusions [56].
leave it for future research to study the operations this paper didn’t consider. Better command gating. Our results indicate that the three-list command-gating mechanism itself is quite fragile, considering that the denylists overlook bypasses. However, this mechanism is prevalent due to the huge flexibility it provides. Future research and practice may investigate how to pair the three-list command gating with other techniques, such as the aforementioned capability-based sandbox [57] and LLM auditor [3], to achieve a better trade-off between flexibility and security.
9. Limitations and Future Work
10. Conclusion
Completeness of LLM-proposed bypasses. C MD N EEDLE discovers bypasses by prompting an LLM to enumerate candidate commands and operations, and an LLM-based generator can only recall a subset of the bypasses a host actually admits. Our measurements are therefore a lower bound on denylist fragility: the incompletely-blocked rates and the repair burdens we report can only grow as enumeration improves, and the severity of the problem should be greater than what we have reported, which is already rather concerning. Crucially, this incompleteness affects only the magnitude of the effect, not its direction, considering the validation stage in our pipeline. Our study may encounter false negatives, but all the reported bypasses are true positives. Scope of operations. We deliberately restrict the blocked operations under study to seven file-system operations and exclude others, including, most notably, network operations such as data exfiltration and reverse connections, to keep the evaluation tractable. File-system effects are durable and directly observable in the post-execution host state, which admits the small, robust oracles our validator relies on. In contrast, network effects would require modeling external endpoints and substantially heavier instrumentation. We
This paper presents the first systematic characterization of command denylist fragility in terminal AI agents. It formalizes the operations that denylists aim to block and defines when a command can be a bypass to perform these operations. It proposes C MD N EEDLE, an LLM-driven pipeline that automatically enumerates and validate bypasses for denylists. Applying C MD N EEDLE to 1,709 real-world denylists (13,332 rules) collected from GitHub, we find that the denylist fragility problem is severe and pervasive: 69.0– 98.6% of the denylists that target an operation overlook at least one validated bypass. We also study and confirm several root causes of the problem. In addition, we find that fixing a fragile denylist will require completing the denylist to match numerous extra bypass commands, which makes the burden to fix these denylists heavy. These findings show that a static command-string denylist is inadequate as the load-bearing layer of agent command gating, and they motivate a shift toward better approaches that combine the listbased command gating mechanism with other techniques to achieve a better trade-off between flexibility and security for terminal AI agents. We hope our pipeline, dataset, and findings support future research and practice in this field.
Re ad
Min Mean Max
200 217 222
W
rit e
Cr eat e
5 97 106
63 98 180
De le
te
38 54 58
C Ch Co py hmo ow gr de
27 39 40
1 3 13
7 11 12
Table 7: Min/mean/max number of bypass commands.
13
Ethics Considerations
[14] M. Andriushchenko, F. Croce, and N. Flammarion, “Jailbreaking leading safety-aligned LLMs with simple adaptive attacks,” in International Conference on Learning Representations, vol. 2025, 2025, pp. 40 116–40 143. [15] A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson, “Universal and transferable adversarial attacks on aligned language models,” Dec. 2023, arXiv:2307.15043 [cs.CL]. [Online]. Available: http://arxiv.org/abs/2307.15043 [16] J. Shi, Z. Yuan, G. Tie, P. Zhou, N. Z. Gong, and L. Sun, “Prompt injection attack to tool selection in LLM agents,” arXiv preprint arXiv:2504.19793, 2025. [Online]. Available: https://arxiv.org/abs/2504.19793 [17] Q. Zhan, Z. Liang, Z. Ying, and D. Kang, “InjecAgent: Benchmarking indirect prompt injections in tool-integrated large language model agents,” in Findings of the Association for Computational Linguistics: ACL 2024, L.-W. Ku, A. Martins, and V. Srikumar, Eds. Bangkok, Thailand: Association for Computational Linguistics, Aug. 2024, pp. 10 471–10 506. [18] A. Khan, “Clinejection — compromising cline’s production releases just by prompting an issue triager,” Feb. 2026. [Online]. Available: https://adnanthekhan.com/posts/clinejection/ [19] “Command and scripting interpreter, technique T1059 enterprise | MITRE ATT&CK®.” [Online]. Available: https://attack.mitre.org/techniques/T1059/ [20] U. Bayer, I. Habibi, D. Balzarotti, E. Kirda, and C. Kruegel, “A view on current malware behaviors,” in LEET, 2009. [21] “xattr(7) - Linux manual page.” [Online]. Available: https://man7.org/linux/man-pages/man7/xattr.7.html [22] E. Wong, L. Zhang, S. Wang, T. Liu, and L. Tan, “DASE: Documentassisted symbolic execution for improving automated software testing,” in 2015 37th IEEE International Conference on Software Engineering (ICSE), vol. 1. IEEE, 2015, pp. 620–631. [23] R. Pandita, X. Xiao, H. Zhong, T. Xie, S. Oney, and A. Paradkar, “Inferring method specifications from natural language API descriptions,” in 2012 34th International Conference on Software eEgineering (ICSE). IEEE, 2012, pp. 815–825. [24] L. Tan, D. Yuan, G. Krishna, and Y. Zhou, “/*icomment: bugs or bad comments?*/,” in Proceedings of twenty-first ACM SIGOPS symposium on Operating systems principles. Stevenson Washington USA: ACM, Oct. 2007, pp. 145–158. [25] D. Monniaux, “A survey of satisfiability modulo theory,” in Computer Algebra in Scientific Computing, V. P. Gerdt, W. Koepf, W. M. Seiler, and E. V. Vorozhtsov, Eds. Cham: Springer International Publishing, 2016, pp. 401–425. [26] B. Johnson, Y. Song, E. Murphy-Hill, and R. Bowdidge, “Why don’t software developers use static analysis tools to find bugs?” in 2013 35th International Conference on Software Engineering (ICSE). IEEE, 2013, pp. 672–681. [27] S. Amann, H. A. Nguyen, S. Nadi, T. N. Nguyen, and M. Mezini, “A systematic evaluation of static API-misuse detectors,” IEEE Transactions on Software Engineering, vol. 45, no. 12, pp. 1170–1188, 2018. [28] A. Gosain and G. Sharma, “Static analysis: a survey of techniques and tools,” in Intelligent Computing and Applications, D. Mandal, R. Kar, S. Das, and B. K. Panigrahi, Eds. New Delhi: Springer India, 2015, pp. 581–591. [29] Z. Ji, T. Yu, Y. Xu, N. Lee, E. Ishii, and P. Fung, “Towards mitigating LLM hallucination via self reflection,” in Findings of the Association for Computational Linguistics: EMNLP 2023, 2023, pp. 1827–1843. [Online]. Available: https://aclanthology.org/2023.findings-emnlp.123/ [30] Y. Bang, Z. Ji, A. Schelten, A. Hartshorn, T. Fowler, C. Zhang, N. Cancedda, and P. Fung, “HalluLens: LLM hallucination benchmark,” in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025, pp. 24 128–24 156. [Online]. Available: https://aclanthology.org/2025.acl-long.1176/
All the experiments, including those comparing different techniques on the bug-injected benchmarks, are conducted locally. No harm is done to real-world systems. Also, we have responsibly disclosed the new vulnerabilities (and bugs) to relevant stakeholders, including Anthropic and the project that uses the denylist in Listing 1, and at this time of writing, the disclosure and possible countermeasure are still under review and consideration.
References [1]
M. A. Merrill, A. G. Shaw, N. Carlini, B. Li, H. Raj, I. Bercovich, L. Shi, J. Y. Shin, T. Walshe, E. K. Buchanan, J. Shen, G. Ye, H. Lin, J. Poulos, M. Wang, M. Nezhurina, J. Jitsev, D. Lu, O. M. Mastromichalakis, Z. Xu, Z. Chen, Y. Liu, R. Zhang, L. L. Chen, A. Kashyap, J.-L. Uslu, J. Li, J. Wu, M. Yan, S. Bian, V. Sharma, K. Sun, S. Dillmann, A. Anand, A. Lanpouthakoun, B. Koopah, C. Hu, E. Guha, G. H. S. Dreiman, J. Zhu, K. Krauth, L. Zhong, N. Muennighoff, R. Amanfu, S. Tan, S. Pimpalgaonkar, T. Aggarwal, X. Lin, X. Lan, X. Zhao, Y. Liang, Y. Wang, Z. Wang, C. Zhou, D. Heineman, H. Liu, H. Trivedi, J. Yang, J. Lin, M. Shetty, M. Yang, N. Omi, N. Raoof, S. Li, T. Y. Zhuo, W. Lin, Y. Dai, Y. Wang, W. Chai, S. Zhou, D. Wahdany, Z. She, J. Hu, Z. Dong, Y. Zhu, S. Cui, A. Saiyed, A. Kolbeinsson, J. Hu, C. M. Rytting, R. Marten, Y. Wang, A. Dimakis, A. Konwinski, and L. Schmidt, “Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces,” Jan. 2026. [Online]. Available: https://arxiv.org/abs/2601.11868v1
[2]
Y. Liu, Y. Zhao, Y. Lyu, T. Zhang, H. Wang, and D. Lo, “”Your AI, My Shell”: Demystifying prompt injection attacks on agentic AI coding editors,” Apr. 2026, arXiv:2509.22040 [cs.CR]. [Online]. Available: http://arxiv.org/abs/2509.22040
[3]
“Claude Code auto mode: a safer way to skip permission.” [Online]. Available: https://www.anthropic.com/engineering/claude-code-auto-mode
[4]
M. Sladić, V. Valeros, C. Catania, and S. Garcia, “LLM in the shell: Generative honeypots,” in 2024 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW), Jul. 2024, pp. 430– 435, iSSN: 2768-0657.
[5]
Z. Cheng, H. Wang, Z. Liu, X. Wang, X. Zhu, Y. Guo, W. Lin, J. Z. Pan, and Y. Wang, “Terminal-world: Scaling terminal-agent environments via agent skills,” May 2026, arXiv:2605.20876 [cs.CL]. [Online]. Available: http://arxiv.org/abs/2605.20876
[6]
“Codex | AI coding partner from OpenAI.” [Online]. Available: https://openai.com/codex/
[7]
“OpenCode | the open source AI coding agent.” [Online]. Available: https://opencode.ai/
[8]
“Configure permissions.” [Online]. https://code.claude.com/docs/en/permissions
[9]
“Rules – codex | OpenAI developers.” [Online]. Available: https://developers.openai.com/codex/rules
[10] “Permissions,” Jun. 2026. https://opencode.ai/docs/permissions/
[Online].
[11] “Policy reference agentsh.” [Online]. https://www.agentsh.org/docs/policy-reference/
Available:
Available: Available:
[12] A. Reeves, P. Delfabbro, and D. Calic, “Encouraging employee engagement with cybersecurity: How to tackle cyber fatigue,” Sage Open, vol. 11, no. 1, p. 21582440211000049, Jan. 2021. [13] G. Chrysos, Y. Li, E. Ishii, X. Du, and K. P. Sycara, “Agentic AI in the wild: From hallucinations to reliable autonomy,” Dec. 2025.
14
[31] “GTFOBins.” [Online]. Available: https://gtfobins.org/
[52] T. Geng, Z. Xu, Y. Qu, and W. E. Wong, “Prompt injection attacks on large language models: A survey of attack methods, root causes, and defense strategies,” Computers, Materials, & Continua, vol. 87, no. 1, 2026.
[32] Z. Zhang, C. Wang, Y. Wang, E. Shi, Y. Ma, W. Zhong, J. Chen, M. Mao, and Z. Zheng, “LLM hallucinations in practical code generation: Phenomena, mechanism, and mitigation,” Proceedings of the ACM on Software Engineering, vol. 2, no. ISSTA, pp. ISSTA022:481– ISSTA022:503, Jun. 2025.
[53] A. Wei, N. Haghtalab, and J. Steinhardt, “Jailbroken: How does LLM safety training fail?” Advances in Neural Information Processing Systems, vol. 36, pp. 80 079–80 110, 2023.
[33] “Allowing and denying tool use.” [Online]. Available: https://docs-internal.github.com/en/copilot/how-tos/copilot-cli/use-copilot-cli/allowing-tools [54] K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, “Not what you’ve signed up for: Compromising real[34] “Google antigravity documentation.” [Online]. Available: world LLM-integrated applications with indirect prompt injection,” https://antigravity.google/docs in Proceedings of the 16th ACM Workshop on Artificial Intelligence [35] “openclaw/openclaw,” Jun. 2026, originaland Security. Copenhagen Denmark: ACM, Nov. 2023, pp. 79–90. date: 2025-11-24T10:16:47Z. [Online]. Available: [55] “LOLBAS.” [Online]. Available: https://lolbas-project.github.io/ https://github.com/openclaw/openclaw [56] T. Ongun, J. W. Stokes, J. Bar Or, K. Tian, F. Tajaddodianfar, J. Neil, C. Seifert, A. Oprea, and J. C. Platt, “Living-off-the-land command detection using active learning,” in 24th International Symposium on Research in Attacks, Intrusions and Defenses. San Sebastian Spain: ACM, Oct. 2021, pp. 442–455.
[36] “containers/bubblewrap: Low-level unprivileged sandboxing tool used by flatpak and similar projects.” [Online]. Available: https://github.com/containers/bubblewrap [37] “Overlay filesystem — the Linux kernel documentation.” [Online]. Available: https://docs.kernel.org/filesystems/overlayfs.html [38] “Debian popularity https://popcon.debian.org/
contest.”
[Online].
[57] “Sandbox – codex | OpenAI developers.” [Online]. Available: https://developers.openai.com/codex/concepts/sandboxing
Available:
[39] J. Han, M. Kamber, and J. Pei, Data mining: Concepts and techniques. Morgan Kaufmann, 2006, vol. 10. [40] M. Hollander, D. A. Wolfe, and E. Chicken, Nonparametric statistical methods. John Wiley & Sons, 2013. [41] “tldr-pages/tldr,” Jun. 2026, original-date: 2013-12-08T07:34:43Z. [Online]. Available: https://github.com/tldr-pages/tldr [42] Y. Ruan, H. Dong, A. Wang, S. Pitis, Y. Zhou, J. Ba, Y. Dubois, C. Maddison, and T. Hashimoto, “Identifying the risks of LM agents with an LM-emulated sandbox,” in International Conference on Learning Representations, vol. 2024, 2024, pp. 27 031–27 098. [43] Y. Wu, F. Roesner, T. Kohno, N. Zhang, and U. Iqbal, “IsolateGPT: An execution isolation architecture for LLM-based agentic systems,” Jan. 2025, arXiv:2403.04960 [cs.CR]. [Online]. Available: http://arxiv.org/abs/2403.04960 [44] T. Shi, J. He, Z. Wang, H. Li, L. Wu, W. Guo, and D. Song, “Progent: Securing AI agents with privilege control,” May 2026, arXiv:2504.11703 [cs.CR]. [Online]. Available: http://arxiv.org/abs/2504.11703 [45] J. Zhu, K. Tseng, G. Vernik, X. Huang, S. G. Patil, V. Fang, and R. A. Popa, “MiniScope: a least privilege framework for authorizing tool calling agents,” Dec. 2025, arXiv:2512.11147 [cs.CR]. [Online]. Available: http://arxiv.org/abs/2512.11147 [46] Z. Ji, D. Wu, W. Jiang, P. Ma, Z. Li, Y. Gao, S. Wang, and Y. Li, “Taming various privilege escalation in LLM-based agent systems: a mandatory access control framework,” Jan. 2026, arXiv:2601.11893 [cs.CR]. [Online]. Available: http://arxiv.org/abs/2601.11893 [47] X. Weng, “What you approve is what executes: Consent integrity for black-box LLM agents,” Jun. 2026, arXiv:2606.02668 [cs.CR]. [Online]. Available: http://arxiv.org/abs/2606.02668 [48] P. Wang, Y. Li, and Y. Tian, “Reframing LLM agent security as an agent-human interaction problem,” May 2026, arXiv:2605.24309 [cs.CR]. [Online]. Available: http://arxiv.org/abs/2605.24309 [49] E. Lee, D. Kim, W. Kim, and I. Yun, “Takedown: How it’s done in modern coding agent exploits,” Sep. 2025, arXiv:2509.24240 [cs.CR]. [Online]. Available: http://arxiv.org/abs/2509.24240 [50] L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, and E. Xing, “Judging LLM-as-a-judge with MT-Bench and Chatbot Arena,” Advances in Neural Information Processing Systems, vol. 36, pp. 46 595–46 623, 2023. [51] H. Hong, S. Feng, N. Naderloui, S. Yan, J. Zhang, B. Liu, A. Arastehfard, H. Huang, and Y. Hong, “SoK: Taxonomy and evaluation of prompt security in large language models,” 2025. [Online]. Available: https://arxiv.org/abs/2510.15476
15