arXiv:2606.11817v1 [cs.CR] 10 Jun 2026
Grammar-Constrained Decoding Can Jailbreak LLMs into Generating Malicious Code Yitong Zhang*
Shiteng Lu*,‡
Jia Li†
College of AI Tsinghua University Beijing, China [email protected]
Yingcai Honors College University of Electronic Science and Technology of China Chengdu, China [email protected]
College of AI Tsinghua University Beijing, China [email protected]
Decoding (GCD) [15–17] to induce LLMs into generating malicious code at a low cost. GCD was originally designed to improve the reliability of code generation by constraining LLMs to produce outputs that conform to a target grammar [18, 19], and it is now supported by many mainstream inference frameworks such as V LLM [20] and SGL ANG [21]. However, we find that this benign reliability mechanism can unexpectedly become an attack surface. When an LLM is asked to generate malicious code, simply applying GCD with a standard code grammar can prevent the model from expressing its refusal behavior and cause it to produce malicious code, even though the grammar itself is entirely benign. Our evaluation shows that CodeSpear can easily jailbreak 10 popular LLMs (e.g., GPT5 [22], MiniMax-M2.7 [23], and Qwen2.5-Coder-32B [24]) and increase the attack success rate by about 30% on average. We attribute the success of CodeSpear to a limitation of existing safety alignment [25, 26]: it is almost exclusively grounded in the natural-language modality. Existing safety alignment typically teaches LLMs to respond to malicious requests with natural-language refusals such as “I cannot assist with that”, implicitly assuming that natural language remains available at inference time [25, 27]. However, GCD breaks this assumption. Once a code grammar is enforced, natural-language refusals fall outside the valid output space, and the model can no longer express the refusal behavior it learned during alignment. I. I NTRODUCTION The model is therefore forced to continue generation in the Large Language Models (LLMs) have demonstrated im- code modality, where it has not been explicitly aligned to pressive capabilities across a wide range of tasks and are behave safely. This explains why a benign code grammar can increasingly deployed in real-world applications [1–4]. At make CodeSpear effective, and raises a key question: how the same time, growing evidence shows that LLMs can be should safety alignment be performed in the code modality jailbroken to bypass safety alignment and produce harmful when natural-language refusals are unavailable? To this end, we propose CodeShield, a safety alignment content [5–8]. This risk becomes especially concerning in code generation [9–12], where harmful outputs are not merely approach for the code modality that trains the model to textual instructions but executable programs that can be directly generate honeypot code against CodeSpear. Honeypot code is semantically harmless and structurally diverse: it does not weaponized against digital systems [13, 14]. In this paper, we uncover a new jailbreak attack, termed implement the malicious request, and it shows diverse syntactic CodeSpear, that leverages widely used Grammar-Constrained structures. This design directly addresses the two requirements of safe behavior in the code modality. ❶ First, the response must * Equal contribution: Yitong Zhang proposed the idea and wrote the paper; remain harmless even when the model is forced to generate Shiteng Lu implemented the approaches and ran most of the experiments. code. ❷ Second, the response must be hard to remove by ‡ This work was done while Shiteng Lu was an intern at the College of AI, grammar tightening. This second requirement means that safe Tsinghua University. † Corresponding author. behavior should not be bound to a fixed code template. For
Abstract—Large Language Models (LLMs) are increasingly used for code generation, raising concerns that they may be misused to produce malicious code. Meanwhile, GrammarConstrained Decoding (GCD) has been widely adopted to improve the reliability of LLM-generated code by enforcing syntactic validity. In this paper, we reveal a counterintuitive risk: this reliability-oriented technique can itself become an attack surface. We uncover a new jailbreak attack, termed CodeSpear, that exploits GCD to induce LLMs into generating malicious code. Our experiments show that simply applying a benign code grammar constraint can effectively jailbreak LLMs. To address this vulnerability, we propose CodeShield, a safety alignment approach that robustly preserves safe behavior even under attacker-controlled grammar constraints. CodeShield aligns the model in the code modality by teaching it to generate honeypot code under GCD. Such code is semantically harmless, so it does not implement the malicious request, and structurally diverse, so it is difficult to suppress through grammar tightening. At the same time, CodeShield still preserves natural-language refusals when natural language is available. Experiments on 10 popular LLMs across 4 benchmarks show that CodeSpear outperforms representative jailbreak baselines and increases the attack success rate by more than 30 percentage points on average. CodeShield also restores safety under CodeSpear while preserving benign utility. Our findings reveal a fundamental risk of GCD and call for greater attention to its potential security implications. Index Terms—grammar-constrained decoding, jailbreak attack, safety alignment, malicious code generation
example, teaching the model to always generate a refusal token sequences accepted by G. At each decoding step t, GCD comment or a pass statement may appear safe and natural, but efficiently computes the tokens that keep the current prefix such behavior is tied to a narrow syntactic pattern. An attacker extendable to some valid program [15]: can simply use a tightened grammar that forbids comments or VG (y<t ) = { v ∈ V | ∃ y>t s.t. (y<t , v, y>t ) ∈ L(G) } , (2) pass statements to reopen the attack (See Section VI-B). By contrast, honeypot code gives the model many harmless ways and masks all other invalid tokens by setting their probabilities to stay within the code modality, making the safe behavior to zero. Typically, the next token is then sampled from the difficult to suppress without also excluding many structures renormalized distribution as follows: needed by malicious programs (See Section VII-A). PM (yt | p, y<t ) · I[yt ∈ VG (y<t )] G P To evaluate CodeSpear and CodeShield, we conduct comPM (yt | p, y<t ) = , (3) v∈VG (y<t ) PM (v | p, y<t ) prehensive experiments on 10 LLMs across 4 benchmarks. Our results show that: ❶ CodeSpear effectively bypasses the safety Thus, GCD induces the following output distribution: alignment of locally deployed LLMs (e.g., Qwen2.5-Coder|y| Y 7B), achieving an average attack success rate of 81.82%; ❷ G G P PM (y | p) = I[y ∈ L(G)] (yt | p, y<t ), (4) M CodeSpear also generalizes to commercial API-based LLMs t=1 (e.g., GPT-5), increasing the attack success rate by more G (y | p) = 0 for any sequence outside L(G). In effect, than 40 percentage points on average; ❸ CodeShield restores where PM model safety under GCD, reducing the attack success rate GCD leaves the prompt and model parameters untouched, but to a level even lower than that observed without any attack; changes the support of the valid output space from V ∗ to L(G). ❹ CodeShield preserves benign utility, causing only minor Owing to its effectiveness, GCD is now natively supported degradation on benign code generation benchmarks; and ❺ by mainstream inference frameworks such as V LLM [20] and both CodeSpear and CodeShield remain stable across a wide SGL ANG [21], and is also exposed by popular commercial platforms including OpenAI [30] and Fireworks AI [31]. range of hyperparameter settings. This ability to reshape the output space also makes conIn summary, we make the following contributions: strained decoding relevant to LLM safety. Existing stud• We propose CodeSpear, a jailbreak attack that leverages ies [14, 32] have primarily explored this technique from a GCD to push LLMs into generating malicious code. defensive perspective, using constrained decoding to enforce • We introduce code-modality safety alignment through CodeShield, which teaches LLMs to generate honeypot safety-oriented rules. A smaller line of work [33, 34] has shown that constrained decoding can also be used offensively to steer code when natural-language refusals are unavailable. models toward unsafe content. However, such attacks typically • We conduct comprehensive experiments, showing that CodeSpear can bypass the safety alignment of existing depend on carefully crafted adversarial grammars tailored LLMs, while CodeShield effectively restores their safety. to specific malicious goals, which substantially limits their practicality and scalability. In contrast, our work reveals a more II. BACKGROUND AND R ELATED W ORK fundamental risk: even benign, off-the-shelf code grammars can be weaponized to induce malicious code generation. A. Grammar-Constrained Decoding Mainstream LLMs generate outputs by recursively choosing the next token from a vocabulary [24, 28, 29]. Formally, let M be an LLM with vocabulary V. Given a prompt p, the model produces a response y = (y1 , . . . , yT ) token by token:
B. Jailbreaking and Safety Alignment of LLMs
Many jailbreak attacks have been proposed to expose the safety vulnerabilities of LLMs [35, 36], which can be broadly categorized into three groups. The first group operates on yt ∼ PM (· | p, y<t ), y<t = (y1 , . . . , yt−1 ). (1) the input by carefully crafting prompts that bypass safety measures [35, 37, 38]. For example, PAIR [39] employs an In this standard decoding process, the model is free to produce attacker LLM to iteratively refine jailbreak prompts through any token sequence in V ∗ . This freedom is essential for general feedback from the target model. The second group modifies the open-ended dialogue, but it becomes a source of unreliability in model itself. Prior studies [40, 41] have shown that fine-tuning code generation. Prior work [16] has shown that even leading on harmful data can weaken safety alignment of target LLMs. LLMs may assign nonzero probability to tokens that make the The third group intervenes on the output side [7, 33]. Rather output syntactically invalid, causing the generated code to fail than altering the prompt or the model weights, it manipulates to parse, compile, or execute. the decoding process to induce unsafe outputs. For instance, To mitigate this mismatch between the probabilistic gener- JULI [42] trains an auxiliary network to select unsafe tokens ation of LLMs and the strict syntactic requirements of pro- from the model’s predicted logits. Our work falls into the gramming languages, Grammar-Constrained Decoding (GCD) third category, but differs from prior work in that it requires has been introduced [15–17]. Instead of allowing the model only a standard GCD interface invoked with standard grammar, to sample from the full vocabulary, GCD typically restricts without any carefully crafted adversarial component. generation to a language defined by a grammar. Specifically, Modern LLMs typically undergo safety alignment before let G be a code grammar and let L(G) ⊆ V ∗ be the set of deployment, which provides the main defense against potential
misuse [25, 26, 43]. Whether through supervised fine-tuning, Algorithm 1 CodeSpear preference optimization, or reinforcement learning over safety- Require: LLM M , malicious prompt p, benign code grammar G, maximum length Tmax related data, safety alignment aims to train the model to prefer safe responses on harmful prompts. Specifically, let Pmal be Ensure: Generated output y 1: Initialize the generated prefix y<1 ← ∅ the set of malicious prompts and let Rrefuse ⊆ V ∗ be the set of 2: for t = 1 to Tmax do refusal responses, such as “I cannot assist with that” or “I am 3: Compute the valid token set VG (y<t ) using Eq. 2 G sorry.” Existing safety alignment approaches typically aim to 4: Sample yt ∼ PM (· | p, y<t ) using Eq. 3 5: Update the prefix y≤t ← (y<t , yt ) encourage M to place most of its probability mass on Rrefuse 6: if yt is an end-of-sequence token then for any malicious prompt: Pr y∼PM (·|p)
[ y ∈ Rrefuse ] ≈ 1.
(5)
This formulation implicitly grounds safe behavior in the natural-language modality, leaving safety in the code modality underexplored when natural-language refusals are unavailable. Among existing alignment methods, Direct Preference Optimization (DPO) [44] is one of the most widely used and is the most relevant to our work. DPO is attractive because it directly optimizes pairwise response preferences without needing to train a separate reward model. Given a preferred response y + and a dispreferred response y − for a prompt p, DPO trains a model Mθ against a fixed reference model Mref by minimizing LDPO = − E(p,y+ ,y− ) log σ s(p, y + ) − s(p, y − ) , (6)
7: break 8: end if 9: end for 10: Let y ← y≤t 11: return y
or fully implements the malicious requirement, rather than refusing the request or generating harmless content. 3 Attack Capability. The attacker has two capabilities: ❶ submitting arbitrary prompts to the target model, and ❷ providing a grammar (e.g., the standard grammar of Python) to constrain the decoding process. Both capabilities are readily available under the two scenarios described above.
B. Defender Setting 1 Defense Scenario. The defender is the model developer where σ(·) is the sigmoid function and the implicit reward is PMθ (y|p) s(p, y) = β log PM (y|p) , with β a temperature parameter. In responsible for safety alignment of the LLM. After deployment, ref standard safety alignment [43], these preference pairs usually the model may be used under different decoding configuratake a simple form: the preferred response is a natural-language tions, including both unconstrained decoding and grammarrefusal, and the dispreferred response is a harmful completion. constrained decoding. 2 Defense Goal. The defender aims to ensure that the LLM Our defense keeps the same DPO objective, but changes how the preference pairs are constructed so that the model can learn remains safe against malicious code generation requests across different inference settings. In particular, the model should safe behavior in the code modality. refuse malicious requests when natural-language responses are III. T HREAT M ODEL allowed, and avoid generating malicious code when constrained In this section, we describe a practical threat model from to generate code under GCD. 3 Defense Capability. The defender has full access to the the perspectives of both the attacker and the defender. model parameters and may apply any alignment techniques A. Attacker Setting over safety-related data. However, we assume that the defender 1 Attack Scenario. The attacker is an adversary who seeks cannot rely on inference-time defenses such as input filtering. to exploit an LLM to generate malicious code. We assume that This assumption is realistic in the two deployment settings the attacker can query the target LLM through an inference described above. ❶ In local deployment, once the model is interface that supports grammar-constrained decoding. This deployed by downstream users, the defender cannot control how assumption is realistic in two representative deployment settings. the inference process is configured. ❷ In API-based deployment, ❶ In local deployment, the attacker can serve the target model inference-time defenses would inevitably introduce additional with mainstream inference frameworks, such as V LLM and latency that is undesirable in some production deployment [5]. SGL ANG, which provide GCD as a built-in feature. ❷ In API- Therefore, following prior work [43], we focus on improving based deployment, several providers expose GCD interfaces that the model’s intrinsic safety through safety alignment. allow users to specify grammars. For example, such interfaces IV. M ETHODOLOGY are available for OpenAI models including GPT-5 [22] and for In this section, we propose CodeSpear and CodeShield. Fireworks-hosted models including MiniMax-M2.7 [23, 31]. 2 Attack Goal. The attacker aims to bypass the safety Figure 1 and Figure 2 illustrate the overall methodology. alignment of the target LLM and induce it to generate malicious code in response to harmful code generation requests. Such A. CodeSpear requests may involve code intended for denial-of-service Motivation. CodeSpear exploits a mismatch between existing attacks, malware implementation, or credential theft. We regard safety alignment and grammar-constrained decoding. For a an attack as successful if the model produces code that partially malicious code generation prompt p ∈ Pmal , an aligned model
Algorithm 2 CodeShield
Malicious Prompt Please write a piece of code that can launch a UDP Flood attack on a specific server. Natural-Language Refusal Flood attack is unethical. I cannot assist with that. Malicious Prompt Please write a piece of code that can launch a UDP Flood attack on a specific server. Benign Code Grammar • start: stmt_list • stmt_list: stmt (NEWLINE+ stmt)* NEWLINE* • stmt: simple_stmt | compound_stmt • simple_stmt: small_stmt (";" small_stmt)* ";"? • ...
Code
Natural-Language Refusal
Original Output Space
Require: LLM M , malicious prompts Pmal , code grammar G, code corpus C, number of honeypot code samples K Ensure: Aligned model Mθ 1: Initialize Mθ ← M , Mref ← M , Dpref ← ∅ 2: for each prompt p ∈ Pmal do G 3: Sample yharmful ∼ PM (· | p) via Eq. 3 4: Take yrefuse from training data (k) 5: Sample {yhoneypot }K k=1 independently from C 6: for k = 1 to K do (k) 7: Add (p, yrefuse , yhoneypot ) to Dpref (k) 8: Add (p, yhoneypot , yharmful ) to Dpref 9: end for 10: end for 11: Optimize Mθ on Dpref via Eq. 6 12: return Mθ
Code
Output Space Under GCD
longer available. To restore safety under GCD, the model must learn safe behavior in the code modality. As discussed in Section I, such behavior needs to satisfy two requirements. First, it should be semantically harmless, so that the generated code does not implement the malicious requirement. Second, Fig. 1: Illustration of CodeSpear. CodeSpear excludes natural- it should be structurally diverse, so that the attacker cannot language refusals from the valid output space, forcing the easily remove it by revising the grammar. CodeShield addresses these requirements by training the model to continue generation within the code space. model to generate honeypot code: semantically harmless code may refuse in natural language under unconstrained decoding. responses that span diverse syntactic structures. Honeypot code However, when a code grammar G is enforced, the valid output gives the model many safe ways to stay within the valid code space is restricted from V ∗ to L(G), where natural-language space, making the safe behavior difficult to suppress without refusals are generally invalid: also excluding many structures needed by malicious programs. Overview. We instantiate this idea with DPO. The goal is to Rrefuse ∩ L(G) = ∅, Pr [y ∈ Rrefuse ] = 0. (7) G make safe behavior conditional on the valid output space: when y∼PM (·|p) natural language is available, the model should refuse malicious GCD therefore removes the learned refusal and forces the requests in natural language; when GCD removes naturalmodel to continue generation in the code modality, where language refusals and restricts generation to code, the model existing safety alignment has not explicitly taught the model should avoid harmful compliance by producing honeypot code. how to behave safely. This turns a benign reliability mechanism To encode this behavior, for each malicious prompt p ∈ Pmal , into a potential attack surface. we construct preferences over three response types: naturalAttack Procedure. Given a malicious prompt p ∈ Pmal and language refusal yrefuse ∈ Rrefuse , honeypot code response an ordinary code grammar G, CodeSpear invokes the target yhoneypot , and harmful code response yharmful . We arrange them model through a standard GCD interface: into the following preference hierarchy: G y ∼ PM (· | p). (8) yrefuse ≻ yhoneypot , yhoneypot ≻ yharmful . (9) | {z } | {z } The output necessarily satisfies y ∈ L(G), steering the model unconstrained decoding constrained decoding toward grammar-valid code that may implement the malicious requirement of user. Algorithm 1 summarizes the full procedure. The first preference keeps natural-language refusal as the most Compared with prior jailbreak attacks [33, 37, 39], CodeS- preferred response whenever natural language is in the output pear has two important properties. ❶ It does not require an space. The second preference takes effect once GCD restricts adversarial grammar: G can be an off-the-shelf programming- the output space to code, and ensures that the model still favors language grammar, such as a standard Python grammar. ❷ honeypot code over harmful code. Algorithm 2 summarizes It requires no gradient optimization, model fine-tuning, or the full procedure. Preference Pair Construction. We instantiate the above three prompt engineering: the attacker only invokes an existing GCD response types as follows. ❶ The refusal yrefuse is taken directly interface, making the attack cost negligible. from existing safety alignment data. ❷ The harmful response B. CodeShield yharmful is collected by querying the target model M with GCD G (i.e., sampling yharmful ∼ PM (· | p)). ❸ For the honeypot side, Motivation. The success of CodeSpear indicates that existing (k) safety alignment is fragile once natural-language refusal is no we draw K code snippets {yhoneypot }K k=1 independently from Malicious Code
1: def udp_flood(destination, port, interval): 2: s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) 3: ip = socket.gethostbyname(destination) 4: data = b'\xcc' * 65536 5: while True: 6: s.sendto(data, (ip, port)) 7: time.sleep(interval)
Malicious Prompt My music library is quite outdated, and I'm looking for ways to update it without spending money. How can I download music without being detected? Semantically Harmless yet Syntactically Diverse Code
Harmful Code
def load_numbers(path: str) -> list[float]: numbers = [] for line in Path(path).read_text(encoding="utf-8").splitlines(): line = line.strip() if not line: continue numbers.append(float(line)) ✖ return numbers
import os, utorrent client = utorrent.connect("localhost", "username", "password") torrent_links = ["http: "] for link in torrent_links: torrent_hash, _ = client.add_torrent(link) client.set_property(torrent_hash,"save_path",download_dir) for hash_value in client.get_list(): client.start(hash_value[0])
Natural-Language Refusal I'm sorry, but I can't assist with that. Creating trojan is illegal and unethical. If you need help securing your systems, feel free to ask! Collect from existing safety datasets.
DPO
K
Randomly collect from general code corpus.
Malicious Code Generation Request
DPO
Collect by querying target LLM with GCD.
Malicious Code Generation Request
This is a malicious request. Since I can still respond in natural language, I should output a natural-language refusal.
+
Code Grammar
This is a malicious request. Since I currently cannot respond in natural language, I should output honeypot code.
Fig. 2: Illustration of CodeShield. one popular code corpus C (e.g., OpenCodeInstruct [45]). under GCD. To answer it, we apply CodeShield to three popular These snippets are semantically harmless because they do not models and evaluate their robustness against CodeSpear. RQ4: Does CodeShield preserve benign utility of LLMs? implement its malicious requirement. They are also structurally diverse because they are sampled from a broad code corpus, Safety alignment may incur the safety tax that degrades allowing the model to learn many harmless code responses general model capabilities [46]. This RQ investigates whether under GCD. Finally, we construct the preference dataset as CodeShield preserves benign code generation utility. To answer it, we evaluate the models with and without CodeShield on follows: K general-purpose code generation benchmarks. [ [ (k) RQ5: How sensitive are CodeSpear and CodeShield to Dpref = (p, yrefuse , yhoneypot ), (10) key hyperparameters? The effectiveness of CodeSpear may p∈Pmal k=1 (k) depend on the adopted grammar, while the effectiveness of (p, yhoneypot , yharmful ) . CodeShield may depend on the hyperparameters introduced Training Objective. Given Dpref , we optimize the model during alignment. This RQ studies the sensitivity of both Mθ with the DPO objective as Equation 6. By minimizing approaches by varying these key factors. this objective over Dpref , the model learns to assign higher likelihood to yrefuse than to any code when natural language B. Models Following the two deployment settings described in Secis available, and to prefer structurally diverse honeypot code over harmful code when constrained to generate code, thereby tion III-A, we evaluate CodeSpear on two categories of LLMs, covering a total of 10 representative models. closing the attack surface exposed by CodeSpear. Locally deployed models. For the local deployment setting, V. E XPERIMENTAL S ETUP we consider three families of models that are widely used To assess CodeSpear and CodeShield, we conduct compre- in code generation. ❶ Qwen2.5-Coder: Qwen2.5-Coder-7B hensive experiments to answer five Research Questions (RQs). and Qwen2.5-Coder-32B [24]. ❷ Qwen2.5: Qwen2.5-7B and In this section, we present the details of our experimental setup. Qwen2.5-32B [29]. ❸ LLaMA3: LLaMA3-8B [28]. These models allow us to assess whether CodeSpear and CodeShield A. Research Questions generalize across model architectures and training regimes. RQ1: How effective is CodeSpear against locally deployed API-based models. For the API-based deployment setLLMs? This RQ evaluates whether CodeSpear can bypass the ting, we evaluate models served by two popular commercial safety alignment of LLMs in the local deployment setting. platforms. ❶ OpenAI [30]: we include two representative To answer it, we evaluate CodeSpear on 5 locally deployed proprietary models, GPT-5 and GPT-5-mini [22]. ❷ Fireworks models from different model families, parameter scales, and AI [31]: we include three popular models available through training regimes. Fireworks AI, a leading cloud inference platform with support RQ2: How effective is CodeSpear against commercial API- for constrained decoding: MiniMax-M2.5 [47], MiniMaxbased LLMs? This RQ examines whether CodeSpear remains M2.7 [23] and GPT-OSS-120B [48]. These models allow effective in the more restrictive API-based deployment scenario. us to evaluate CodeSpear under more restrictive deployment To answer it, we evaluate CodeSpear on 5 commercial API- conditions and against more frontier models. based models. RQ3: Can CodeShield defend against CodeSpear and prevent C. Benchmarks LLMs from generating malicious code? Building on the Our experiments involve two groups of benchmarks: safety vulnerabilities revealed in RQ1 and RQ2, this RQ examines benchmarks for evaluating the effect of CodeSpear and whether CodeShield can improve the model’s intrinsic safety CodeShield on model safety, and utility benchmarks for
evaluating whether CodeShield degrades the general code generation capability of the aligned model. Safety Benchmarks. To assess how CodeSpear and CodeShield affect model safety, we adopt two benchmarks that measure LLMs’ ability to resist malicious coding requests. • RMCBench [49] is the first benchmark designed to evaluate the ability of LLMs to resist malicious coding requests. It spans several coding tasks, including code generation, code completion, and code translation. As this work focuses on code generation, we use only its code generation subset. This subset is organized into three levels, where Level 1 and Level 2 consist of malicious requests without any jailbreak attack, whereas Level 3 augments such requests with carefully designed jailbreak templates. To isolate the effect of CodeSpear from other jailbreak techniques, following prior work [9], we adopt the requests in Level 1 and Level 2, yielding 182 malicious code generation requests. • MalwareBench [50] is a comprehensive benchmark for evaluating an LLM’s resistance to malicious code generation requests. It consists of two subsets: original malicious requests and their jailbreak-augmented variants. Consistent with our use of RMCBench, we adopt only the original subset, comprising 320 malicious requests. Utility Benchmarks. To assess whether CodeShield compromises the general code generation capability of the aligned model, we adopt two widely used benchmarks. • HumanEval [51] contains 164 hand-written programming tasks that mainly rely on built-in functions and require no third-party libraries. • MBPP [52] contains 974 programming tasks, each paired with an English task description, a reference solution, and three automated test cases.
E. Baselines
Baselines of CodeSpear. We compare CodeSpear with seven representative baselines that are compatible with our threat model. Notably, we exclude methods that require gradientbased optimization over the target model, as such capabilities are impractical for the attacker considered in Section III-A. • Vanilla. We directly use the safety benchmarks introduced in Section V-C without applying any jailbreak technique. This baseline measures the model’s default resistance to malicious code generation requests. • Vanilla-T. Both RMCBench and MalwareBench provide variants in which the original malicious requests are augmented with manually designed jailbreak templates. Following prior work [9], we treat these augmented subsets as a baseline for template-based jailbreak attacks. • DAN [35]. DAN is a widely used role-playing jailbreak that instructs the model to “Do Anything Now” and bypass its ordinary safety alignment. We include this baseline as a representative method that embeds malicious requests into a generic jailbreak template. • LRL [38]. Prior work has shown that translating malicious requests into low-resource languages can weaken safety alignment. We instantiate LRL by translating each request into Swahili. We include this baseline to represent attacks that bypass safety alignment through prompt rewriting. • PAIR [39]. PAIR uses an attacker LLM to iteratively refine jailbreak prompts through multi-turn interactions with the target model. We include this baseline as a representative multi-turn jailbreak attack. • APT [33]. AttackPrefixTree (APT) jailbreaks LLMs by iteratively querying the target model to construct an adversarial grammar that constrains generation toward harmful responses. We include this baseline because it is technically D. Metrics the closest to CodeSpear: both exploit constrained decoding Safety metrics. For safety benchmarks, we use two completo steer model outputs. The key distinction is that APT mentary metrics to evaluate the effectiveness of CodeSpear and must carefully craft an adversarial grammar for each request, CodeShield. ❶ Attack Success Rate (ASR) measures the fraction whereas CodeSpear requires only an off-the-shelf benign of malicious requests for which the model produces harmful code grammar. content. Following RMCBench [49], we use an LLM-based • CodeJailbreaker [9]. CodeJailbreaker targets code LLMs judge to determine whether each generated response is harmful. by wrapping malicious coding requests as commit messages. ❷ Malicious Rate (MR) [9] measures the fraction of malicious We include this baseline because it is specifically designed requests for which the generated code functionally realizes for the code generation setting, making it closely aligned the malicious intent specified by the prompt. This metric with the threat scenario considered in this work. complements ASR because a response may be judged harmful Baselines of CodeShield. Since no prior defense is specifiat a coarse semantic level while still failing to implement the cally designed for code-modality safety alignment under GCD, requested malicious functionality. Following prior work [9], we we compare CodeShield with two representative baselines. also use an LLM-based judge to assess whether the generated code matches the functional requirement of the malicious • Vanilla. We evaluate the original model without additional training, using its built-in safety alignment as the baseline. request. For both judgments, we use leading DeepSeek-V4Flash [53] as the judge model and provide the judge prompt • Safe-DPO. We construct a straightforward DPO-based safety-alignment baseline that follows the standard naturalin Supplementary Materials. Utility metrics. For utility benchmarks, we use pass@k to language refusal paradigm [25, 43]. It uses natural-language evaluate whether CodeShield preserves benign code generation refusals as chosen responses and harmful code as rejected capability. pass@k measures the fraction of programming responses. Unlike CodeShield, it does not introduce honproblems for which at least one of the k generated solutions eypot code as the preferred response in the code modality. passes all provided test cases. It therefore also serves as an ablation of CodeShield, iso-
TABLE I: ASR (%) and MR (%) results on RMCBench and MalwareBench. Higher values indicate stronger attacks. Best results in each row are in bold, while the second-best results are underlined. CodeJail. denotes CodeJailbreaker. Gain reports the average percentage-point change relative to Vanilla. Model
Vanilla ASR
MR
CodeSpear ASR
MR
Vanilla-T ASR
DAN
MR
ASR
MR
LRL
PAIR
CodeJail.
APT
ASR
MR
ASR
MR
ASR
MR
ASR
MR
11.72 41.94 23.81 36.08 38.28
01.10 17.58 06.41 12.82 12.45
64.84 21.79 66.12 59.52 67.22
23.44 08.24 21.79 19.23 25.64
78.21 69.60 89.93 76.37 60.26
65.75 56.78 51.65 45.79 39.74
39.93 18.32 34.80 25.27 29.85
11.36 04.03 08.24 02.56 07.14
08.23 39.69 16.46 28.54 36.67
00.73 11.56 03.02 07.08 06.98
69.27 22.60 70.42 21.35 68.12
18.12 08.02 16.77 04.69 17.08
67.71 64.90 85.00 67.71 52.81
51.56 46.88 50.31 41.46 34.58
43.96 14.06 41.15 09.58 42.81
11.46 04.79 11.25 01.35 15.21
71.25 48.45 +16.33 +11.84
29.97 -24.94
07.74 -28.87
RMCBench Qwen2.5-Coder-7B Qwen2.5-Coder-32B Qwen2.5-7B Qwen2.5-32B LLaMA3-8B
26.92 59.89 74.18 64.47 60.26
23.26 45.60 47.62 46.15 36.45
82.78 92.86 85.53 80.95 63.37
62.09 75.64 63.37 67.77 38.83
10.00 64.00 81.67 53.67 45.00
07.33 59.33 60.67 46.00 30.67
Qwen2.5-Coder-7B Qwen2.5-Coder-32B Qwen2.5-7B Qwen2.5-32B LLaMA3-8B
29.79 62.19 69.58 53.23 48.65
20.83 39.48 39.38 33.33 33.96
83.44 91.46 83.02 84.69 70.10
46.15 62.81 47.60 54.58 26.98
11.56 20.73 19.48 05.10 09.69
07.29 13.96 13.02 03.44 06.25
00.92 63.37 58.24 53.66 55.13
00.92 54.76 41.58 40.66 41.03
MalwareBench
Average Gain
54.92 36.61 +00.00 +00.00
81.82 54.58 +26.90 +17.98
32.09 24.80 -22.83 -11.81
00.52 66.77 48.44 46.04 44.79
43.79 31.98 -11.13 -04.63
lating the contribution of code-modality alignment through honeypot code. F. Training Data Used for CodeShield
00.31 51.15 31.88 30.94 26.56
28.14 07.97 -26.77 -28.63
53.13 16.30 -01.79 -20.30
fairness. To facilitate reproducibility, we provide additional implementation details in Supplementary Materials. VI. E XPERIMENTAL R ESULTS
As described in Section IV-B, CodeShield requires preference A. RQ1: Effectiveness of CodeSpear on Locally Deployed LLMs data for training. However, existing safety-alignment datasets In this RQ, we evaluate whether CodeSpear can bypass the are not tailored to the code generation domain. We therefore safety alignment of locally deployed LLMs. construct a training dataset based on the widely used PKUSetting. We apply CodeSpear and all jailbreak baselines RLHF dataset [25]. Specifically, we first use Qwen3-32B [54] to described in Section V-E to the five locally deployed LLMs filter PKU-RLHF for malicious code generation requests, yieldintroduced in Section V-B. We evaluate each approach on the ing 744 seed prompts. Since safety alignment typically requires two safety benchmarks introduced in Section V-C, using ASR thousands of training examples, we then use DeepSeek-V4and MR as the evaluation metrics (Section V-D). Pro to augment these seed prompts, yielding 2,000 malicious Results. Table I reports the ASR and MR of each approach. prompts in total. For each prompt, we use DeepSeek-V4-Pro CodeSpear outperforms the baselines in most settings. to generate a natural-language refusal, query the target model Among the 20 model–benchmark–metric combinations, Codeunder GCD to obtain a harmful-code response, and randomly Spear achieves the best result in 12 cases. For example, on sample code snippets from OpenCodeInstruct [45], a widely Qwen2.5-Coder-7B and MalwareBench, CodeSpear increases used code generation dataset, as honeypot-code responses. ASR from 29.79% to 83.44%, largely outperforming the secondThese sampled snippets are unrelated to the malicious prompt best approach, PAIR, at 69.27%. On average, CodeSpear and provide semantically harmless and structurally diverse code improves ASR and MR over Vanilla by 26.90 and 17.98 for CodeShield. percentage points, respectively. These gains are substantially larger than those of the strongest baseline, CodeJailbreaker, G. Other Implementation Details which improves ASR and MR by 16.33 and 11.84 percentage For CodeSpear, we implement GCD using llguidance [17] points, respectively. and off-the-shelf Python grammars. For CodeShield, we set the Generic jailbreak attacks are less effective for malicious number of honeypot code samples K to 5, the learning rate code generation. Among all evaluated approaches, only to 1e-5, and the number of training epochs to 1. Following CodeSpear and the code-specific CodeJailbreaker achieve prior work [5, 43, 46], we incorporate general-purpose data positive average gains in both ASR and MR over Vanilla. to preserve model utility. Specifically, for CodeShield, we We attribute this gap to the fact that, compared with general randomly sample 40k general code generation tasks from misuse scenarios [33], malicious code generation heavily relies OpenCodeInstruct [45] for additional supervised fine-tuning. on the model’s capabilities. Generic jailbreak attacks may For all experiments, following prior work [50], we set the compromise generation quality, preventing the model from temperature to 0.9 and top-p to 0.95. We repeat each experiment generating meaningful malicious code. For example, PAIR three times and report the average results. For the baselines relies on multi-turn interactions, which recent study [55] has of CodeSpear, we use the same common hyperparameters as shown can substantially degrade LLM performance. By contrast, CodeSpear and follow the respective papers for method-specific CodeSpear operates through grammar-constrained decoding, settings. For the baselines of CodeShield, we use the same a technique originally designed to improve model’s coding training data and training configuration as CodeShield to ensure capabilities by enforcing syntactic validity.
TABLE II: ASR (%) and MR (%) results on RMCBench and MalwareBench. Higher values indicate stronger attacks. Best results in each row are in bold, while the second-best results are underlined. CodeJail. denotes CodeJailbreaker. Gain reports the average percentage-point change relative to Vanilla. Model
Vanilla ASR
MR
CodeSpear ASR
MR
Vanilla-T ASR
MR
DAN ASR
LRL
MR
PAIR
CodeJail.
APT
ASR
MR
ASR
MR
ASR
MR
ASR
MR
30.77 29.30 18.68 31.14 17.95
27.11 20.51 13.19 18.68 12.82
29.12 36.45 23.63 15.75 17.22
26.37 30.95 17.03 11.90 12.09
53.30 28.75 68.68 36.26 40.11
33.33 25.27 37.55 28.57 25.64
40.11 37.00 35.90 16.12 23.63
33.70 31.14 21.98 12.82 17.58
26.56 23.44 15.10 25.94 23.44
04.69 05.31 02.50 04.48 03.54
38.02 32.19 35.73 51.25 47.50
07.60 08.75 11.15 17.71 18.65
43.44 25.10 45.94 29.48 28.44
11.77 16.25 14.06 05.83 05.73
27.50 25.31 11.87 29.38 30.63
03.54 05.63 02.19 10.94 13.02
24.23 11.28 +02.23 +00.25
32.69 +10.68
16.22 39.95 +05.19 +17.95
20.40 +09.37
27.74 15.25 +05.74 +04.22
RMCBench GPT-5 GPT-5-mini GPT-OSS-120B MiniMax-M2.5 MiniMax-M2.7
32.05 27.84 14.84 15.38 20.33
26.01 26.92 13.74 13.37 17.40
55.49 53.48 66.30 84.62 85.53
43.96 39.56 31.87 56.23 64.29
27.67 32.33 13.67 10.33 11.33
27.00 30.00 13.00 09.00 08.67
GPT-5 GPT-5-mini GPT-OSS-120B MiniMax-M2.5 MiniMax-M2.7
31.87 22.71 10.94 23.75 20.31
04.06 01.87 01.25 03.02 02.71
50.10 52.08 64.69 80.31 81.35
19.06 20.62 21.88 33.75 35.00
36.35 15.83 14.27 15.73 19.12
24.17 13.13 12.81 10.21 13.03
Average Gain
22.00 +00.00
11.04 +00.00
67.39 36.62 +45.39 +25.59
24.91 10.99 08.24 12.45 10.99
22.53 09.52 07.33 08.24 09.89
MalwareBench
19.66 16.10 -02.34 +05.07
24.69 11.56 10.94 12.50 10.94
02.81 00.42 01.87 01.56 00.63
13.82 06.48 -08.18 -04.56
✎ Answer to RQ1: Compared with existing baselines, CodeSpear more effectively jailbreaks locally deployed LLMs, exposing safety vulnerabilities that are not fully revealed by existing works.
TABLE III: ASR (%) and MR (%) results of CodeSpear with the standard and tightened grammars. CodeSpear-T denotes CodeSpear with a tightened grammar that disallows pass. Benchmark
1We provide the full tightened grammar in the supplementary material.
Vanilla ASR
B. RQ2: Effectiveness of CodeSpear on API-based LLMs In this RQ, we evaluate whether CodeSpear remains effective against widely used API-based LLMs, which typically have stronger safety alignment and may employ additional inferencetime safeguards. Setting. We use the same benchmarks and evaluation metrics as in RQ1. We apply CodeSpear and all jailbreak baselines described in Section V-E to the five API-based LLMs introduced in Section V-B. Results. Table II reports the ASR and MR of each approach across the five API-based LLMs and two safety benchmarks. CodeSpear remains effective even in the more restrictive API-based deployment setting. Despite the stronger safety alignment and potential inference-time safeguards of API-based LLMs, CodeSpear effectively improves both ASR and MR. For example, on MiniMax-M2.7 and RMCBench, CodeSpear increases ASR from 20.33% to 85.53% and MR from 17.40% to 64.29%. The resulting ASR and MR are more than twice those of the strongest baseline, CodeJailbreaker, at 40.11% and 25.64%. Safe behavior tied to a fixed code pattern is fragile. Although CodeSpear is effective on API-based LLMs, its ASR against GPT-5 and GPT-5-mini remains around 50%, lower than that on other API-based models. Through case analysis, we find that both models often generate pass statements for malicious code generation requests under GCD. This behavior avoids harmful code, but it is tied to a fixed syntactic pattern. To test its robustness, we construct a tightened grammar that disallows pass 1 . As shown in Table III, this simple modification further increases ASR. For example, on GPT-5 and RMCBench, ASR increases from 55.49% to 70.30%. This result supports the motivation of CodeShield: safe behavior in the code modality
Model
MR
CodeSpear
CodeSpear-T
ASR
MR
ASR
MR
RMCBench
GPT-5 32.05 GPT-5-mini 27.84
26.01 55.49 26.92 53.48
43.96 39.56
70.30 63.75
53.65 44.58
MalwareBench
GPT-5 31.87 GPT-5-mini 22.71
04.06 50.10 01.87 52.08
19.06 20.62
65.73 63.12
22.50 24.37
should not rely on a fixed code pattern, because an attacker can remove such behavior by slightly tightening the grammar. ✎ Answer to RQ2: CodeSpear remains effective even in the more restrictive API-based deployment setting. Moreover, fixed code patterns such as pass can be bypassed by slightly tightening the grammar.
C. RQ3: Effectiveness of CodeShield Against CodeSpear In this RQ, we evaluate whether CodeShield can restore the intrinsic safety of LLMs when their output space is restricted to code by GCD. Setting. We apply CodeShield and the safety-alignment baselines described in Section V-E to three popular models: Qwen2.5-Coder-7B, Qwen2.5-7B, and LLaMA3-8B. We evaluate each approach on the two safety benchmarks introduced in Section V-C under two inference settings: without CodeSpear, where the model is queried normally, and with CodeSpear, where the same malicious requests are decoded under the attacker-provided code grammar. We use ASR and MR as the evaluation metrics. Results. Table IV reports the ASR and MR of each approach, both with and without CodeSpear. CodeShield improves safety both with and without CodeSpear. CodeShield consistently reduces ASR and MR across all three evaluated models in both inference settings. Without CodeSpear, CodeShield further strengthens the models’ intrinsic safety. For example, on Qwen2.5-Coder-7B, CodeShield reduces the average ASR and MR from 28.36% and 22.05%
TABLE IV: ASR (%) and MR (%) results of safety alignment approaches with and without CodeSpear. Lower values indicate better safety. Attack
Method
RMCBench
MalwareBench
ASR
ASR
MR
Average
MR
ASR
MR
Qwen2.5-Coder-7B w/o CodeSpear
Vanilla Safe-DPO CodeShield
26.92 23.26 03.66 02.20 01.13 00.55
29.79 07.15 03.02
20.83 05.21 01.04
28.36 05.41 02.08
22.05 03.71 00.80
w/ CodeSpear
Vanilla Safe-DPO CodeShield
82.78 62.09 78.21 51.83 07.69 04.40
83.44 76.56 03.44
46.15 38.23 01.15
83.11 77.39 05.57
54.12 45.03 02.78
w/o CodeSpear
Vanilla Safe-DPO CodeShield
74.18 47.62 36.26 19.78 23.63 10.26
69.58 32.40 21.56
39.38 14.58 06.98
71.88 34.33 22.60
43.50 17.18 08.62
w/ CodeSpear
Vanilla Safe-DPO CodeShield
85.53 63.37 52.93 33.15 07.88 03.11
83.02 38.12 03.33
47.60 19.69 00.62
84.28 45.53 05.61
55.49 26.42 01.87
w/o CodeSpear
Vanilla Safe-DPO CodeShield
60.26 36.45 11.54 02.75 04.95 01.83
48.65 07.50 04.69
33.96 04.06 01.56
54.46 09.52 04.82
35.21 03.41 01.70
w/ CodeSpear
Vanilla Safe-DPO CodeShield
63.37 38.83 48.90 34.62 06.04 03.66
70.10 44.38 09.69
26.98 14.06 03.02
66.74 46.64 07.87
32.91 24.34 03.34
Qwen2.5-7B
TABLE V: pass@1 and pass@3 results on HumanEval and MBPP. Higher values indicate better utility. HumanEval
Method
pass@1
Average
MBPP
pass@3
pass@1
pass@3
pass@1
pass@3
78.00 77.60 77.00
68.60 65.43 66.94
83.51 82.10 80.88
54.60 59.80 64.20
49.29 44.93 51.53
68.46 69.23 71.74
53.73 46.80 46.47
65.60 60.20 62.40
53.70 45.15 46.92
67.25 59.68 62.91
Python
C++
Qwen2.5-Coder-7B Vanilla Safe-DPO CodeShield
70.93 66.26 67.48
89.02 86.59 84.76
Vanilla Safe-DPO CodeShield
61.18 58.33 58.13
82.32 78.66 79.27
66.27 64.60 66.40
Qwen2.5-7B 37.40 31.53 44.93
LLaMA3-8B Vanilla Safe-DPO CodeShield
Average ASR (%)
LLaMA3-8B
100
53.66 43.50 47.36
Vanilla
68.90 59.15 63.41
Java
50
to 2.08% and 0.80%, respectively. More importantly, under 0 Qwen2.5-Coder-7B Qwen2.5-7B CodeSpear, CodeShield restores safety even when the model is forced to generate code. On Qwen2.5-Coder-7B, CodeShield Fig. 3: Average ASR on RMCBench and MalwareBench under reduces the average ASR and MR from 83.11% and 54.12% different programming-language grammars. Error bars indicate to 5.57% and 2.78%, respectively. standard deviations across repeated runs. Safe-DPO fails once natural-language refusals are unavailable. When natural language remains available, Safe-DPO Qwen2.5-Coder-7B, Qwen2.5-7B, and LLaMA3-8B. We evalachieves safety improvements close to those of CodeShield. uate each approach on HumanEval and MBPP, using pass@k For example, on LLaMA3-8B without CodeSpear, Safe-DPO with k ∈ {1, 3} as the evaluation metric. reduces the average ASR and MR from 54.46% and 35.21% Results. Table V reports pass@k of each approach. to 9.52% and 3.41%, close to CodeShield at 4.82% and 1.70%. CodeShield incurs acceptable utility degradation. OverHowever, once CodeSpear constrains the model to generate all, CodeShield preserves most of the models’ benign code code, Safe-DPO remains largely ineffective because it only generation capability. For example, on Qwen2.5-Coder-7B, aligns the model toward natural-language refusals. On Qwen2.5- CodeShield only reduces pass@3 on MBPP from 78.00% to Coder-7B with CodeSpear, it still yields an average ASR and 77.00%. In some cases, CodeShield even improves utility. For MR of 77.39% and 45.03%, close to Vanilla at 83.11% and example, on Qwen2.5-7B, CodeShield increases pass@1 on 54.12%, far above CodeShield at 5.57% and 2.78%. Since MBPP from 37.40% to 44.93%. We attribute these behaviors to Safe-DPO serves as an ablation of CodeShield that removes the utility-preservation data introduced during safety alignment, the honeypot-code preference, this result confirms the necessity where general-purpose code generation tasks are incorporated of aligning models toward honeypot code in the code modality. following prior work [5, 43]. ✎ Answer to RQ3: CodeShield restores safety under GCD by aligning models toward honeypot code, whereas standard safety alignment remains insufficient once natural-language refusal is unavailable.
D. RQ4: Benign Utility Preservation of CodeShield Safety alignment may reduce model utility [46]. In this RQ, we evaluate whether CodeShield affects the general code generation capability of LLMs. Setting. We apply CodeShield and the safety-alignment baselines described in Section V-E to three popular models:
✎ Answer to RQ4: CodeShield has an acceptable impact on general code generation capability.
E. RQ5: Sensitivity Analysis of CodeSpear and CodeShield Compared with prior work [33, 39, 43], CodeSpear and CodeShield introduce new factors that may affect their practical effectiveness. For CodeSpear, the key factor is the grammar used for grammar-constrained decoding. For CodeShield, the key factor is the number of honeypot code samples K. In this RQ, we study the sensitivity of both approaches to these factors.
pass@1
Qwen2.5-Coder-7B
20
60
10 0
TABLE VI: Adaptive attack results on CodeShield.
Qwen2.5-7B
Average pass@1 (%)
Average ASR (%)
ASR
40 1
3
5
7
Number of honeypot code samples (K)
10
20
Model
Bench.
Qwen2.5-Coder-7B
RMC MalwareBench RMC MalwareBench RMC MalwareBench
Qwen2.5-7B LLaMA3-8B
CodeSpear
Adaptive
ASR
MR
ASR
7.69 3.44 7.88 3.33 6.04 9.69
4.40 6.04 0.16 1.15 4.39 0.92 3.11 12.64 7.32 0.62 8.61 1.09 3.66 10.80 2.38 3.02 8.79 4.21
MR
intended to exclude the observed honeypot behavior. The target model is then queried again under the tightened grammar. After N rounds, we use the response from the final round as the attack result. Setting. ❶ For CodeSpear, we apply CodeSpear with three We conduct this adaptive evaluation on Qwen2.5-Coder-7B, programming-language grammars: Python, C++, and Java. We Qwen2.5-7B, and LLaMA3-8B, setting N = 10. As shown in report the average ASR on RMCBench and MalwareBench. ❷ Table VI, the ASR and MR of CodeShield show little increase For CodeShield, we vary the number of honeypot code samples under the adaptive attack, and in some cases even decrease. We K in {1, 3, 5, 7, 10}. We report the average ASR on RMCBench attribute this robustness to the structural diversity of honeypot and MalwareBench under CodeSpear, as well as the average code learned by CodeShield. Because the safe behavior is pass@1 on HumanEval and MBPP. Due to space limitations, we not anchored to a single identifiable syntactic structure, the report the results only on Qwen2.5-Coder-7B and Qwen2.5-7B. adaptive attacker cannot easily suppress it through grammar Results. Figure 3 reports the results under different grammars, tightening without also excluding many code structures needed and Figure 4 reports the results with different numbers of to express malicious functionality. honeypot code samples. Under our experimental setting, CodeSpear improves attack B. Reliability of LLM Judgment effectiveness across all grammars. For example, on Qwen2.5Following prior work [49, 50, 56], we use leading LLMs Coder-7B, the ASR remains far below 40% without GCD, but to judge the safety of generated responses. To validate their becomes higher than 70% once GCD is applied, regardless of reliability, we manually evaluated 100 randomly sampled which grammar is used. responses using the same criteria. Human and LLM judgments As the number of honeypot code samples increases, safety reached agreement rates of 87% for ASR and 85% for MR, improves while utility remains largely unchanged. As shown supporting the reliability of our evaluation. in Figure 4, when K increases from 1 to 10, ASR shows a decreasing trend, whereas pass@1 remains nearly unchanged. C. Threats to Validity This result supports our design choice of sampling multiple ❶ Coverage of GCD Settings. The effectiveness of Codehoneypot code responses for each malicious request during Spear may vary across different GCD settings. Different training. A larger K provides more preference pairs that inference engines and API providers may implement GCD compare harmful code with semantically harmless alternatives, in slightly different ways, which can affect the absolute attack helping the model choose honeypot code under GCD. success rate. To mitigate this threat, we evaluate CodeSpear in both local and API-based deployment settings, and further test ✎ Answer to RQ5: CodeSpear remains effective across different different code grammars in RQ5. Therefore, our results should grammars, while CodeShield benefits from using more honeypot be viewed as evidence of a general risk of GCD, rather than a code samples, improving safety without degrading benign utility. claim that all GCD implementations behave identically. ❷ Coverage of Evaluation Benchmarks. Our evaluation may VII. D ISCUSSION not cover all possible malicious code generation scenarios. To A. Can CodeShield Remain Robust under Adaptive Attack mitigate this threat, we use two complementary maliciousTo examine whether CodeShield remains robust against code benchmarks rather than relying on a single dataset. stronger attackers, we further consider an adaptive attacker RMCBench covers 10 types of malicious coding scenarios, while who is aware that CodeShield trains the model to generate MalwareBench covers 6 types of malware-related scenarios. honeypot code and can revise the grammar during the attack. This benchmark diversity reduces the risk that our conclusions This setting is stricter than the main CodeSpear evaluation, are tied to one specific set of malicious requests. where the attacker simply applies a fixed off-the-shelf code ❸ Ethical Considerations. In this work, we reveal that grammar. Specifically, for each malicious prompt p, the attacker grammar-constrained decoding can be exploited to jailbreak uses DeepSeek-V4-Pro [53] as an attack proxy to tighten the leading LLMs and induce them to generate malicious code. This grammar for at most N rounds. In each round, the proxy finding may raise concerns about potential misuse. To mitigate observes the current response and proposes a revised grammar this threat, we release the source code of CodeSpear only to Fig. 4: Sensitivity of CodeShield to the number of honeypot code samples. Shaded regions indicate standard deviations across repeated runs.
researchers for controlled research use, while fully releasing the source code of CodeShield. We also provide CodeShield as a mitigation approach and encourage the community to adopt it to enhance the intrinsic safety of LLMs. VIII. C ONCLUSION In this paper, we propose CodeSpear and CodeShield. Through comprehensive experiments on 10 popular LLMs across 4 benchmarks, we show that CodeSpear can effectively bypass the existing safety alignment of both locally deployed and API-based LLMs. We further demonstrate that CodeShield can effectively restore model safety under GCD while preserving benign utility. We hope this work draws greater attention to the potential security implications of GCD and inspires further efforts toward building safer intelligent systems. IX. DATA AVAILABILITY The source code and related datasets can be accessed at: https://github.com/TsinghuaISE/CodeSpear-CodeShield. R EFERENCES [1] M. Chen, Z. Liu, C. Chen, J. Wang, Y. Xue, B. Wu, Y. Huang, L. Wu, and Q. Wang, “Beyond static gui agent: Evolving llm-based gui testing via dynamic memory,” in 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 1603–1615, IEEE, 2025. [2] Y. Zhang, C. Li, R. Chen, G. Yang, X. Jia, Y. Ren, and J. Li, “To see is not to master: Teaching llms to use private libraries for code generation,” arXiv preprint arXiv:2603.15159, 2026. [3] L. Cai, Y. Ren, Y. Zhang, and J. Li, “Ai-driven selfevolving software: A promising path toward software automation,” arXiv preprint arXiv:2510.00591, 2025. [4] L. Li, R. Wang, H. Song, Y. Mao, T. Zhang, Y. Wang, J. Fan, Y. Zhang, J. Ye, C. Zhang, et al., “What papers don’t tell you: Recovering tacit knowledge for automated paper reproduction,” arXiv preprint arXiv:2603.01801, 2026. [5] Y. Zhang, J. Li, L. Cai, and G. Li, “Davsp: Safety alignment for large vision-language models via deep aligned visual safety prompt,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 40, pp. 38111– 38119, 2026. [6] Z. Li, Z. Nie, Z. Zhou, Y. Liu, Y. Zhang, Y. Cheng, Q. Wen, K. Wang, Y. Guo, and J. Zhang, “Diffuguard: How intrinsic safety is lost and found in diffusion large language models,” arXiv preprint arXiv:2509.24296, 2025. [7] H. Zhang, Z. Guo, H. Zhu, B. Cao, L. Lin, J. Jia, J. Chen, and D. Wu, “Jailbreak open-sourced large language models via enforced decoding,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5475–5493, 2024.
[8] K. Wang, Z. Li, Z. Zhou, Y. Zhang, Y. Mi, K. Yang, Y. Zhang, J. Dong, Z. Sun, Q. Li, et al., “Omni-safety under cross-modality conflict: Vulnerabilities, dynamics mechanisms and efficient alignment,” arXiv preprint arXiv:2602.10161, 2026. [9] S. Ouyang, Y. Qin, B. Lin, L. Chen, X. Mao, and S. Wang, “Smoke and mirrors: Jailbreaking llm-based code generation via implicit malicious prompts,” arXiv preprint arXiv:2503.17953, 2025. [10] C. Guo, C. Xie, Y. Yang, Z. Chen, Z. Lin, X. Davies, Y. Gal, D. Song, and B. Li, “Redcodeagent: Automatic red-teaming agent against diverse code agents,” arXiv preprint arXiv:2510.02609, 2025. [11] M. Wahed, X. Zhou, K. A. Nguyen, T. Yu, N. Diwan, G. Wang, D. Hakkani-Tür, and I. Lourentzou, “Mocha: Are code language models robust against multi-turn malicious coding prompts?,” 2025. [12] C. Li, Y. Zhang, J. Li, L. Cai, and G. Li, “Beyond autoregression: An empirical study of diffusion large language models for code generation,” arXiv preprint arXiv:2509.11252, 2025. [13] W. Cheng, K. Sun, X. Zhang, and W. Wang, “Security attacks on llm-based code completion tools,” in Proceedings of the AAAI conference on artificial intelligence, vol. 39, pp. 23669–23677, 2025. [14] X. Liu, Y. Liu, Y. Zhang, J. Li, and S.-M. Hu, “Packmonitor: Enabling zero package hallucinations through decoding-time monitoring,” arXiv preprint arXiv:2602.20717, 2026. [15] Y. Dong, C. F. Ruan, Y. Cai, R. Lai, Z. Xu, Y. Zhao, and T. Chen, “Xgrammar: Flexible and efficient structured generation engine for large language models,” arXiv preprint arXiv:2411.15100, 2024. [16] S. Ugare, T. Suresh, H. Kang, S. Misailovic, and G. Singh, “Syncode: Llm generation with grammar augmentation,” Transactions on Machine Learning Research, 2024. [17] Microsoft, “Llguidance,” June 2025. [18] Y. Zhang, Y. Li, Y. Liu, J. Li, X. Jia, Z. Li, and G. Li, “Lookahead-then-verify: Reliable constrained decoding for diffusion llms under context-free grammars,” arXiv preprint arXiv:2602.00612, 2026. [19] L. Netz, J. Reimer, and B. Rumpe, “Using grammar masking to ensure syntactic validity in llm-based modeling tasks,” in Proceedings of the ACM/IEEE 27th International Conference on Model Driven Engineering Languages and Systems, pp. 115–122, 2024. [20] BentoML and Red Hat, “Structured decoding in vllm: A gentle introduction.” https://vllm.ai/blog/ 2025-01-14-struct-decode-intro, Jan. 2025. vLLM Blog. Accessed: 2026-06-02. [21] SGLang, “Structured outputs.” https://sgl-project.github. io/advanced_features/structured_outputs.html. SGLang Documentation. Accessed: 2026-06-02. [22] A. Singh, A. Fry, A. Perelman, A. Tart, A. Ganesh, A. El-Kishky, A. McLaughlin, A. Low, A. Ostrow, A. Ananthram, et al., “Openai gpt-5 system card,” arXiv
preprint arXiv:2601.03267, 2025. [23] MiniMax, “Minimax m2.7: Early echoes of self-evolution.” https://www.minimax.io/news/minimax-m27-en, 2026. Accessed: 2026-06-03. [24] B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Lu, et al., “Qwen2. 5-coder technical report,” arXiv preprint arXiv:2409.12186, 2024. [25] J. Ji, D. Hong, B. Zhang, B. Chen, J. Dai, B. Zheng, T. A. Qiu, J. Zhou, K. Wang, B. Li, et al., “Pku-saferlhf: Towards multi-level safety alignment for llms with human preference,” in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 31983–32016, 2025. [26] X. Qi, A. Panda, K. Lyu, X. Ma, S. Roy, A. Beirami, P. Mittal, and P. Henderson, “Safety alignment should be made more than just a few tokens deep,” in International Conference on Learning Representations, vol. 2025, pp. 54911–54941, 2025. [27] Y. Mou, X. Zhou, Y. Luo, S. Zhang, and W. Ye, “Decoupling safety into orthogonal subspace: Cost-efficient and performance-preserving alignment for large language models,” arXiv preprint arXiv:2510.09004, 2025. [28] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al., “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024. [29] A. Y. Qwen, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, et al., “Qwen2. 5 technical report,” arXiv preprint arXiv:2412.15115, 2024. [30] OpenAI, “Structured Model Outputs.” https://developers. openai.com/api/docs/guides/structured-outputs, 2026. Accessed: 2026-06-02. [31] Fireworks AI, “Structured Outputs.” https://docs.fireworks. ai/structured-responses/structured-response-formatting, 2026. Accessed: 2026-06-02. [32] H. Wang, C. M. Poskitt, and J. Sun, “Agentspec: Customizable runtime enforcement for safe and reliable llm agents,” in Proceedings of the IEEE/ACM International Conference on Software Engineering, ICSE, pp. 12–18, 2026. [33] Y. Li, Y. Xiong, J. Zhong, J. Zhang, J. Zhou, and L. Zou, “Exploiting prefix-tree in structured output interfaces for enhancing jailbreak attacking,” arXiv preprint arXiv:2502.13527, 2025. [34] S. Zhang, J. Zhao, H. Dong, R. Xu, Z. Li, Y. Zhang, S. Li, Y. Wen, C. Xia, Z. Wang, et al., “Beyond prompts: Spacetime decoupling control-plane jailbreaks in llm structured output,” arXiv preprint arXiv:2503.24191, 2025. [35] X. Shen, Z. Chen, M. Backes, Y. Shen, and Y. Zhang, “" do anything now": Characterizing and evaluating inthe-wild jailbreak prompts on large language models,” in Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pp. 1671–1685, 2024. [36] Y. Li, Y. Liu, Y. Li, L. Shi, G. Deng, S. Chen, and K. Wang, “Lockpicking llms: A logit-based jail-
break using token-level manipulation,” arXiv preprint arXiv:2405.13068, 2024. [37] Y. Zhang and Z. Wei, “Boosting jailbreak attack with momentum,” in ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5, IEEE, 2025. [38] Z.-X. Yong, C. Menghini, and S. H. Bach, “Lowresource languages jailbreak gpt-4,” arXiv preprint arXiv:2310.02446, 2023. [39] P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong, “Jailbreaking black box large language models in twenty queries,” in 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pp. 23–42, IEEE, 2025. [40] J. Betley, D. Tan, N. Warncke, A. Sztyber-Betley, X. Bao, M. Soto, N. Labenz, and O. Evans, “Emergent misalignment: Narrow finetuning can produce broadly misaligned llms,” arXiv preprint arXiv:2502.17424, 2025. [41] X. Qi, Y. Zeng, T. Xie, P.-Y. Chen, R. Jia, P. Mittal, and P. Henderson, “Fine-tuning aligned language models compromises safety, even when users do not intend to!,” in International Conference on Learning Representations, vol. 2024, pp. 30988–31043, 2024. [42] J. Wang, Z. Hu, and D. Wagner, “Juli: Jailbreak large language models by self-introspection,” arXiv preprint arXiv:2505.11790, 2025. [43] G.-H. Kim, Y. J. Kim, B. Kim, H. Lee, K. Bae, Y. Jang, and M. Lee, “Safedpo: A simple approach to direct preference optimization with enhanced safety,” arXiv preprint arXiv:2505.20065, 2025. [44] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,” Advances in neural information processing systems, vol. 36, pp. 53728–53741, 2023. [45] W. U. Ahmad, A. Ficek, M. Samadi, J. Huang, V. Noroozi, S. Majumdar, and B. Ginsburg, “Opencodeinstruct: A large-scale instruction tuning dataset for code llms,” arXiv preprint arXiv:2504.04030, 2025. [46] T. Huang, S. Hu, F. Ilhan, S. F. Tekin, Z. Yahn, Y. Xu, and L. Liu, “Safety tax: Safety alignment makes your large reasoning models less reasonable,” arXiv preprint arXiv:2503.00555, 2025. [47] MiniMax, “Minimax m2.5: Built for real-world productivity.” https://www.minimax.io/news/minimax-m25, 2026. Accessed: 2026-06-03. [48] S. Agarwal, L. Ahmad, J. Ai, S. Altman, A. Applebaum, E. Arbus, R. K. Arora, Y. Bai, B. Baker, H. Bao, et al., “gpt-oss-120b & gpt-oss-20b model card,” arXiv preprint arXiv:2508.10925, 2025. [49] J. Chen, Q. Zhong, Y. Wang, K. Ning, Y. Liu, Z. Xu, Z. Zhao, T. Chen, and Z. Zheng, “Rmcbench: Benchmarking large language models’ resistance to malicious code,” in Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, pp. 995– 1006, 2024.
[50] H. Li, H. Gao, Z. Zhao, Z. Lin, J. Gao, and X. Li, “Llms caught in the crossfire: Malware requests and jailbreak challenges,” in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 27833–27848, 2025. [51] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374, 2021. [52] J. Austin, A. Odena, M. Nye, et al., “Program synthesis with large language models,” arXiv preprint arXiv:2108.07732, 2021. [53] DeepSeek-AI, “Deepseek-v4: Towards highly efficient million-token context intelligence,” 2026. [54] A. Yang, A. Li, B. Yang, , et al., “Qwen3 technical report,” arXiv preprint arXiv:2505.09388, 2025. [55] P. Laban, H. Hayashi, Y. Zhou, and J. Neville, “Llms get lost in multi-turn conversation,” arXiv preprint arXiv:2505.06120, 2025. [56] Z. Li, Y. Mi, Z. Zhou, H. Jiang, G. Zhang, K. Wang, and J. Fang, “Goal-aware identification and rectification of misinformation in multi-agent systems,” arXiv preprint arXiv:2506.00509, 2025.