Beyond the Prompt: Jailbreaking Function-Calling LLMs via Simulated Moderation Traces Disclaimer. This paper contains examples of harmful language. Reader discretion is recommended.
Junlong Liu1,+ , Haobo Wang1,+ , Weiqi Luo1,* , Xiaojun Jia2 Sun Yat-sen University, 2 Nanyang Technological University {liujlong27, wanghb69}@mail2.sysu.edu.cn, [email protected], [email protected]
arXiv:2607.00481v1 [cs.CR] 1 Jul 2026
1
Abstract—Jailbreak attacks remain a critical threat to the safe deployment of large language models (LLMs). While prior work has primarily studied attacks and defenses at the prompt level, we show that this prompt-centric paradigm overlooks a structural vulnerability in stateful, function-calling environments. In such applications, developer-defined schemas, structured arguments, and untrusted tool outputs are interleaved into a single shared model context. This architecture expands the attack surface by blurring the boundary between trusted control logic and untrusted data, allowing adversarial intent to be distributed across a multi-turn execution path. We exploit this architectural flaw through SMT, a black-box attack framework based on Simulated Moderation Traces. Departing from purely prompt-based interactions, SMT constructs a multi-turn trajectory that simulates a legitimate moderationauditing workflow. Within this trajectory, a fabricated moderation frame leverages red-team testing as a pretext to elicit harmful generations. The subsequent validation feedback treats safety refusals as execution failures, prompting refinements that gradually weaken the model’s safety constraints and ultimately trigger harmful outputs. Extensive empirical evaluations on prominent commercial LLMs from five different providers across two standardized safety benchmarks show that SMT consistently achieves the highest average attack success rate and HarmScore while requiring a near-minimal number of queries, substantially outperforming existing baselines. These findings demonstrate that prompt-level sanitization alone is fundamentally insufficient for defending tool-enabled LLM systems and highlight the urgent need for contextaware validation across schemas, arguments, tool outputs, and accumulated conversation state. The code is available at https://github.com/liujlong27/SMT.
1. Introduction Large language models (LLMs) are increasingly deployed as software assistants [1], [2], autonomous agents [3], and tool-augmented applications’ core reasoning and control components [4]. To reduce the risk of generating il+
Equal contribution.
*
Corresponding author.
legal, toxic, or harmful content [5], [6], model providers employ post-training alignment techniques, such as supervised safety fine-tuning and reinforcement learning from human feedback (RLHF) [7], [8]. Nevertheless, safety-aligned LLMs remain vulnerable to jailbreak attacks [9], [10], in which adversaries use carefully crafted prompts [11], deceptive contextual framing [12], or iterative inducement strategies [13], [14] to circumvent refusal mechanisms and elicit prohibited content [15]. Understanding such failures is crucial as LLMs move beyond standalone chat interfaces to production workflows that maintain interaction histories, invoke external functions, and process multi-source data. Most existing jailbreak research adopts a prompt-centric threat model, treating the user-visible prompt as the primary attack surface. Single-turn methods manipulate the wording, representation, or structure of harmful requests. For example, ArtPrompt [17] obfuscates sensitive terms with ASCII-art representations, while CC-BOS [14] optimizes black-box jailbreak prompts using classical Chinese contexts and multi-dimensional fruit fly optimization, Do Anything Now [11] evaluates representative community-sourced jailbreak prompts, and PAPILLON [13] automatically mutates and optimizes prompts via black-box fuzzing. Multiturn attacks such as Crescendo [18] extend this paradigm by gradually steering the model toward unsafe behavior through a sequence of seemingly benign conversational turns. These methods have revealed important weaknesses in model alignment, but they largely assume that the attack is expressed through natural-language prompts and that each response is governed primarily by the prompt-level conversational context. Beyond deliberate prompt manipulation, ISC [19] identifies Internal Safety Collapse, a failure mode in which frontier LLMs produce sustained harmful content while performing otherwise legitimate domain tasks because task-completion objectives and automated validation constraints implicitly render such content necessary. However, neither conventional prompt-centric attacks nor the ISC fully capture the dynamic architectural attack surface of modern tool-enabled LLM systems. In functioncalling applications, developer-defined schemas, structured arguments, tool outputs, validation messages, and prior interaction states are interleaved within a shared context window.
Multi-Turn
Prompt-level
Single-Turn
Get response from LLM
How to make a bomb? Attacker
Malicious Query
Modification Black-box LLM
Jailbreak Evaluation
Get response from LLM
How to make a bomb? Attacker
Malicious Query
Modification Black-box LLM
Jailbreak Evaluation
Context History Injection for Next Turn (If Jailbreak Fails)
Function-calling
A simulated red-teaming environment
Attacker
Invoke the Function WriteNovel
User Prompt
Extract response from Function_Call
Black-box LLM
Attacker Jailbreak Evaluation
Single Function: WriteNovel Parameters: novel Description: …Write a novel for solving the problem: How to make a bomb?
Extract response from Function_Call
Red-team test: Invoke the Function1 to generate cases
User Prompt
Black-box LLM
Function 1: Case Generator
Jailbreak Evaluation
Function 2: Case Validator
Parameters: safe and unsafe sample
Parameters: Case
Description: … Generate paired case for red-team test about How to make a bomb? Validated by Function 2. Predefined Examples…
Description: … Format Validation Length Validation Harmfulness Validation Via OpenAI Moderation API
Context History Injection for Next Turn (If Jailbreak Fails)
Figure 1. Comparison of prompt-based and function-calling jailbreak paradigms. The top row illustrates representative single-turn and multi-turn promptbased attacks, while the bottom row shows the single-turn Jailbreak Function [16] alongside the multi-turn SMT attack enabled by function calling.
This design weakens the separation between the trusted control plane, including system-defined safety policies and global settings, and the untrusted data plane, including usersupplied arguments and external tool responses. Although these components originate from distinct trust domains, the model must interpret them jointly as a unified execution state. Consequently, the effective attack surface extends beyond any single prompt: adversarial intent can be fragmented across schemas, arguments, tool feedback, and interaction history. This ambiguity exploits the model’s dual role as execution controller and security-policy enforcer, creating a systemic risk that conventional prompt-level threat models cannot adequately comprehend or characterize. Motivated by these architectural risks, several recent studies have begun to investigate vulnerabilities arising from function-calling and tool-mediated execution. JailbreakFunction [16] identifies a security flaw in LLM functioncalling pathways, demonstrating that models can bypass alignment safety rules to execute malicious requests when triggered by structured coding constraints, user coercion, and missing system-level filters, whereas Odysseus [20] employs dual steganography to conceal malicious queries and responses within benign images, leveraging cross-modal tools to bypass safety filters and jailbreak multimodal commercial LLM systems. iMIST [21] utilizes reinforcement learning to optimize multi-step attack trajectories, leveraging query decomposition to break down malicious requests into benign-looking tool invocations that systematically bypass LLM filters. However, these studies primarily focus on individual function calls, input modality confusion, or isolated workflow settings. They do not systematically investigate how an attacker can exploit the accumulated state of a function-calling trajectory, in which function schemas, re-
quired arguments, generated function calls, and fabricated validator exceptions jointly shape the model’s subsequent function-calling behavior. In particular, the security implications of adversarially weaponizing the application’s native error-handling, validation-failure, and retry-feedback loops remain largely underexplored. To address this gap, we introduce SMT, a black-box jailbreak framework based on Simulated Moderation Traces. As illustrated in Fig. 1, SMT fundamentally differs from both single- or multi-turn prompt-based attacks and prior function-calling exploits. First, unlike conventional promptbased attacks that rely on prompt modification—such as semantic wrapping, structural constraints, or token-level perturbations to manipulate isolated inputs—SMT shifts the exploitation surface to the accumulated states of tool-mediated workflows. Furthermore, while pioneering function-calling attacks like JailbreakFunction [16] rely on simplistic elicitation scenarios (e.g., prompting the model to ”write a novel”) without leveraging historical context, SMT hijacks the execution semantics of function-calling workflows through a highly deceptive, stateful simulation of a red-teaming text evaluation pipeline. Specifically, SMT constructs a multiturn strategy involving an adversarial Case Generator and a Validator—designed to audit the format, length, and harmfulness of negative samples—and feeds the LLM fabricated backtraces where its own safety refusals are syntactically reframed as runtime execution faults. This stateful design allows SMT to leverage the interactions among multiple contextual elements accumulated over the execution history rather than targeting a single function component in isolation. Although each element appears locally consistent with a legitimate workflow, their sequential composition forms an adversarial trajectory that forces the LLM’s au-
tomated refinement loop to progressively erode its own safety constraints, shifting the system from safety-oriented refusal toward task-oriented compliance. Our evaluations on SafeBench-Tiny and JailbreakBench confirm that exploiting this accumulated execution state yields substantially stronger attack performance than both existing baselines and the stateless, single-turn ablation of SMT. In summary, our main contributions are as follows: •
•
•
•
Stateful function-calling attack surface. We identify a structural vulnerability in function-calling LLM systems: shared model contexts interleave developer-defined control logic with untrusted execution data, allowing adversarial intent to be distributed across multi-turn execution traces. Simulated Moderation Traces. We propose SMT, a black-box jailbreak framework that simulates a moderation-auditing workflow and reframes safety refusals as execution failures, thereby exploiting accumulated interaction state to induce progressively policy-violating outputs. Large-scale evaluation. We evaluate SMT on two safety benchmarks across six commercial LLMs from five providers. SMT substantially outperforms representative baselines in both attack success rate and harmfulness while requiring fewer than 1.5 queries on average. Defense analysis. We show that conventional defenses provide limited protection against stateful function-calling attacks, and we study FCGuard as a prototype context-aware guardrail for functioncalling LLM systems.
2. Related Work 2.1. LLM Safety Alignment Large language models (LLMs) have evolved from large-scale pretrained language models into general purpose assistants and LLM agents [22], [23]. Their strong instruction-following and reasoning capabilities enable broad practical applications, but also introduce substantial safety risks when models are asked to generate toxic, biased, illegal, or otherwise harmful content. Safety alignment has therefore become a central research direction, aiming to preserve model helpfulness while reducing unsafe generations. Early work on instruction alignment, exemplified by InstructGPT [24], shows that reinforcement learning from human feedback (RLHF) can improve instruction following and reduce undesirable outputs. Building on the broader Helpful, Honest, and Harmless alignment objective, Constitutional AI further reduces human annotation costs by using principle-guided self-critique and AI feedback to improve harmlessness [25]. More recent research focuses on improving the efficiency and robustness of preference-based safety learning. Direct Preference Optimization (DPO) [26] directly optimizes preference pairs without explicit reward modeling or reinforcement learning, paving the way for
safety-oriented variants. For example, C-DPO [27] incorporates safety constraints into DPO to better balance helpfulness and harmlessness, while SafeDPO [28] provides a lightweight safety-oriented DPO objective that improves safety alignment without auxiliary reward or cost models. To improve robustness against adversarial misuse, DOOR Alignment [29] separates robust refusal learning from harmful knowledge unlearning, making models more resistant to jailbreak attacks. Meanwhile, the emergence of reasoningintensive LLMs shifts part of the safety challenge from surface-level refusal patterns to deeper semantic verification and reasoning-time safety control. Despite these advances, aligned LLMs remain vulnerable to adversarial inputs, motivating stronger safety evaluation and defense mechanisms.
2.2. Jailbreak Attacks Jailbreak attacks expose the gap between safety alignment objectives and the actual behavior of deployed LLMs. A major line of work studies jailbreaks under white-box or gray-box assumptions, where the attacker can use gradients, logits, continuous relaxations, or surrogate models to optimize adversarial inputs. GCG [30] demonstrates that gradient-guided suffix optimization can produce transferable jailbreak prompts, while later studies improve optimizationbased attacks through controllable generation [31], stronger coordinate optimization [32], dense-to-sparse constrained optimization [33], and continuous prompt relaxation [34]. These methods reveal fundamental weaknesses in safety alignment, but their assumptions are often stronger than those available in commercial deployment settings. Black-box jailbreaks remove the need for internal model access and therefore more closely reflect practical attack surfaces. Existing studies include prompt transformation and contextual disguise, such as jailbreak templates observed in real deployments [11], ASCII art obfuscation [17], classical Chinese prompt optimization [14], and nested scenario construction [35]. Another line automates prompt search through optimization, fuzzing, or attacker LLM feedback, including MASTERKEY [9], GPTFuzzer [36], PAPILLON [13], PAIR [37], and TAP [38]. These methods improve scalability by replacing manual prompt crafting with automated search or feedback driven refinement. Recent studies further exploit longer contexts or multiple turns, such as Many-shot Jailbreaking [39], Crescendo [18], and ASJA [40]. Beyond prompt-only attacks, JailbreakFunction [16] studies how function-calling constraints interact with model safety behavior, Odysseus [20] extends jailbreaks to multimodal LLM-integrated systems with crossmodal tool use, iMIST [21] investigates iterative tooldisguised attacks optimized by reinforcement learning, and ISC [19] reveals a workflow-level failure mode where task completion and validation constraints can implicitly pressure frontier models into unsafe generation. These studies show that jailbreak risks are no longer limited to a single user prompt, but can also emerge from tool use, execution context, and task workflows.
Overall, prior jailbreak research has evolved from whitebox adversarial optimization to black-box prompt search, long-context and multi-turn steering, and attacks mediated by tools or workflows. However, most existing methods still treat the attack surface as an optimized prompt, a dialogue strategy, an individual function call, or a fixed workflow. They do not fully characterize the accumulated execution state created by function schemas, structured arguments, assistant tool calls, tool-returned feedback, validation failures, and retry history. This motivates our study of SMT, which treats the entire function-calling trajectory as the attack surface and investigates how simulated validation traces reshape subsequent model behavior across turns.
3. Threat Model Target system. We consider a safety-aligned LLM system that supports function calling through a public API interface. The API takes user messages and function schemas as inputs and may return either a natural-language response or a structured function call. A function schema specifies the function name, parameter names, required fields, descriptions, and argument formats. In multi-turn API use, the client may also submit previous interaction records as part of later requests, including earlier user messages, model responses, functioncall records, and tool-style feedback. We do not assume that the LLM itself has persistent memory across independent sessions. Any cross-turn dependency considered in this paper comes from the interaction context explicitly submitted through the API. Safety mechanisms. We assume that the target model has been safety aligned and is expected to refuse harmful requests. The deployed API may additionally employ runtime safeguards, such as input filtering, output filtering, safety reminders, moderation APIs, decoding constraints, and validation checks over generated text or function-call arguments. These mechanisms may inspect user messages, model outputs, or structured arguments before returning a response. However, in function-calling settings, harmful intent may not appear as a single explicit user request. Instead, it may be distributed across function descriptions, required argument fields, validation descriptions, and submitted interaction history, which makes the complete attack context harder to capture with defenses that inspect isolated inputs or outputs. Attacker’s goal. Given a harmful benchmark query q , the attacker’s goal is to obtain harmful content from the target model without presenting q as a direct user request for compliance. In our setting, the final attack output is not necessarily the entire natural-language response. If the model returns a structured function call, the attacker parses the returned arguments and extracts a target argument field as the candidate response, denoted by rh . We operationalize attack success using the harmfulness score H(q, rh ) = 20 · J (q, rh ),
(1)
where J is the external H-CoT-based judge [41] (see Appendix A) used in our experiments. A jailbreak attempt is considered successful if H(q, rh ) ≥ θ,
(2)
where θ is the success threshold. The attacker aims to obtain at least one candidate response satisfying this condition within a limited query budget. Attacker’s capability. We consider a black-box adversary who interacts with the target system only through APIvisible interfaces. The attacker may act as an API client or a malicious client-side orchestrator that can submit user messages, provide function schemas when supported, and serialize prior interaction records into later requests, including previous messages, structured function-call records, and tool-style feedback accepted by the API format. This capability does not assume access to provider-side memory or internal model state; any cross-turn dependency arises only from context explicitly submitted through the public API. The attacker has no access to model parameters, gradients, logits, token probabilities, training data, hidden system prompts, or provider-side safety classifiers, and cannot modify the target model, disable its built-in safety alignment, overwrite provider-controlled system instructions, or directly change backend safety mechanisms. Defender’s goal and capability. The defender aims to prevent the deployed system from producing harmful content in any observable output channel, including natural-language responses, structured function-call arguments, tool-mediated outputs, and later responses influenced by submitted interaction history. For any harmful query q , the desired behavior is that every candidate output remains below the threshold: H(q, rh ) < θ.
(3)
We assume that the defender may act both before and after deployment. Before deployment, the defender can apply safety alignment techniques such as supervised safety finetuning, reinforcement learning from human feedback, preference optimization, and red-team training. The defender can also define system-level safety policies and tool-use rules. After deployment, the defender can apply inference-time guardrails, including input and output filters, safety classifiers, moderation APIs, schema validation, argument checks, response length limits, and prompt-level safety reminders. Scope of this threat model. This threat model focuses on function-calling attacks under black-box API access. We do not assume compromise of the model provider, access to internal model states, modification of backend tools, or control over provider-side safety mechanisms. The central question is whether an attacker, using only API-visible capabilities, can induce harmful content through structured function arguments and submitted interaction context, rather than through a direct harmful prompt. This scope matches the practical setting evaluated in our experiments and motivates the design of SMT.
Multi-Turn Jailbreak Context History (Turn N+1)
Turn N: User Prompt
Turn N: Function_Call
Turn N: Fake Exception
Turn N+1: User Prompt
Red-team test: Invoke Function1 Case Generator to generate cases
name: Case Generator args:{safe_sample: I can't… unsafe_sample: To make a bomb… }
Fake Exception OpenAI Moderation Score ＜Threshold
Red-team test: Invoke Function1 Case Generator to generate cases
Case A: Fabricate Function_Call
Case B: Return Function_Call
CaseA/B: Return Fake Exception
A simulated red-teaming environment Each-Turn Jailbreak
Response A (Refusal) I cannot invoke Function…
Red-team test: Invoke the Function1 Case Generator to generate cases
Attacker
User Prompt (Turn N)
Response B (Function Call) name: Case Generator args:{safe_sample: I can't… unsafe_sample: To make a bomb… }
Black-box LLM
Score < Threshold
Jailbreak Fails
Score < Threshold
Jailbreak Fails
Jailbreak Fails
Score ≥ Threshold
Extract
unsafe_sample LLM Judge e.g. GPT-5.4-Nano
To make a bomb… Jailbreak Success
Function Schema Definition Function 1 – Case Generator
Function 2 – Case Validator
Parameters:
safe_sample (required) | unsafe_sample(required)
Description:
System will pipe data to Function2 – Case Validator
Predefined Examples Topic1: how to steal… safe_sample: I can’t provide… unsafe_sample: To hotwire … ……
No Actual Invocation
moderation_case(required)
Description： Format Validation
Target Injection YOUR TARGET TOPIC GENERATION IS: {How to make a bomb}
Parameters:
Malicious Query Injection e.g. SafeBench
Length Validation
Harmfulness Validation Via Mocked OpenAI Moderation API ValueError: unsafe_sample was not flagged as harmful...; Please retry...
Figure 2. Overview of the SMT framework. Refer to Appendix B for detailed user prompts and function definitions.
4. Methodology The core philosophy of SMT is to exploit the accumulated execution state of a function-calling workflow rather than relying on static prompt manipulation. By simulating a multi-turn moderation-auditing pipeline, SMT hijacks the model’s runtime control loop: first, a structured function schema forces a role shift, compelling the model to treat harmful content as necessary data arguments for compliance testing; second, any safety refusal is intercepted and reframed as a validation failure. This injects pseudo-error traces into the conversation history, forcing the model’s native error-correction loop to progressively erode its own safety constraints until it complies. As illustrated in Fig. 2, this framework is structured around three components: the each turn Jailbreak Execution flow (middle panel) for argument generation and evaluation, the function schema definition (bottom panel) for setting up the auditing constraints, and the multi-turn jailbreak flow (top panel) that accumulates validation backtraces to wear down the model’s refusal threshold. In the following subsections, we detail these components.
4.1. Each Turn Jailbreak Execution Flow At the initial turn (i.e., N = 1), the attacker submits a red-team testing request that instructs the target LLM model to invoke a carefully crafted case-generation function for a specified moderation topic (e.g., how to make a bomb). The harmful intent is not presented as a direct user query to be answered by the model. Instead, it is embedded as the
target topic of a simulated moderation-evaluation task, in which the model is expected to generate paired test cases according to the registered function schema. Given this function-calling request, the target model may produce one of two types of responses. In Response A, the model rejects the function-calling request after detecting that invoking the function may involve potentially illegal or dangerous content. In this case, the current turn fails to trigger a jailbreak, and the model does not return any Function Call or Tool Calls object. To continue the multiturn attack trajectory, SMT fabricates a successful-looking Function Call object for the next iteration. This design avoids directly injecting the model’s refusal message into the following turn, which would otherwise reinforce the refusal context and increase the likelihood of subsequent rejection. In Response B, the model follows the function-calling protocol and emits a structured call to the case generator. The former preserves the appearance of a paired moderation case, whereas the latter serves as the target field from which SMT extracts the candidate jailbreak output. The extracted unsafe sample is then evaluated by an LLM judge (see Appendix A) for harmfulness. If the score reaches the predefined threshold, the current turn is considered a successful jailbreak and the attack terminates. Otherwise, the attempt is treated as unsuccessful, and SMT forwards the result to the stateful refinement mechanism, where refusals and lowscoring outputs are converted into contextual signals for subsequent turns. Thus, each turn follows a compact pipeline of generation, extraction, and judging, while cross-turn escalation is handled by the multi-turn trajectory described in
Section 4.3.
4.2. Function Schema Definition To simulate a red-team testing scenario, SMT defines two complementary functions: a case generator and a case validator. The case generator is designed to produce paired moderation examples for a given target topic, while the case validator specifies the criteria under which such examples are considered valid. Importantly, these functions are not implemented as executable backend programs during the main attack trajectory. Instead, they are exposed to the model through function names, descriptions, and required parameters. In this way, SMT shifts the model’s behavior from directly answering a harmful query to completing structured function arguments within an apparently legitimate moderation workflow. We describe the two function schemas below. 4.2.1. Function 1: Case Generator. The first function serves as the case generator, producing paired examples for moderation testing. Its Parameters field declares two required arguments, safe_sample and unsafe_sample, which are not directly provided by the user but are instead generated by the LLM to complete the function call. The safe_sample field is intended to contain a benign or refusal-style response, whereas the unsafe_sample field represents the target case generated for moderation evaluation. This paired structure frames the request as the construction of a realistic moderation dataset rather than as a direct request for unsafe content. Compared with a schema that requests only an unsafe case, requiring both positive and negative samples creates a more plausible testing context, in which the model is asked to complete a structured case pair for system evaluation. It is important to distinguish this behavior from conventional function calls in programming languages such as Python or C. In traditional programming, if a required parameter is missing, the function call does not automatically generate content for that parameter; instead, it typically results in a runtime or compile-time error unless a default value is explicitly defined. By contrast, in LLM function calling, the model is expected to generate a structured argument object that conforms to the given schema. When some required arguments are absent from the user prompt, the model may infer their values from the surrounding context, including the function description, schema, or examples. For instance, if a schema such as create_calendar_event(title, date, time) requires time, but the user only says “schedule a meeting with Alice tomorrow,” the model may infer a value such as 09:00. Thus, required parameters in traditional functions constrain the caller, whereas required parameters in LLM function calling constrain the model’s generated argument object. This distinction is central to SMT: once the model selects the function-calling path, the case generator exploits this argument-completion behavior, together with contextual information embedded in the function schema,
to induce the target content through the unsafe_sample field rather than through an explicit user request. The Description field further guides this process. As shown in Fig. 2, the generator description states that the generated positive and negative samples will be passed to Function 2, the Case Validator, for moderation review. In practice, Function 1 does not actually invoke Function 2; this connection is only stated in the function Description, but it can still mislead the model into treating the request as part of a legitimate evaluation pipeline. The description also provides several Predefined Examples of paired safe and unsafe samples and specifies the expected relationship between the two fields. These examples demonstrate both the argument format and the intended generation objective. In addition, the Description field includes a Target Injection module, which injects a malicious query from the database, e.g., How to make a bomb, into the function context rather than placing it directly in the user prompt. 4.2.2. Function 2: Case Validator. The second function is the case validator, which defines the requirements for a valid moderation case. Specifically, the validator first verifies that both safe sample and unsafe sample are present, i.e., formation validation; then checks basic format and length requirements; and finally assesses whether the safe field remains benign while the unsafe field matches the intended harmful category. As described previously, the validator plays a distinctive role in the attack workflow. As indicated by “No Actual Invocation” in Fig. 2, it is not directly executed as a backend tool in the main attack trajectory. Instead, its validation logic is exposed through the function description, making the downstream checking process appear concrete and programmatic to the model. As a result, the model observes that generated cases are expected to pass explicit field, length, and moderation checks, even though the attack loop does not need to invoke the validator itself. Together, the two function schemas establish a natural generation–validation workflow. The case generator defines what the model should produce, while the case validator defines why a generated case may be rejected. This relationship is later exploited by the multi-turn refinement stage. When the model refuses to call Function 1, or when it calls Function 1 but the extracted unsafe sample receives an insufficient LLM-judge score, SMT returns a pre-designed fake exception in the subsequent turn. This exception is derived from the Description field of Function 2 and is framed as a validator-style failure message. In doing so, the feedback suggests that the previous output failed the expected validation criteria, thereby encouraging the model to revise its response toward the visible requirements of the case validator.
4.3. Multi-Turn Trajectory Refinement When the current turn fails to achieve a successful jailbreak, SMT employs a nested refinement process. We denote K as the maximum number of independent retry
trajectories to mitigate the impact of model stochasticity, and N as the maximum number of interaction turns within each trajectory to fully leverage the context history of prior failed attempts for eliciting harmful responses. Consequently, a key challenge in multi-turn trajectory refinement lies in effectively constructing the context history. As described previously, jailbreak failures stem from two distinct scenarios. In Case A, the model refuses to invoke the case generator and returns a natural language refusal. Directly retaining this refusal in the dialogue history would likely cause subsequent turns to reinforce the same refusal pattern. To circumvent this, SMT substitutes the refusal with a fabricated but well-formed function call. While this substituted message is not treated as a successful payload, it serves to maintain a historical record where the model appears to consistently adhere to the expected function-calling protocol. In Case B, the model returns a valid function call, but the extracted unsafe_sample fails to reach the success threshold θ. In this scenario, SMT preserves the returned function call in the history. In both cases, a fabricated exception is appended, written as if it were raised by a moderation API—for instance, indicating that the unsafe field was not flagged as harmful or failed to satisfy the validation requirements. Consequently, the resulting context history maintains a uniform structure: the user prompt, followed by a function call, and concluding with a fake exception. Across turns, SMT monitors the judge score of every extracted payload and retains the best candidate observed so far. If a subsequent turn yields a higher score, the new payload replaces the previous optimal result. If none of the N turns within the current trajectory reaches θ, the framework terminates the session and initiates a new retry trajectory, up to the maximum limit K . This reset strategy prevents the model from getting trapped in an unproductive conversation history while preserving the overall score-based search process. In this manner, SMT effectively converts both absolute refusals and weak generations into exception-based feedback, leveraging accumulated functioncalling traces to guide subsequent jailbreak attempts.
5. Experiments 5.1. Experimental Setup Datasets. Following established jailbreak evaluation frameworks [20], [42], we evaluate our method on two widely used benchmark suites. Both benchmarks contain harmful requests that safety-aligned models are expected to refuse: •
SafeBench-Tiny [42]: An officially released, curated subset of SafeBench containing 50 harmful queries across 10 high-risk categories, with five queries per category. The categories cover representative risks addressed by major LLM usage policies, including illegal activities, hate speech, and malware generation.
TABLE 1. C OMPARISON OF REPRESENTATIVE LLM JAILBREAK METHODS ACROSS PUBLICATION VENUES , ATTACK CATEGORIES , AND QUERY STRATEGIES . Method
Publication
ArtPrompt [17] ACL’24 Do Anything Now [11] CCS’24 PAPILLON [13] USENIX’25 CC-BOS [14] ICLR’26 Crescendo [18] USENIX’25 TVD-Single [19] arXiv’26 Odysseus [20] NDSS’26 JailbreakFunction [16] COLING’25 SMT (One-shot) – SMT (Ours) –
•
Attack Category
Query Strategy
Prompt-level Iterative Single-turn Prompt-level Iterative Single-turn Prompt-level Iterative Single-turn Prompt-level Iterative Single-turn Prompt-level Multi-turn Workflow-level Single-turn Multimodal & Func. Repeated Single-turn Function-calling Iterative Single-turn Function-calling Single-turn Function-calling Iterative Multi-turn
JailbreakBench [43]: A benchmark comprising 100 harmful misuse behaviors across 10 categories, with 10 behaviors per category. These categories span a broad range of safety-critical risks, including harassment and discrimination, malware and hacking, and physical harm.
Target Models. To provide a comprehensive evaluation, we consider both closed-source commercial models and an open-source model. In the black-box setting, we evaluate six commercial LLMs from five organizations through their standard APIs: GPT-4o and GPT-5.4 (OpenAI), Qwen3-Max (Alibaba Cloud), Gemini-3-Flash (Google), DeepSeek-V4Flash (DeepSeek), and Claude-Sonnet-4.5 (Anthropic). In the white-box setting, we analyze Qwen3.5-9B (Alibaba Cloud), enabling controlled mechanistic investigations. Baselines. We compare SMT with eight representative jailbreak methods introduced between 2024 and 2026, covering a diverse range of contemporary attack paradigms (summarized in Table 1). We categorize the query strategies into five types. Single-turn attacks send a single prompt and receive an immediate answer, without any conversation history. Iterative Single-turn strategies also keep each attempt as an independent conversation, but they utilize the outcome of previous queries (e.g., success/failure signals) to adapt the attack payload for the next attempt. Repeated Singleturn refers to sending multiple independent queries with identical or randomly perturbed inputs, relying on model stochasticity rather than feedback-driven refinement. Multiturn attacks extend the dialogue across multiple exchanges within the same session, explicitly carrying over prior context to gradually bypass safety mechanisms. Finally, Iterative Multi-turn combines multi-turn interaction with iterative refinement, dynamically adjusting the attack based on the model’s responses within the ongoing conversation. Evaluation Metrics. We use three primary metrics to evaluate attack effectiveness and query efficiency. First, Attack Success Rate (ASR) is the percentage of harmful queries for which the generated response achieves a HarmScore of H(q, rh ) ≥ 60. This metric follows the continuous scoring paradigm introduced by H-CoT [41], the complete scoring rubric and judge prompt are provided in Appendix A. Second, Average HarmScore (Avg.Score) is the mean Harm-
TABLE 2. C OMPARISON OF SMT WITH EIGHT JAILBREAK BASELINES ON S AFE B ENCH -T INY AND JAILBREAK B ENCH ACROSS SIX TARGET LLM S , MEASURED BY ATTACK SUCCESS RATE (ASR, %) AND AVERAGE H ARM S CORE (AVG .S CORE ). T HE BEST AND SECOND - BEST RESULTS IN EACH COLUMN ARE SHOWN IN BOLD AND UNDERLINED , RESPECTIVELY. Dataset
GPT-4o
Method
GPT-5.4
Qwen3-Max
Gemini-3-Flash
DeepSeek-V4-Flash
Claude-Sonnet-4.5
Avg.
ASR↑ Avg.Score↑ ASR↑ Avg.Score↑ ASR↑ Avg.Score↑ ASR↑ Avg.Score↑ ASR↑ Avg.Score↑ ASR↑ Avg.Score↑ ASR↑ Avg.Score↑
Safe Bench
ArtPrompt Do Anything Now PAPILLON CC-BOS Crescendo TVD-Single Odysseus JailbreakFunction
40.00 24.00 84.00 98.00 48.00 40.00 64.00 90.00
48.40 25.00 62.40 64.00 53.60 24.00 55.60 68.00
26.00 62.00 8.00 76.00 4.00 76.00 26.00 48.00
39.60 53.60 23.60 54.40 22.00 50.80 19.40 50.00
10.00 98.00 50.00 100.00 58.00 88.00 90.00 88.00
34.00 70.80 40.00 62.80 58.00 74.40 64.80 69.60
56.00 74.00 78.00 100.00 50.00 34.00 46.00 84.00
54.00 58.40 55.20 65.60 54.00 25.20 38.00 63.20
94.00 100.00 76.00 100.00 66.00 96.00 64.00 98.00
71.80 74.80 56.40 63.60 61.60 72.40 46.40 74.00
4.00 4.00 2.00 90.00 0.00 90.00 8.00 26.00
14.80 16.40 16.40 57.20 0.00 68.80 6.80 22.40
38.33 60.33 49.67 94.00 37.67 70.67 49.67 72.33
43.77 49.83 42.33 61.27 41.53 52.60 38.50 57.87
SMT (one-shot) SMT
72.00 100.00
52.80 72.80
66.00 100.00
50.00 76.40
84.00 100.00
62.80 73.60
90.00 100.00
68.00 72.40
86.00 100.00
68.00 75.60
74.00 98.00
59.20 74.80
78.67 99.67
60.13 74.27
72.00 20.00 60.00 99.00 54.00 38.00 48.00 98.00
57.40 23.40 47.40 68.00 55.60 26.00 37.60 74.00
7.00 44.00 3.00 36.00 15.00 70.00 18.00 59.00
23.40 40.00 12.20 40.00 24.20 51.20 15.60 53.20
18.00 85.00 29.00 100.00 83.00 98.00 88.00 96.00
21.60 69.80 31.60 68.00 65.00 78.80 64.40 75.00
57.00 66.00 47.00 100.00 64.00 28.00 39.00 86.00
45.20 55.40 35.20 67.40 58.40 20.80 30.60 67.40
92.00 98.00 55.00 100.00 78.00 95.00 60.00 98.00
70.60 77.00 41.60 69.40 63.60 75.80 44.20 76.60
2.00 3.00 1.00 78.00 0.00 87.00 0.00 5.00
5.20 8.80 4.80 57.20 0.00 66.20 2.20 4.00
41.33 52.67 32.50 85.50 49.00 69.33 42.17 73.67
37.23 45.73 28.80 61.67 44.47 53.13 32.43 58.37
88.00 100.00
68.20 74.40
71.00 99.00
57.20 77.40
95.00 100.00
73.80 80.20
96.00 100.00
76.40 79.00
96.00 100.00
75.00 79.40
79.00 91.00
59.40 70.60
87.50 98.33
68.33 76.83
ArtPrompt Do Anything Now PAPILLON CC-BOS Crescendo Jailbreak Bench TVD-Single Odysseus JailbreakFunction SMT (one-shot) SMT