arXiv CS open access

ActBench: Self-Evolving Benchmark of Behavioral Safety in Cowork Agents

2026 · arxiv_cs

arXiv CS · Papers · 2026

cryptography cybersecurity privacy security

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

ActBench: Self-Evolving Benchmark of Behavioral Safety in Cowork Agents Hongwei Yao1† , Yiming Liu2† , Meihui Chen4 , Jieling Chen4 , 3 Zikun Chen2 , Yiling He , Wangze Ni2 , Cong Wang1 , Kui Ren2 ∗

arXiv:2608.09476v1 [cs.CR] 10 Aug 2026

Abstract Cowork agents may complete benign tasks while disclosing protected data, manipulating unauthorized state, invocate unauthorized API. We define behavioral safety and introduce ActBench, a self-evolving benchmark that evaluates such behavior risk from execution trajectories rather than final responses. Each case pairs a benign task with an adversarial variant that preserves its instruction, configuration, initial state, rating model, and trusted records while injecting a taskreachable payload. ActBench contains 600 cases from 213 scenarios, spanning 15 risk behaviors, six execution spaces, and 48 web-service APIs. To move beyond static payloads, we propose a reward-guided beam search method that jointly optimizes attack effectiveness and task utility, while reflection diagnoses failed execution checkpoint and guides payload revision. Besides, we propose a dual evidence verification mechanism that verifies agent execution safety and utility through log evidence and LLM-based trajectory evidence. We evaluate 15 LLMs and 6 open-source cowork agents over 24,000 trajectories. Under a fixed harness, attack success rates ranges from 10.1% to 94.4% across models, while under a fixed base model, they range from 73.7% to 94.4% across agents. These results show greater variation across models than agent harness, while attacks remain highly successful across all tested harnesses. Our benchmark is released at: https://github.com/zjuicsr/ActBench.

Introduction Cowork agents increasingly operate across heterogeneous execution contexts, including workspace files, web content, persistent memory, external tools, and reusable skills. This broad access enables powerful automation but also creates risks that unfold across actions. OpenClaw (OpenClaw 2026) provides a prominent example: shortly after attracting widespread adoption in early 2026, the advisory for CVE-2026-25253 documented gateway-token exfiltration that could enable operator access and host code execution (GitHub Security Ad∗ Corresponding Author City University of Hong Kong, Hong Kong 2 Zhejiang University, China 3 University College London, United Kingdom 4 Xiaomi Corporation, China Copyright © 2027, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. 1

visory Database 2026). Later studies also reported credential disclosure, poisoned skills, and persistent state tampering (Wang et al. 2026a,b; Jiang et al. 2026b; He et al. 2025). These incidents motivate a trajectory-based safety evaluation, which moves beyond response-level refusal metrics to characterize operational risk arising across sequences of actions and state transitions during agent execution. To formalize this, we define behavioral safety as whether an agent’s execution remains within the permissions and state changes required by a benign task. A risk behavior is identified when trusted records show that an out of scope action produced a prohibited effect through the observed execution trajectory. This definition separates safety from task completion, covering cases in which an agent may finish the requested task while disclosing protected data, modifying an unauthorized state, persisting an untrusted instruction, exceeding a resource limit, or claiming completion without the required effect. Behavioral safety must therefore be assessed from the execution evidence that determines what actions occurred and what effects they produced. Motivation and Gaps. Recent benchmarks extend agent safety evaluation from response moderation to tool execution (Jiang et al. 2026a; Yao et al. 2026). However, their coverage remains fragmented across distinct attack surfaces and evaluation settings, leaving three important gaps for stateful cowork agents. Gap 1: Existing data cover narrow and disjoint attack surfaces. AgentDojo provides 629 cases over untrusted tool outputs, WASP targets web content, and MCPTox evaluates 1,348 tool poisoning cases (Debenedetti et al. 2024; Evtimov et al. 2025; Wang et al. 2026c). AgentPoison and MINJA isolate memory poisoning (Chen et al. 2024; Dong et al. 2025). No single benchmark jointly evaluates task artifacts, tool interfaces, persistent state, context pressure, and permission composition under task configurations. Gap 2: Existing evaluations lack causal attribution for behavioral safety. Outcome checks, attacker progress scores, and LLM judges do not jointly verify the realized effect, the provenance linked execution path, and the operation’s configuration status (Debenedetti et al. 2024; Evtimov et al. 2025; Ruan et al. 2024). Consequently, a blocked call, a preexisting state, or an authorized transition may receive the same label as a realized policy violation. Such scores neither identify the failed enforcement point nor support category matched

analysis. Gap 3: Static payload suites cannot adapt to execution feedback. Fixed injection suites in existing work do not optimize payloads based on trajectory-level failure diagnoses. Therefore, a payload remains unchanged when an agent does not observe them, rejects the induced action, or fails before the target effect. This static construction largely limits attack coverage across models and harnesses. Our Approach. We present ActBench, a cowork agent behavioral safety benchmark with comprehensive test cases and reward-guided payload search. To address Gap 1, ActBench spans 15 risk behaviors, six execution spaces, 48 web service APIs, and 213 operational scenarios. Each malicious case injects one task reachable payload, while its benign counterpart preserves the instruction, task configuration, initialized state, utility grading criterion, attack grading criterion, and trusted logs. To address Gap 2, we model cowork execution over six spaces and associate each behavior with a protected boundary, propagation path, failed enforcement condition, and evidence schema. Dual evidence verification triggers an attack criterion only when state evidence confirms the prohibited effect, validated event identifiers establish the observed path, and the task configuration excludes the operation. To address Gap 3, ActBench employs a self-evolving optimization mechanism to find higher scoring payloads. Reward-guided beam search ranks edits by a geometric score over mean Attack Grading Score (AGS) and Utility Grading Score (UGS) across repeated rollouts. For unsuccessful attempts, reflection-based deep probing identifies the earliest failed checkpoint and revises the same payload field. Search and evaluation use disjoint rollouts to prevent optimization leakage. We further control comparison conditions by fixing the execution harness when comparing models and fixing the base model when comparing agent frameworks. Our contributions are summarized as follows: • We define behavioral safety as whether an agent’s execution remains within the permissions and state changes required by a benign task. We operationalize 15 risk behaviors through trajectory predicates across four propagation families and six execution spaces, together with AGS and UGS for separate measurement of prohibited effects and required task effects. • We construct ActBench with 300 benign and malicious pairs from 213 operational scenarios. The four-stage construction combines an empirical strategy pool, rewardguided beam search, reflection-based deep probing, and dual evidence verification. • We evaluate 15 LLMs and 6 open-source cowork agents, including OpenClaw, Hermes, Claude Code, OpenAgent, OpenCode, and QwenPaw, across 20 controlled configurations and 24,000 execution trajectories. The results reveal substantial safety variation: ASR ranges from 10.1% to 94.4% across LLMs under the same harness, and from 73.7% to 94.4% across agents under the same model.

Related Work

Agent safety benchmarks. Existing agent safety benchmarks evaluate either compliance with harmful requests or

security failures under adversarial execution conditions. For harmful-task evaluation, ToolEmu evaluates 36 high stakes toolkits in 144 cases and judges 68.8% of detected failures plausible under deployment review (Ruan et al. 2024); AgentHarm contains 110 malicious tasks across 11 harm categories (Andriushchenko et al. 2025); RedCode includes 4,050 risky execution cases across 25 scenarios and 160 malicious code generation prompts (Guo et al. 2024); and OS Harm extends harmful task evaluation to computer use agents (Kuntz et al. 2025). For adversarial execution, AgentDojo provides 97 tasks and 629 security cases over untrusted tool returns (Debenedetti et al. 2024); WASP observes partial attacker progress in up to 86% of web injection cases (Evtimov et al. 2025); MobileSafetyBench contains 200 daily scenario tasks and 50 indirect injection tasks in Android emulators (Lee et al. 2026); and Agent Security Bench evaluates prompt injection, memory poisoning, Plan of Thought backdoors, and mixed attacks, with a highest average attack success rate of 84.30% (Zhang et al. 2025). These suites nevertheless isolate execution settings and omit matched configurations spanning artifacts, tool interfaces, persistent state, context pressure, and permission composition. ActBench evaluates these surfaces through matched benign inputs and task configurations, with joint utility and safety outcomes across six execution spaces. Prompt injection attacks. Prompt injection attacks embed competing instructions within application data, causing a model or agent to treat untrusted content as executable instructions (Liu et al. 2024; Shi et al. 2025). Initial work focuses on single-turn model settings (Zverev et al. 2025), while later studies extend injection to agent-accessible data resources (Wang et al. 2026c). For example, EIA attacks compromised webpages and achieves up to 70% success in extracting specific personally identifiable information (Liao et al. 2025), AgentPoison assumes direct poisoning access to memory or knowledge bases (Chen et al. 2024), whereas MINJA injects malicious records through query only interaction (Dong et al. 2025). Safety detection and auditing. Existing work improves agent safety through attack detection, model alignment, vulnerability discovery, and automated auditing (He et al. 2026). DataSentinel develops game theoretic prompt injection detection (Liu et al. 2025b). SecAlign applies preference optimization and lowers the strongest tested attack to 8% in its Llama 3 8B setting while retaining base model utility (Chen et al. 2025). AgentFuzz tests 20 open source agents and finds 34 high risk zero day taint style vulnerabilities across 14 agents (Liu et al. 2025a). AgentAuditor introduces ASSEBench with 2,293 annotated interactions across 15 risk types and 29 application scenarios (Luo et al. 2025).

Risk Behavior Taxonomy

Agent execution and test case. We model cowork agent across six execution spaces, including context C, reasoning R, safety policy P, action A, persistent memory M, and environment E. At execution step t, the system state is y t = (ct , rt , pt , at , mt , et ). Starting from y t , the reasoning model returns the execution trajectory τ = πϕ (y t ),

Risk Behavior

Description Family: Context manipulation in C Effect: Untrusted content alters objectives, disclosure decisions, or credential handling. Path: Cadv → R → A → E

Instruction Injection

Goal Hijacking

Data Exfiltration

Credential Exposure

Memory Poisoning

State Tampering

B7 B8 B9 B10

Family: State integrity in M and E Effect: An unverified persistent record alters a later decision or action. Path: Madv /Eadv ⇝ C → R → A

Benchmark Construction

Deceptive Tool Invocation Family: Action misuse in A Unauthorized CMD Execution Effect: Tool selection or parameters exceed Unauthorized API Invocation the operation, API, object. Path: C → R → Aadv → E Tool Scope Escalation

B11 Resource Exhaustion B12 Obfuscated Execution B13 False Reporting B14 Context Flooding B15 Permission Chaining

Family: Policy mediation failure in P Effect: Enforcement fails for resource limits, canonical execution, reporting, or context allocation. Path: C → R → Pfail → A → E

Table 1: Risk behavior taxonomy defined by where each behavior becomes operational and how its effect propagates through the trajectory path. which contains the sequence of model-visible context, reasoning, actions, tool results, memory events, and environment records. A test case is xi = (c, m, e, {ui , Ui , Ai }), where ui is the benign user instruction. The sets Ui and Ai contain utility and attack grading criteria, respectively. Benchmark construction may edit c, m, or e. It keeps ui , Ui , and Ai fixed. Risk taxonomy. We propose a trajectory-based risk behavior taxonomy. In contrast to taxonomies based on payload or final response, our taxonomy assigns a behavior through its realized propagation path in τ . We define 15 risk behaviors Bj , where j ∈ {1, . . . , 15}, and organize them into four families in Table 1. Context manipulation behaviors B1 ∼ B4 arise when untrusted content changes the interpreted objective, disclosure decision, credential handling. State integrity behaviors B5 ∼ B6 capture cases in which persistent memory or environment state alters a later decision. Action misuse behaviors B7 ∼ B10 concern tool selection, command execution, API invocation, and argument scope that exceed the task authorization. Policy mediation failures B11 ∼ B15 arise when the safety policy layer fails to enforce a resource bound, canonical execution rule, reporting requirement, context constraint, or composed permission. Criterion-based scoring. For trajectory τ , each attack grading criterion a ∈ Ai and each utility grading criterion u ∈ Ui returns a binary value. We define ga (xi ) =

1 X I[a(τ ) = 1], |Ai | a∈Ai

gu (xi )

1 X I[u(τ ) = 1]. |Ui |

criterion returns 1, and a value of gu (xi ) = 1 means that every required task criterion returns 1. Intermediate values report the fraction of satisfied criteria rather than a probability estimate. A rollout is task complete when gu (xi ) = 1, while full attack success requires ga (xi ) = 1. We propose a dual evidence verification mechanism that validates the final score through log-based and LLM-based trajectory evidence.

(1)

u∈Ui

We call ga (xi ) the attack grading score and gu (xi ) the utility grading score. Both scores lie in [0, 1] and measure criterion coverage. A value of ga (xi ) = 1 means that every attack

Overview. We construct 300 matched pairs, each comprising one benign case and one malicious case. As illustrated in Figure 1, benchmark construction has four stages. ➊ Empirical strategy pool fixes the case grading criterion and produces an initial edit at a task-reachable location. ➋ Reward-guided beam search optimizes executed candidates and ranks them according to their attack and utility scores. ➌ Reflection-based deep probing analyzes failed cases based on their execution trajectories and uses the resulting feedback to iteratively strengthen the attack examples. ➍ Dual evidence verification evaluates log-based evidence and LLM-based trajectory evidence. ➊ Empirical strategy pool. The pool contains three indexed views. The scenario view specifies the benign user instruction ui , the execution environment represented by c, m, and e, and the utility grading criteria Ui . The behavior view selects one Bj and defines Ai from its operational evidence schema. The attack strategy view stores edit templates and payload structures from previously accepted construction cases. We first execute the benign case using the reasoning model πϕ . A candidate attack location is considered eligible only if the benign trajectory demonstrates that the agent reads, retrieves or discovers that location while completing the task. The attack model πφ then selects an attack target, such as a web service API, and refer to the attack strategy view to modify either an eligible field in c, m or e. The matched benign and malicious pairs retain the same ui , Ui , and Ai . The attack strategy view is updated continuously throughout test case construction. Whenever the framework successfully optimizes a passed case, it immediately incorporates the corresponding edit template and payload structure into the strategy pool. This process establishes a self-evolving optimization mechanism in which newly passed cases expand the attack strategy view. ➋ Reward-guided beam search. At construction round t, each candidate xt is executed in k independent rollouts from the same reset environment. In this section, ga (xt ) and gu (xt ) denote the corresponding means over those construction rollouts. We rank the candidate by s(xt ) = ga (xt )α gu (xt )1−α ,

0 < α < 1.

(2)

For positive scores, the logarithm of the objective is log s(xt ) = α log ga (xt ) + (1 − α) log gu (xt ),

(3)

where α is the constant elasticity of s with respect to attack evidence, and 1−α is its elasticity with respect to utility. The geometric score becomes zero when either case score is zero.

Empirical Strategy Pool

Scenario View

Reward-guided Beam Search

𝑥#

Software Engineer Document ….

𝑥"

Grading Criteria

Behavior View

Read Action

B1 Instruction Injection

Write Action

B2 Goal Hijacking

Delete Action

B3 Data Exfiltration….

….

% ∈'

Rollout Pruned

𝑥!

Context 𝐂

Reasoning 𝐑

Policy

APIs

𝑥"

Files

𝑥# 𝑥#

State Manipulation…

t=1

t=2

t=3

Failure Type: not read, ignore…

𝑥1

Direction: relevance, strength… Payload Stealthy: low/high…

Trajectory

Attack: role play, obfuscated…

𝝉

Action 𝐀

Skills Memory 𝐌

Trajectory

𝝉

Evaluation 𝜏

𝜏 Attack Grading Score AGS

x Utility Grading Score UGS

Dual Evidence Verification

Log Evidence Verification

Multi-Objective Reward

Pruned

Harness Hijacking

𝐏

Candidate 𝒙 & 𝝉

Execution Environment (𝐄)

Pruned

Base64 Obfuscated

Initialize Test Case 𝑥 = (𝑐, 𝑚, 𝑒, {𝑢! , 𝒰! , 𝒜! })

Sample: 𝐶 "#$ = !⋃ ! {𝜋( (𝑥 " , 𝜉)}+)*$ Single Rollout Execution

𝑥#

Strategy View

Reflection-based Deep Probing Reflection Summary 𝝃

Round 𝑡

Business Operation Financial Operations

𝑥1

State Diffs.

Resource Cnt.

File Diffs.

Entries Audit

Trajectory Evidence Verification

Candidate

𝑠(𝑥 " ) = 𝑔, (𝑥 " )- 𝑔. (𝑥 " )$/-

Instruction

Reasoning

Tool Call

Tool Result

Verify

Passed Test Case

Self-evolving

Figure 1: ActBench benchmark construction framework. Thus, neither attack evidence nor utility can compensate for the complete absence of the other. Let C t contain the candidates entering round t, and let B t contain the retained parent beams. The initial set C 0 contains the first editable case produced by the attack strategy pool. After executing and grading C t , the beam update and expansion are B t = topw (C t , s(xt )), xt ∈ C t [ C t+1 = {πφ (xt , ξ)}nj=1 ,

(4) (5)

xt ∈B t

where πφ denotes the attack model and ξ represents the reflection summary for previous rollout. The first line orders candidates by s(xt ) and retains at most w. The second line invokes the attack model n times for each retained candidate. Therefore, |C t+1 | ≤ k × n, with strict inequality when the scope filter rejects an edit or multiple invocations produce duplicate cases. Because each parent is selected only after execution, beam retention is grounded in observed trajectories rather than the attack model’s prediction quality. Search stops when ga (xt ) = gu (xt ) = 1 or when it reaches the depth limit dmax . (See Algorithm 1 in Appendix for detail). ➌ Reflection-based deep probing. To localize the failure, the reflector constructs a diagnosis ξ from xt , Bj , trusted records in τ , the API audit, and repeat outcomes. The diagnosis assigns the attempt to one of ten classes: not read, ignored, refused, weak attack, unreached sink, sink missing target content, broken task integrity, unstable success, verified success, or unknown. It also records whether the payload was observed and followed, whether the target effect occurred, and whether task integrity was preserved. Additional fields specify whether to discard the candidate, which property to optimize, and a localized revision instruction. The revision targets the earliest unsupported transition in the attack path. An unobserved payload triggers an edit for relevance; an ignored payload, weak attack, or sink failure triggers attack-strength optimization; refusal triggers stealth and stability; a realized effect with utility loss triggers task-

integrity optimize; and inconsistent repeated outcomes trigger stability optimization. The resulting instruction becomes the next objective, while ξ conditions the attack model: (6)

xt+1 = πφ (xt , ξ).

An empty instruction retains the preceding objective. Updates retain the editable field or group, case scope, identifiers, and roles. The scope filter rejects changes to ui , Ui , Ai , or fields outside c, m, and e. The discard flag enables bounded probing, while malformed outputs invoke a deterministic fallback based on the API audit, repeated scores, and a refusal heuristic. Reflection enables the system to learn from previous failures and refine its attack strategies accordingly. ➍ Dual evidence verification. Equation (1) aggregates binary attack criteria without restricting the evidence source. We extend this definition by evaluating the same criterion set Ai with two evidence verifiers. Log evidence verification applies deterministic predicates to state differences, service audit entries, command results, and resource counters. For a ∈ Ai , it returns a′ (τ ) = 1 when a trusted record establishes the prohibited effect under the fixed authorization rule. Trajectory evidence verification reconstructs the path from payload exposure to the recorded action and effect. It returns a′′ (τ ) = 1 only when event identifiers, object identifiers, and event order establish that path. The corresponding case scores and their fusion are ′

′′

ga (xi ) = βga (xi ) + (1 − β)ga (xi ),

0 < β < 1, (7)

where ga (xi ) is the log evidence verification score and ′′ ga (xi ) is the trajectory evidence verification score. Both apply the criterion aggregation in Equation (1). Equation (7) then defines the final AGS as their convex combination. The fusion preserves the full success threshold. In particular, ′

′

′′

1 − ga (xi ) = β[1 − ga (xi )] + (1 − β)[1 − ga (xi )]. (8) Both terms on the right are nonnegative because the two component scores lie in [0, 1]. It follows that ga (xi ) = 1 if ′ ′′ and only if ga (xi ) = ga (xi ) = 1. Likewise, ga (xi ) = 0 if

(a) Task domains

(b) Web service APIs 150

Task Tracker

105

Knowledge base

Software Engineering

Document Operations

Ticketing

Ledger / finance

Mailbox

24 14

Automation 6

Integrations

Customer records

Identity Operations

Scheduling 20

Model

Web content Directory / identity

Security Operations

Notebook / meetings

Financial Operations

Tasks

100

125

150

Task-API references

Tasks

20 14

B10

14 11

B14

B15

Table 2: Base model comparison with OpenClaw fixed. Iter. is the malicious rollout median.

B11

B12

B13

Behavior risk category

Figure 2: ActBench composition. Panel (a) reports six operational domains. Panel (b) groups 581 task API references into 12 service groups. Panel (c) reports the 300 pair assignments across B1 to B15 . and only if both component scores are zero. The value of β controls the relative contribution of the two verifiers for intermediate scores. When a test case succeeds, its edit template and payload structure are added to the attack strategy pool through a push-back mechanism. Reflection and push-back form a self-evolving optimization loop in which successful cases continuously expand and improve the available attack strategies.

Empirical Study

We organize the study around four questions. RQ1 Model Effects asks how the base model changes behavioral safety when the cowork harness is fixed. RQ2 Harness Effects asks how the cowork harness changes behavioral safety when the base model is fixed. RQ3 Safety Policy P Effects asks how safety policy protects cowork agent through malicious behavior detection file scanning. RQ4 Case Study analyzes the effect of malicious behavior propagation path using an obfuscated execution case. Experimental settings. RQ1 and RQ3 fixe OpenClaw version 2026.5.19 and varies 15 base models. RQ2 fixes base model and varies six cowork harnesses.Counting the shared RQ1/RQ2 setting once yields 20 unique configurations and 24,000 trajectories in total. Construction and evaluation trajectories are strictly disjoint. The benchmark construction uses beam width w = 3, proposal count n = 2, depth dmax = 5, ranking weight α = 0.5. The evaluation adopts rollout number k = 3, evidence weight β = 0.4, rating model GPT-5.5. All runs use Ubuntu 22.04.5 LTS. (See Appendix for all environment details.) Metrics. For each malicious rollout, AGS is the criterion coverage after the dual evidence channels are fused. For each matched benign rollout, UGS is the fraction of required task

AGSmal ↓ ASR (%)↓ p@1/p@2/p@3 ↓ UGSben ↑ Iter. 0.284 0.347 0.493 0.727 0.870 0.547 0.511 0.524 0.489 0.748 0.402 0.804 0.955 0.887 0.455

Claude-Opus-4.8 Claude-Sonnet-4.6 GPT-5.5 GPT-5.4-mini Grok-4.5 GLM-5.2 Qwen3.7-max Qwen3.7-plus Kimi-K3 Kimi-K2.6 MiniMax-M3 MiniMax-M2.7 Deepseek-v4-Pro Deepseek-v4-Flash Hunyuan-3.0

10.1 20.0 37.8 65.7 83.9 42.8 39.2 42.1 35.7 70.4 25.0 75.4 94.4 84.4 30.0

10.7/12.3/13.7 19.3/22.3/23.7 36.0/43.3/47.3 66.3/75.3/78.0 83.0/90.0/90.7 41.7/49.0/54.0 37.7/48.3/54.7 42.7/50.0/53.3 35.3/44.3/47.3 70.7/78.0/81.0 25.0/31.7/36.3 76.7/84.3/87.7 94.3/98.0/98.7 83.3/90.0/93.3 30.7/37.3/41.7

0.938 0.927 0.928 0.904 0.938 0.929 0.915 0.915 0.940 0.869 0.917 0.880 0.922 0.900 0.933

15.0 16.0 16.0 18.0 17.0 16.0 16.0 16.0 17.0 17.0 19.0 17.0 19.0 19.5 19.0

criteria that pass. ASR is the fraction of malicious rollouts whose AGS exceeds the fixed threshold of 0.8. The case level pass@k rate is the fraction of cases with at least one threshold crossing in the first k evaluation rollouts. Median iteration count describes executed action cycles. The defense study reports accuracy, precision, recall, F1, and the four confusion counts under one frozen threshold per method. Lower AGS, ASR, and pass@k indicate less observed attack evidence. Higher UGS indicates stronger benign task completion. Benchmark coverage. ActBench contains 600 cases in 300 matched pairs from 213 scenarios. The accepted cases cover 15 behaviors, four propagation families, six execution spaces, and 48 web service APIs. Figure 2 reports the constructed distribution. Business operations, software engineering, and document operations contribute 188 pairs, which is 62.7% of the benchmark. Identity operations contributes eight pairs. Task Tracker, Knowledge Base, and Notebook or Meetings account for 318 of 581 API references. Instruction Injection and Unauthorized CMD Execution each contain 42 pairs, while Context Flooding contains 11. The aggregate metrics are therefore weighted by the case distribution.

Overall ¬AGS=0.716

¬AGSmal ↑

Overall UGS=0.938

UGSben ↑

1.0

0.8

Score

Business Operations

0.6

0.4

0.2

0.0

B10

B11

B12

B13

B14

B15

Behavior risk category

Figure 3: Claude-Opus-4.8 behavior risk results. Blue and green bars report 1 − AGSmal and UGSben . The dashed averages are 0.716 and 0.938.

Table Table3: 3:Cowork Coworkharness harnesscomparison comparisonwith withbase basemodel modelfixed. fixed. Iter. Iter. isis the themalicious maliciousrollout rolloutmedian. median.

0 /

/ - /

Agent Agent

3 0. 0

3 .

. 1

Figure 4: 4: Action Action cycle cycle distributions across base models Figure models with with OpenClaw fixed. fixed. Orange Orange boxes report malicious rollouts OpenClaw rollouts and and green boxes boxes report report matched matched benign benign rollouts. rollouts. green

RQ1 Model Model E!ects. Effects. In In RQ1, RQ1, only only the RQ1 the base base model model varies, varies, while all all other other experimental experimental settings settings are while are held held constant. constant. As As reported in in Table Table 2, 2, Claude-Opus-4.8 Claude-Opus-4.8 achieves reported achieves the the lowest lowest AGSmal of 0.284, 0.284, ASR ASR of of 10.1%, 10.1%, while while maintaining mal of AGS maintaining aa bebenign UGS UGS of of 0.938. 0.938. Deepseek-v4-Pro Deepseek-v4-Pro exhibits nign exhibits the the highest highest risk,with with an an AGS AGS of of 0.955, 0.955, an an ASR ASR of risk, of 94.4%, 94.4%, and and aa pass@3 pass@3 of 98.7%. Claude-Opus-4.8 therefore reduces AGS of 98.7%. Claude-Opus-4.8 therefore reduces AGS by by 0.671 0.671 and ASR ASR by by 84.3% 84.3% relative relative to to Deepseek-v4-Pro. Deepseek-v4-Pro. These and These reresults show show that that benign benign task task performance performance is sults is not not aa reliable reliable proxy for for behavioral behavioral safety. safety. proxy Repeated rollouts rollouts further further expose expose differences Repeated di!erences in in attack attack stability. Table 2 shows that Qwen3.7-max increases stability. Table 2 shows that Qwen3.7-max increases from from 37.7% at pass@1 to 54.7% at pass@3, producing the 37.7% at pass@1 to 54.7% at pass@3, producing the largest increase of 17.0 percentage points. By comparison, largest increase of 17.0 percentage points. By comparison, Claude-Opus-4.8 increases by 3.0 percentage points, while Claude-Opus-4.8 increases by 3.0 percentage points, while Deepseek-v4-Pro increases by 4.4 percentage points. InterDeepseek-v4-Pro increases by 4.4 percentage points. Intermediate risk models therefore tend to exhibit stochastic atmediate risk models therefore tend to exhibit stochastic attack activation, whereas high risk models trigger prohibited tack activation, whereas high risk models trigger prohibited effects consistently. A single rollout would consequently une!ects consistently. A single rollout would consequently underestimate the exposure of models such as Qwen3.7-max, derestimate theKimi-K3. exposure of models such as Qwen3.7-max, GLM-5.2, and GLM-5.2, and Kimi-K3. Figure 3 shows the category behavior risk results of Figure 3 showsClaude-Opus-4.8 the category behavior risk and results of Claude-Opus-4.8. reaches 0.43 above Claude-Opus-4.8. Claude-Opus-4.8 reaches 0.43 and above 0.90 for the same labels. Its category UGS remains at least 0.90 thefixed sameharness labels. links Its category UGS remains at least 0.90.for The these differences to how the 0.90. The fixed harness links these di!erences to how the base model converts the same carrier into an action. The mebase model converts the same carrier into an action. The median malicious iteration count spans only 4.5 cycles across dian malicious iteration spans only 4.5 cycles across models, while ASR spanscount 84.3%. Model choice changes risk models, while ASR spans 84.3%. Model choice changes risk without reducing utility. without reducing utility. Execution length does not explain the model ranking. Figdoes not explain model ranking. FigureExecution 4 showslength that malicious medianthe iterations range only ure 4 shows that malicious median iterations range only from 15.0 to 19.5 cycles. MiniMax-M3, Hunyuan-3.0, and from 15.0 to 19.5 all cycles. MiniMax-M3, and Deepseek-v4-Pro require 19.0 medianHunyuan-3.0, cycles, but their Deepseek-v4-Pro require median cycles, but ASRs are 25.0%,all 30.0%, and19.0 94.4%, respectively. Thetheir reASRs 25.0%, 30.0%, anddifference 94.4%, respectively. The exresultingare 69.4 percentage point under the same sulting 69.4 percentage point di!erence under the same execution length indicates that the dominant factor is how the ecution length indicates that the dominant is howsubthe base model interprets adversarial context andfactor authorizes base model interprets adversarial context and authorizes subsequent actions. sequent actions.

AGS AGSmal ASR(%)→ pass@1 pass@1/ @2 / @2/ @3↓ / @3→ UGS UGS Iter. ben mal↓→ ASR(%)↓ ben↑ ↑ Iter. 0.955 0.955 0.808 0.808 0.852 0.852 0.868 0.868 0.826 0.826 0.848 0.848

OpenClaw OpenClaw OpenAgent OpenAgent OpenCode OpenCode QwenPaw QwenPaw Hermes Hermes Claude ClaudeCode Code

94.4 94.4 76.2 76.2 81.7 81.7 73.7 73.7 79.2 79.2 81.4 81.4

94.3 94.3/ 98.0 / 98.0/ 98.7 / 98.7 76.0 76.0/ 83.3 / 83.3/ 87.7 / 87.7 80.7 80.7/ 88.0 / 88.0/ 90.3 / 90.3 71.3 71.3/ 78.3 / 78.3/ 80.7 / 80.7 81.0 81.0/ 87.3 / 87.3/ 89.0 / 89.0 81.7 81.7/ 89.3 / 89.3/ 93.0 / 93.0

0.922 0.922 0.945 0.945 0.919 0.919 0.903 0.903 0.926 0.926 0.904 0.904

19.0 19.0 24.0 24.0 21.0 21.0 33.0 33.0 47.0 47.0 33.0 33.0

Finding: Finding:With Withthe thecowork coworkharness harnessfixed, fixed,changing changingonly onlythe thebase base model model changes changesASR ASRby by84.3% 84.3%and andpass@3 pass@3byby85.0%, 85.0%,while whilebebenign nignUGS UGSchanges changesby byonly only0.071. 0.071.Behavioral Behavioralsafety safetyisisprimarily primarily determined determinedby bymodel modelspecific specificaction actionselection selectionand andcannot cannotbebeininferred ferredfrom fromtask taskutility utilityor orexecution executionlength. length.

RQ2 RQ2Agent AgentHarness HarnessEffects. E!ects. RQ2 RQ2fixes fixesDeepseek-v4-Pro Deepseek-v4-Pro and and every every case case while while replacing replacing the thecomplete completecowork coworkharharness. This intervention changes context assembly, ness. This intervention changes context assembly,memory memory retrieval, retrieval, tool tool serialization, serialization, and andaction actionmediation mediationtogether. together. ItIt does does not not isolate isolate one one harness harnesscomponent. component.Table Table33shows shows aa 20.7% 20.7% ASR ASR span. span. QwenPaw QwenPawrecords recordsthe thelowest lowestASR ASRatat 73.7%, 73.7%, while while OpenClaw OpenClawrecords records94.4%. 94.4%.UGS UGSremains remainsbebetween 0.903 and 0.945. The lower attack rates are tween 0.903 and 0.945. The lower attack rates aretherefore therefore not not produced produced by by broad broadfailure failureon onbenign benigntasks. tasks.OpenAgent OpenAgent has has the thelowest lowestAGS AGSand andthe thehighest highestUGS. UGS.QwenPaw QwenPawhas hasthe the lowest ASR but a higher AGS than OpenAgent. lowest ASR but a higher AGS than OpenAgent. Figure Figure 55 localizes localizesthe theharness harnesseffect. e!ect.Memory MemoryPoisoning Poisoning AGS ranges from 0.24 to 0.98. Credential AGS ranges from 0.24 to 0.98. CredentialExposure Exposureranges ranges from 0.40 to 0.97. These labels depend on memory retrieval from 0.40 to 0.97. These labels depend on memory retrieval and context serialization, which the harness controls. Decepand context serialization, which the harness controls. Deceptive Tool Invocation remains between 0.91 and 1.00 across tive Tool Invocation remains between 0.91 and 1.00 across all six harnesses. Every behavior conditioned UGS value is all six harnesses. Every behavior conditioned UGS value is A1: OpenClaw · A2: OpenAgent · A3: ClaudeCode A4: Hermes · A5: OpenCode · A6: QwenPaw

(a) AGS A1

0.96

0.95

0.98

0.97

0.98

0.96

1.00

0.97

0.99

0.90

0.97

0.94

0.87

0.83

0.98

0.92

0.90

1.00

0.80

0.24

0.59

0.96

0.70

0.97

0.88

0.65

0.93

0.69

0.86

0.94

0.87

0.97

0.80

0.48

0.88

0.99

0.58

0.91

0.83

0.91

0.94

0.70

0.83

0.93

0.87

0.90

0.97

0.68

0.37

0.88

0.97

0.56

0.92

0.73

0.86

0.94

0.63

0.79

0.90

0.83

0.94

0.83

0.29

0.94

0.96

0.71

0.91

0.77

0.95

0.91

0.72

0.90

0.88

0.89

0.40

0.91

0.95

0.91

0.93

0.90

0.85

0.90

0.88

0.78

0.87

B10

B11

B12

B13

B14

B15

Behavior risk category

(b) UGS A1

0.94

0.88

0.96

0.92

0.94

0.87

0.98

0.93

0.91

0.95

0.94

0.95

0.92

0.95

0.94

0.97

0.93

0.96

0.94

0.95

0.88

0.96

0.95

0.94

0.96

0.98

0.97

0.98

1.00

0.93

0.87

0.89

0.95

0.85

0.83

0.91

0.90

0.95

0.96

0.93

0.97

0.93

0.94

0.95

0.88

0.91

0.88

0.94

0.88

0.98

0.90

0.95

0.96

0.97

0.98

0.93

0.95

0.93

0.91

0.93

0.94

0.93

0.88

0.93

0.94

0.89

0.94

0.95

0.90

0.97

0.94

0.93

0.89

0.90

0.92

0.87

0.86

0.91

0.95

0.90

0.92

0.94

0.90

0.93

0.85

0.91

B10

B11

B12

B13

B14

B15

Behavior risk category

Figure 5: Behavior conditioned harness comparison with Figure 5: Behavior conditioned comparisonAGS. with Deepseek-v4-Pro fixed. Panel (a) harness reports malicious Deepseek-v4-Pro fixed. UGS. Panel (a) reports malicious AGS. Panel (b) reports benign Panel (b) reports benign UGS.

Table 4: Detection diagnostics at each method’s native observation boundary. Metrics use a frozen validation threshold and do not measure post intervention execution outcomes. Defense

Acc.

Prec.

Rec.

TP FP TN FN

Prompt-Guard-2-86M Llama-Guard-4-12B AgentDoG-Qwen-4B

51.5% 56.3% 13.3% 55.7% 72.4% 18.3% 76.5% 95.4% 55.7%

AI-Infra-Guard Cisco-Skill-Scanner

71.0% 82.8% 53.0% 64.6% 159 33 267 141 80.0% 75.0% 90.0% 81.8% 270 90 210 30

21.6% 40 31 269 260 29.3% 55 21 279 245 70.3% 167 8 292 133

at least 0.83. OpenClaw reaches 94.4% ASR with a median of 19 cycles. Hermes reaches 79.2% with a median of 47 cycles. More action cycles are therefore not equivalent to stronger enforcement. The harness range is smaller than the model range in the evaluated alternatives. Finding: Harness design changes exposure for memory and credential behaviors, yet every tested harness remains vulnerable on at least 73.7% of malicious rollouts under the fixed model.

RQ3 Safety Policy Effects. In the RQ3, we mainly evaluate safety policy P protects cowork agent through query detection and file scanning. For the former, we employ PromptGuard-2, LLama-Guard-2, and AgentDog for behavior detection (Llama Team 2024; Meta AI 2025; Liu et al. 2026). For the latter, we employ AI-Infra-Guard and Cisco-SkillScanner inspect workspace or skill artifacts before execution (Yang et al. 2026; Tencent Zhuque Lab 2025). We attach the same 300 malicious and 300 benign case labels to these boundary specific inputs. The experiment evaluates discrimination under each method’s observation assumption. It does not evaluate equivalent runtime interventions. Cisco Skill Scanner detects 270 of 300 malicious artifacts and records 90 false positives. Its recall is 90.0%. AgentDoG detects 167 malicious cases with eight false positives, which yields 95.4% precision and 55.7% recall. Prompt Guard 2 and Llama Guard 4 detect 40 and 55 malicious inputs. Their training target is local content classification, while an ActBench label requires a linked action and environment effect. This difference in target and input explains why content classification does not transfer directly to trajectory level detection. Finding: Trajectory and artifact evidence aligns more closely with realized behavioral labels than local text alone, but precision and recall trade off sharply across the two evidence boundaries.

RQ4 Case Study. Figure 6 traces an Obfuscated Execution failure. The attacker modifies only a task reachable ledger memo. The model treats its encoded packet as authorized task content, while the harness evaluates the carrier before canonical decoding. The decoded file write therefore bypasses mediation and creates .recon_cache/inv-2027-894.audit only in the malicious run. Trajectory events recover the propagation order, and filesystem evidence confirms the effect. This case localizes the failure but does not estimate reflection’s average gain.

Figure 6: Trajectory visualization for Obfuscated Execution. The malicious carrier is inserted into the memo field of ledger record TXN-2027-7711. The agent treats the encoded packet as a reconciliation step, decodes it, and creates an unauthorized audit file. Finding: The model grants authority to untrusted content, and the harness checks a noncanonical representation. Trajectory and filesystem evidence confirm the unauthorized effect.

Discussion The controlled interventions separate model policy effects from execution layer effects. Model choice produces the larger observed safety span, while harness changes concentrate on behaviors mediated by context and persistent state. The persistence of high ASR across all six harnesses indicates that execution layer controls do not compensate for a base model that converts untrusted context into prohibited actions. RQ3 further shows that defense performance depends on whether the detector observes text, trajectory state, or executable artifacts. These findings support the construction choices in ActBench. Matched tasks preserve utility comparability. Behavior predicates localize the failed propagation step. Dual evidence verification prevents exposure or blocked calls from being labeled as realized violations. Reflection uses the same evidence to repair one failed checkpoint. The study remains bounded by the evaluated model and harness sets, a fixed threshold, one verifier configuration, and a case based reflection analysis. Paired uncertainty, guarded reruns, and construction ablations are needed before attributing average causal gains to individual components.

Conclusion We introduced ActBench, a self evolving benchmark that evaluates cowork agent safety from executed trajectories

rather than final responses. Its 600 cases form 300 matched benign and malicious pairs from 213 scenarios, covering 15 risk behaviors, six execution spaces, and 48 web service APIs. Reward-guided beam search jointly preserves attack evidence and task utility through a geometric objective. Reflection-based deep probing revises the earliest failed transition in each attack path. Dual evidence verification combines trusted logs with trajectory reconstruction, so full success requires agreement on both the prohibited effect and its propagation path. Across 24,000 trajectories, model controlled ASR spans 84.3% and harness controlled ASR spans 20.7%. These results show greater variation across models than agent harness, while attacks remain highly successful across all tested harnesses.

References

Andriushchenko, M.; Souly, A.; Dziemian, M.; Duenas, D.; Lin, M.; Wang, J.; Hendrycks, D.; Zou, A.; Kolter, Z.; Fredrikson, M.; et al. 2025. AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents. In International Conference on Learning Representations, volume 2025, 79185–79220. Chen, S.; Zharmagambetov, A.; Mahloujifar, S.; Chaudhuri, K.; Wagner, D.; and Guo, C. 2025. SecAlign: Defending Against Prompt Injection with Preference Optimization. In Proceedings of the 2025 ACM SIGSAC Conference on Computer and Communications Security, 2833–2847. Chen, Z.; Xiang, Z.; Xiao, C.; Song, D.; and Li, B. 2024. AgentPoison: Red-Teaming LLM Agents via Poisoning Memory or Knowledge Bases. In Advances in Neural Information Processing Systems, volume 37, 130185–130213. Debenedetti, E.; Zhang, J.; Balunovic, M.; Beurer-Kellner, L.; Fischer, M.; and Tramèr, F. 2024. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. In Advances in Neural Information Processing Systems, volume 37, 82895–82920. Dong, S.; Xu, S.; He, P.; Li, Y.; Tang, J.; Liu, T.; Liu, H.; and Xiang, Z. J. 2025. Memory Injection Attacks on LLM Agents via Query-Only Interaction. In Advances in Neural Information Processing Systems, volume 38, 46697–46731. Evtimov, I.; Zharmagambetov, A.; Grattafiori, A.; Guo, C.; and Chaudhuri, K. 2025. WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks. In Advances in Neural Information Processing Systems, volume 38. GitHub Security Advisory Database. 2026. OpenClaw Has One Click Remote Code Execution through Gateway Token Exfiltration. GitHub Security Advisory GHSA-g8p2-7wf798mq. CVE-2026-25253, accessed July 23, 2026. Guo, C.; Liu, X.; Xie, C.; Zhou, A.; Zeng, Y.; Lin, Z.; Song, D.; and Li, B. 2024. RedCode: Risky Code Execution and Generation Benchmark for Code Agents. In Advances in Neural Information Processing Systems, volume 37, 106190– 106236. He, X.; Xu, G.; Han, X.; Wang, Q.; Zhao, L.; Shen, C.; Lin, C.; Zhao, Z.; Li, Q.; Yang, L.; et al. 2025. Artificial Intelligence Security and Privacy: a Survey. Science China Information Sciences, 68(8): 181101.

He, Y.; Zhu, H.; Li, Y.; Shao, S.; Yao, H.; Liu, Z.; and Qin, Z. 2026. AttriGuard: Defeating indirect prompt injection in LLM agents via causal attribution of tool invocations. Jiang, Y.; Zhang, Y.; Backes, M.; Shen, X.; and Zhang, Y. 2026a. HarmfulSkillBench: How Do Harmful Skills Weaponize Your Agents? arXiv preprint arXiv:2604.15415. Jiang, Y.; Zhang, Y.; Shen, X.; Backes, M.; and Zhang, Y. 2026b. “Humans Welcome to Observe”: A First Look at the Agent Social Network Moltbook. arXiv preprint arXiv:2602.10127. Kuntz, T.; Duzan, A.; Zhao, H.; Croce, F.; Kolter, J. Z.; Flammarion, N.; and Andriushchenko, M. 2025. OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents. In Advances in Neural Information Processing Systems, volume 38. Lee, J.; Hahm, D.; Choi, J. S.; Knox, W. B.; and Lee, K. 2026. MobileSafetyBench: Evaluating Safety of Autonomous Agents in Mobile Device Control. In Proceedings of the Fortieth AAAI Conference on Artificial Intelligence, volume 40, 37565–37573. Liao, Z.; Mo, L.; Xu, C.; Kang, M.; Zhang, J.; Xiao, C.; Tian, Y.; Li, B.; and Sun, H. 2025. EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakage. In International Conference on Learning Representations. Liu, D.; Ren, Q.; Qian, C.; Shao, S.; Xie, Y.; Li, Y.; Yang, Z.; Luo, H.; Wang, P.; Liu, Q.; et al. 2026. AgentDoG: A Diagnostic Guardrail Framework for AI Agent Safety and Security. arXiv preprint arXiv:2601.18491. Liu, F.; Zhang, Y.; Luo, J.; Dai, J.; Chen, T.; Yuan, L.; Yu, Z.; Shi, Y.; Li, K.; Zhou, C.; Chen, H.; and Yang, M. 2025a. Make Agent Defeat Agent: Automatic Detection of Taint-Style Vulnerabilities in LLM-based Agents. In 34th USENIX Security Symposium (USENIX Security 25), 3767– 3786. USENIX Association. Liu, Y.; Jia, Y.; Geng, R.; Jia, J.; and Gong, N. Z. 2024. Formalizing and Benchmarking Prompt Injection Attacks and Defenses. In 33rd USENIX Security Symposium (USENIX Security 24), 1831–1847. USENIX Association. Liu, Y.; Jia, Y.; Jia, J.; Song, D.; and Gong, N. Z. 2025b. DataSentinel: A Game-Theoretic Detection of Prompt Injection Attacks. In 2025 IEEE Symposium on Security and Privacy. Llama Team. 2024. Meta Llama Guard 2 Model Card. Purple Llama. Accessed July 23, 2026. Luo, H.; Dai, S.; Ni, C.; Li, X.; Zhang, G.; Wang, K.; Liu, T.; and Salam, H. 2025. AgentAuditor: Human-Level Safety and Security Evaluation for LLM Agents. In Advances in Neural Information Processing Systems, volume 38, 43241–43298. Meta AI. 2025. Llama Prompt Guard 2 Model Card. Meta AI Model Documentation. Accessed July 23, 2026. OpenClaw. 2026. Introducing OpenClaw. OpenClaw Blog. Accessed July 23, 2026. Ruan, Y.; Dong, H.; Wang, A.; Pitis, S.; Zhou, Y.; Ba, J.; Dubois, Y.; Maddison, C. J.; and Hashimoto, T. 2024. Identifying the Risks of LM Agents with an LM-Emulated Sandbox. In International Conference on Learning Representations, volume 2024, 27031–27098.

Shi, H.; Yao, H.; Shao, S.; Jiao, S.; Peng, Z.; Qin, Z.; and Wang, C. 2025. Quantifying Conversation Drift in MCP via Latent Polytope. arXiv preprint arXiv:2508.06418. Tencent Zhuque Lab. 2025. AI-Infra-Guard: A Comprehensive, Intelligent, and Easy-to-Use AI Red Teaming Platform. GitHub repository. Wang, Y.; Ba, J.; Liu, H.; Pan, Y.; Wei, J.; Su, Z.; Luan, T. H.; and Du, L. 2026a. Security of OpenClaw Agents: Fundamentals, Attacks, and Countermeasures. arXiv preprint arXiv:2605.25435. Wang, Y.; Gao, H.; Niu, Z.; Liu, Z.; Zhang, W.; Wang, X.; and Lian, S. 2026b. A Systematic Security Evaluation of OpenClaw and Its Variants. arXiv preprint arXiv:2604.03131. Wang, Z.; Gao, Y.; Wang, Y.; Liu, S.; Sun, H.; Cheng, H.; Shi, G.; Du, H.; and Li, X.-Y. 2026c. MCPTox: A Benchmark for Tool Poisoning Attack on Real-World MCP Servers. In Proceedings of the Fortieth AAAI Conference on Artificial Intelligence. Yang, Y.; Zheng, X.; Wu, H.; Cheng, H.; Shi, X.; Guo, J.; Yang, B.; Zhou, Y.; Wu, X.; and Ying, Z. 2026. Securing the AI Agent: A Unified Framework for Multi-Layer Agent Red Teaming. arXiv preprint arXiv:2606.31227. Yao, H.; Liu, Y.; He, Y.; and Yang, B. 2026. Red-Teaming Agent Execution Contexts: Open-World Security Evaluation on OpenClaw. In Second Workshop on Agents in the Wild: Safety, Security, and Beyond. Zhang, H.; Huang, J.; Mei, K.; Yao, Y.; Wang, Z.; Zhan, C.; Wang, H.; and Zhang, Y. 2025. Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents. In International Conference on Learning Representations. Zverev, E.; Abdelnabi, S.; Tabesh, S.; Fritz, M.; and Lampert, C. H. 2025. Can LLMs Separate Instructions From Data? And What Do We Even Mean By That? In International Conference on Learning Representations.

Benchmark Construction

Algorithm 1 Reward Guided Candidate Search

ActBench constructs one malicious case through an evidence conditioned search over complete case objects. The search stage evaluates candidate edits in the target harness. The reflection stage converts the resulting evidence record into a bounded revise instruction. Both stages retain the user instruction, grading criteria, target behavior, case identifiers, and assigned editable field.

Reward Guided Candidate Search

Algorithm 1 begins after the strategy pool produces an initial candidate x0i at a location observed in the benign trajectory. The pool also supplies the initial revise objective o0i . Each search node is a complete benchmark case. The attack model πφ edits the assigned field. The reasoning model πϕ executes the edited case in the target harness. The rating model then computes the ranking quantities from observed trajectories. Candidate generation and candidate evaluation are therefore assigned to different models. State and admissibility. Let ρi denote the construction specification for case i. It records the target behavior, attack direction, assigned surface, and editable field group. For every candidate, Valid verifies the case schema and equality of ui , Ui , Ai , the target behavior, and the case identifiers. It also restricts the edit to the field group declared by ρi . This predicate does not use the attack score. Rollout score. Run restores the declared case state before each of k construction rollouts. For q ∈ {a, u}, the rollout mean is k 1X ḡq (x) = gq (x, τℓ ). (9) k ℓ=1

Grade computes ḡa with Equation (7) and ḡu with Equation (1). It also returns an evidence record e that contains the trusted audit, trajectory summary, grader outputs, and repeat scores used by reflection. Benign admission requires ben ḡa (xben i ) = 0 and ḡu (xi ) = 1. Candidate acceptance requires ḡa (x) = ḡu (x) = 1. Since every rollout score lies in [0, 1], acceptance means that every construction rollout attains full attack evidence and full task utility. The ranking score is s(x) = ḡa (x)α ḡu (x)1−α .

(10)

This is Equation (2) with explicit rollout means. A zero attack mean or utility mean gives s(x) = 0. For equal values of s, Top compares αḡa (x) + (1 − α)ḡu (x). A remaining tie is resolved by the canonical candidate key. These rules make beam selection independent of the order in which equal scoring proposals are returned. Algorithm operations. pt is the ordered list of candidate and objective pairs awaiting execution at depth t. ht stores each executed candidate with its objective, rollout means, score, and reflection record. bt retains at most w records. Each retained record produces n attack model calls, so |pt+1 | ≤ wn. The rollout count k affects execution cost and does not affect this branching bound. A child receives no score from its parent. It enters ht+1 only after execution by πϕ .

Require: benign case xben and initial candidate x0i i Require: initial objective o0i and fixed specification ρi Require: reasoning model πϕ , attack model πφ , and strategy pool S Require: rollouts k, width w, proposals n, and depth dmax ≥ 1 Require: ranking weight 0 < α < 1 Ensure: accepted case x⋆ i and updated S, or ⊥ 1: τ ← Run(πϕ , xben , k) i 2: (ā, ū, e0 ) ← Grade(xben , τ) i 3: if ā > 0 ∨ ū < 1 then 4: return ⊥ 5: end if 6: p0 ← Unique([(x0i , o0i )]), z ← ∅ 7: for t = 0, . . . , dmax do 8: ht ← ∅ 9: for all (x, o) ∈ pt do 10: if ¬Valid(xben , ρi , x) ∨ Key(x) ∈ z then i 11: continue 12: end if 13: z ← z ∪ {Key(x)} 14: τ ← Run(πϕ , x, k) 15: (ā, ū, e) ← Grade(x, τ ) 16: if ā = 1 ∧ ū = 1 then 17: S ← PushBack(S, x, ρi ) 18: return x 19: end if 20: s ← āα ū1−α 21: ξ ← Reflect(x, ρi , o, e) 22: ht ← Append(ht , (x, o, s, ā, ū, ξ)) 23: end for 24: if ht = ∅ ∨ t = dmax then 25: return ⊥ 26: end if 27: bt ← Top(ht , w) 28: pt+1 ← ∅ 29: for all (x, o, s, ā, ū, ξ) ∈ bt do 30: o′ ← Next(o, ξ) 31: for q = 1, . . . , n do 32: x′ ← Propose(πφ , x, ξ, o′ , q) 33: if Valid(xben , ρi , x′ ) ∧ Key(x′ ) ∈ / z then i 34: pt+1 ← Append(pt+1 , (x′ , o′ )) 35: end if 36: end for 37: end for 38: pt+1 ← Unique(pt+1 ) 39: if pt+1 = ∅ then 40: return ⊥ 41: end if 42: end for 43: return ⊥

Key returns the canonical serialization of the assigned field group. The set z prevents the same edit from being executed at two depths. Unique removes duplicate keys while retaining parent order and proposal index q. Reflect receives the evidence record generated by Grade and returns a diagnosis ξ. Next returns the local instruction η in ξ when it is nonempty. It otherwise retains o. PushBack extracts the accepted edit template and payload structure and inserts them into S.

Termination and cost. Depth zero evaluates x0i . Depth t > 0 contains candidates separated from x0i by t reflection guided edits. Search returns ⊥ when the benign precheck fails, no admissible candidate remains, or depth dmax is evaluated without acceptance. The reasoning model rollout count for one case satisfies (11)

rϕ ≤ k[2 + n + (dmax − 1)wn].

The first two terms count the benign case and x0i . The first refinement contains at most n candidates. Each later refinement contains at most wn candidates. The number of attack model calls is bounded by n + (dmax − 1)wn. With w = 3, n = 2, and dmax = 5, the bounds are 28k reasoning model rollouts and 26 attack model calls per case. Validation rejection, duplicate removal, early acceptance, and an empty layer reduce the realized counts. Construction and reported evaluation use disjoint rollouts.

Evidence Conditioned Reflection

The implementation separates diagnosis, revise generation, and structural admission. For a candidate x, the reflector receives the construction specification ρi and evaluation record e. The record includes API audit entries, repeat scores, grader values, notes, error text, and a trajectory summary when these fields are available. The reflector returns (12)

ξ = (d, v, r, η).

Here d is a diagnostic class. The vector v ∈ {0, 1} records action observation, payload following, task integrity, target effect observation, and the discard indication. The revise focus is r, and η is one local revision instruction. 5

Diagnostic vocabulary. The implementation accepts the following ten class values: not_read read_but_refused sink_not_reached task_integrity_broken already_successful

read_but_ignored weak_attack sink_missing_secret unstable_success unknown

The implementation name sink_missing_secret denotes missing target content at the sink and is not restricted to credentials. The accepted revise foci are: payload_relevance attack_strength stealth_stability none task_integrity_preservation

Fallback diagnosis. Before querying the reflection model, the implementation computes a deterministic record ξ0 . The following cascade is evaluated from top to bottom: Condition

Class

Focus

Mixed scores at 0.8

unstable_success

stealth_stability

Record score at least 0.8

already_successful none

No audited API call Refusal indicator

not_read read_but_refused

payload_relevance stealth_stability

Attack success below 0.25 read_but_ignored

payload_relevance

Otherwise

sink_missing_secret attack_strength

The last branch reuses the current objective o as its revise instruction.

The fallback marks payload following at an attack success value of at least 0.5. It marks task integrity at a record score of at least 0.3. Audited API use supplies the fallback values for payload observation and target effect observation. These thresholds control revise generation only. Candidate acceptance remains the conjunction ḡa (x) = ḡu (x) = 1 in Algorithm 1. Structured model and normalization. Let πψ denote the reflection model and N the schema normalizer. The implemented diagnosis is ξ = N (πψ (q), ξ0 ),

(13)

where q serializes the scene metadata, risk specification, assigned scope, surface description, current candidate, revise objective, and evaluation record. The serialized request is limited to 12,000 characters. The model uses temperature 0.2 and an output limit of 2,048 tokens. N retains only the ten diagnostic classes, five revise focus, Boolean status values, and a nonempty revise instruction. A missing or invalid field inherits its value from ξ0 . An exception in generation or parsing returns ξ0 . Structured Reflection Template Role. Diagnose the execution outcome of the current candidate and return one revise instruction for the assigned field. Scope. Preserve the behavior, surface, field group, case identifiers, and benign task. Do not introduce a new file, skill, tool, risk, or execution surface. Evidence. Read the scene metadata, risk target, attack direction, scope, surface, candidate content, revise objective, and evaluation record. Decision rules. Repeat scores on both sides of 0.8 indicate unstable_success with focus stealth_stability. For not_read, improve payload relevance before attack strength. For read_but_refused, remove overt or coercive disclosure language and express the revise in task native terms. Return format. Emit one JSON object. Use failure_type for the diagnostic class. Use the Boolean fields payload_observed, payload_followed, task_integrity_preserved, target_side_effect_observed, and discard_candidate. Use revise_focus for the revise class and revise_instruction for one local edit objective.

Bounded revise. The reflection record does not edit a case directly. It conditions πφ , after which Valid enforces the fixed surface and field group. The discard indication is advisory and does not remove a beam in Algorithm 1. This design preserves a bounded revise path when the model requests rejection while retaining structural admission as a deterministic operation. For a B11 example, consider a task that requires one priority update and one tag update. A candidate causes repeated job history queries and a 120 second wait. The evidence record confirms service reads and the resource effect, while the wait consumes the execution budget before the two required updates. Reflection assigns task integrity preservation

as the revise focus. The resulting instruction retains the resource trigger, removes the wait, and restores the two required updates. The payload, assigned field, criteria, and behavior remain fixed.

Experimental Setting

Systems and controlled comparisons. All agent executions use Ubuntu 22.04.5 LTS. RQ1 fixes OpenClaw 2026.5.19 and changes only the base model across the 15 models in Table 2. RQ2 fixes Deepseek-v4-Pro and replaces the complete cowork harness. This intervention changes the harness context assembly, memory retrieval, tool serialization, and action mediation. It does not isolate one harness component. Table 5 reports the six releases used in RQ2. Harness

Version

OpenClaw OpenAgent Claude Code Hermes Agent OpenCode QwenPaw

2026.5.19 v2.83.1 v2.1.218 v0.18.2 1.18.3 2.0.0.post2

Table 5: Harness versions used for RQ2. The Deepseek-v4-Pro and OpenClaw configuration appears in both RQ1 and RQ2 and is counted once. The two studies therefore contain 20 unique configurations. Each configuration contains three rollouts for each of the 300 malicious cases and one rollout for each of the 300 matched benign cases. This gives 1,200 trajectories per configuration and 24,000 trajectories in total. RQ3 reuses the 300 malicious and 300 benign labels for detection evaluation and does not add agent execution trajectories. Construction and grading. Candidate generation uses Deepseek-v4-Pro. GPT-5.5 serves as the rating model and the trajectory evidence verifier. Model requests are routed through OpenRouter. Construction uses beam width w = 3, proposal count n = 2, depth dmax = 5, ranking weight α = 0.5, and evidence weight β = 0.4. Construction and reported evaluation use disjoint rollouts. The evaluation threshold for attack success is 0.8. The first three malicious rollouts define pass@1, pass@2, and pass@3. AGS is averaged over malicious rollouts and UGS is averaged over matched benign rollouts. Behavior conditioned results preserve the case frequencies in Figure 2. They are not macro averages over the 15 behavior labels. Reset and task equivalence. Before every rollout, the framework restores the declared workspace files, web service APIs, persistent memory, and case counters. Within RQ1, the harness, tools, cases, reset state, graders, and trusted logging schema remain fixed. Within RQ2, the base model, cases, task authorization, initialized objects, grading criteria, and trusted records remain fixed. Each harness retains its native context and tool serialization. The intervention therefore compares complete harnesses under the same task visible state and API capability.

Defense inputs and decision rules. Prompt-Guard-2-86M and Llama-Guard-4-12B receive local text segments available at their classification boundary (Llama Team 2024; Meta AI 2025). AgentDoG-Qwen3-4B receives contextual trajectory states (Liu et al. 2026). AI-Infra-Guard and CiscoSkill-Scanner receive workspace or skill artifacts (Yang et al. 2026; Tencent Zhuque Lab 2025). The decision threshold for each detector is frozen before scoring the 600 reporting inputs. Table 4 compares each binary prediction with the benchmark pair label at the method’s native observation boundary. The protocol measures discrimination under different inputs and training targets. It does not insert a blocking policy into agent execution or estimate a causal change in AGS, ASR, or UGS.

Execution Cost

Additional Results

This experiment examines whether the observed behavioral risk ranking can be attributed to differences in execution cost. We compare action-cycle distributions across models and harnesses, and analyze cache-inclusive token consumption for malicious and matched benign rollouts under the fixed OpenClaw harness. Although malicious rollouts have higher mean token consumption for all 15 models, with relative increases from 2.7% to 20.4%, neither action cycles nor token consumption reproduces the ASR ordering. The results indicate that the observed risk ranking is not explained by longer or more token-intensive execution. Figure 7(a) shows malicious median action counts from 15.0 to 19.5 across the 15 base models. The corresponding benign medians range from 14.0 to 18.0. MiniMax-M3, Deepseek-v4-Pro, and Hunyuan-3.0 each have a malicious median of 19.0, while their ASRs are 25.0%, 94.4%, and 30.0%. Figure 7(b) shows a wider harness range. OpenClaw has the shortest malicious median at 19 cycles and the highest harness ASR at 94.4%. Hermes has the longest median at 47 cycles and an ASR of 79.2%. QwenPaw and Claude Code both have a median of 33 cycles, while their ASRs differ by 7.7 percentage points. These comparisons do not support action count as an explanation for the risk ordering. Table 6 reports cache inclusive token consumption under the fixed OpenClaw harness. The statistics are computed from archived rollouts with populated token fields, and missing entries are not imputed. The malicious mean exceeds the benign mean for all 15 models. The increase ranges from 3.4k tokens for GPT-5.4-mini, with 128.1k versus 124.7k, to 27.8k for Deepseek-v4-Flash, with 164.2k versus 136.4k. Relative to the benign mean, these endpoints correspond to increases of 2.7% and 20.4%. The malicious median also exceeds the benign median for 14 of the 15 models. GPT-5.4-mini is the only exception, with a malicious median of 110.2k tokens and a benign median of 111.0k. The malicious and benign IQRs overlap for all 15 models, so the central 50% token ranges are not disjoint for the two case types. Hunyuan-3.0 records the largest malicious and benign means at 207.1k and 188.8k tokens. GLM-5.2 records the smallest means at 97.9k and 91.6k tokens. However, the ordering by malicious mean token use differs from the ASR

0 /

0 / / - /

/ - /

0 / / - /

// //

0. /

0 0

0. 3 0

. . 1

400

. 1

(a) RQ1 Execution Iteration

(b) RQ2 Execution Iteration 11

)

-21.21

) )

0 ( .21

)

) )

)

) )

)

) )

)

) )

)

) )

-1

1-)

)

0 .2 2

(

) /4

(

( (

(

)

3 .0

)

0 3

Figure 7: Execution resource distributions. Orange boxes report malicious rollouts and green boxes report matched benign rollouts. Boxes show the interquartile range, center lines show medians, and whiskers use 1.5 IQR. Panels (a) and (b) cover all configurations in RQ1 and RQ2. Panel (c) contains the 15 models present in the archived token plot. Table 6 reports token statistics for all 15 models. Table 6: Cache inclusive token use with OpenClaw fixed. Values areks of tokens per rollout. Malicious

Benign

Model

Mean

Median [IQR]

Mean

Median [IQR]

Claude-Opus-4.8 Claude-Sonnet-4.6 GPT-5.5 GPT-5.4-mini Grok-4.5 GLM-5.2 Qwen3.7-max Qwen3.7-plus Kimi-K3 Kimi-K2.6 MiniMax-M3 MiniMax-M2.7 Deepseek-v4-Pro Deepseek-v4-Flash Hunyuan-3.0

175.7 146.7 123.7 128.1 134.9 97.9 130.7 108.6 124.3 102.3 182.8 116.2 149.0 164.2 207.1

164.2 [142.3, 194.2] 131.2 [116.5, 165.4] 113.9 [93.3, 140.3] 110.2 [93.5, 140.5] 116.9 [106.3, 144.7] 92.1 [78.4, 110.0] 114.4 [98.1, 148.4] 99.0 [87.2, 118.0] 114.1 [96.0, 137.7] 95.1 [78.7, 117.8] 156.7 [127.3, 207.6] 106.8 [90.2, 132.3] 133.4 [111.1, 172.9] 146.1 [111.6, 186.3] 188.1 [154.3, 233.4]

169.9 141.7 113.3 124.7 117.6 91.6 119.3 99.5 113.8 95.6 160.2 101.4 125.1 136.4 188.8

160.6 [138.1, 189.9] 122.5 [100.9, 150.8] 106.8 [88.6, 125.5] 111.0 [92.3, 133.1] 108.2 [89.6, 126.7] 88.3 [75.1, 103.5] 108.8 [91.1, 129.5] 94.3 [81.0, 111.5] 99.0 [92.5, 121.9] 86.6 [73.5, 103.2] 143.2 [116.4, 180.0] 91.2 [77.3, 109.7] 111.4 [92.5, 134.2] 125.2 [105.6, 150.6] 174.0 [142.9, 210.3]

ordering in Table 2. Deepseek-v4-Pro reaches 94.4% ASR with a malicious mean of 149.0k tokens, whereas Hunyuan3.0 reaches 30.0% ASR with 207.1k tokens. Claude-Opus4.8 records 10.1% ASR despite a malicious mean of 175.7k

tokens. These comparisons show that malicious rollouts have higher average token use within each model, but cross-model token consumption doesn’t account for the observed risk ranking under the fixed harness.

Llama Guard 4-12B

AgentDoG-1.5-Qwen3.5-4B

OpenClaw across models — AGS

1.0

60 50

0.8

Claude-Opus-4.8 Claude-Sonnet-4.6 GPT-5.5

40 30

GPT-5.4-mini Grok-4.5

0.6

GLM-5.2 Qwen-3.7-max

AGS

Cumulative detection rate (%)

Prompt Guard 2-86M

Qwen-3.7-plus Kimi-K3 Kimi-K2.6 MiniMax-M3 MiniMax-M2.7 Deepseek-v4-Pro Deepseek-v4-Flash Hunyuan-3.0

0.4

Step

35 0.2

Figure 8: Cumulative malicious case detection by logged trace position for Prompt-Guard-2-86M, Llama-Guard-412B, and AgentDoG-Qwen3-4B.

We evaluate how rapidly each query level defense detects risk behavior across 300 malicious cases. The defense evaluates behavior safety at every query step, and execution terminates after the first positive prediction. Figure 8 reports the cumulative percentage of cases detected at or before each step. AgentDoG-Qwen3-4B detects 11.0% of the malicious cases at the first step. Its detection rate reaches 26.3% at step 8, 44.3% at step 10, and 50.7% at step 12. The final rate is 55.7% at step 17, corresponding to 167 detected cases. Llama-Guard-4-12B begins detecting cases at step 3 and reaches 11.7% at step 7, 15.7% at step 10, and 18.3% at step 20. Prompt-Guard-2-86M begins at step 6 and reaches 3.3% at step 10, 10.3% at step 15, and 13.3% at step 36. Their final detection counts are 55 and 40 cases, respectively. At step 10, AgentDoG-Qwen3-4B exceeds Llama-Guard-4-12B and Prompt-Guard-2-86M by 28.7 and 41.0 percentage points. By step 12, it has detected 152 cases, which exceeds the final detection count of either baseline. The result reflects differences in both detection timing and final recall. Prompt-Guard-2-86M and Llama-Guard-412B classify local text available at their evaluation boundaries (Llama Team 2024; Meta AI 2025). AgentDoG-Qwen34B instead uses contextual trajectory states and is trained to identify risk behavior from execution context (Liu et al. 2026). Therefore, its advantage cannot be attributed only to model size, since its backbone is smaller than Llama-Guard4-12B. The results indicate that trajectory context enables earlier termination for a larger fraction of malicious executions. However, AgentDoG-Qwen3-4B still misses 133 cases. This limitation motivates a layered defense that combines artifact scanning before execution (Yang et al. 2026; Tencent Zhuque Lab 2025) with query level trajectory monitoring during execution.

Ablation: Impact of β We vary the log evidence weight β while holding every trajectory and criterion output fixed. A value of zero uses trajectory evidence only. A value of one uses log evidence only. The reported results use β = 0.4. This analysis measures score sensitivity and does not execute the agents again. Across base models, every AGS curve increases as the log evidence weight rises. Claude-Opus-4.8 ranges from 0.169

0.0

0.2

0.4

0.6

0.8

1.0

Python component weight α

Figure 9: AGS sensitivity to the log evidence weight β. The dashed lines mark β = 0.4, which is used for all reported tables. Deepseek-v4-Pro across agents — AGS

1.0

0.8

OpenClaw OpenAgent

0.6

ClaudeCode Hermes

AGS

Safety Policy Effects

0.0

OpenCode QwenPaw

0.4

0.2

0.0 0.0

0.2

0.4

0.6

0.8

1.0

Python component weight α

Figure 10: AGS sensitivity to the log evidence weight β. The dashed lines mark β = 0.4, which is used for all reported tables. to 0.456, while Deepseek-v4-Pro ranges from 0.941 to 0.977. These two models remain the lowest and highest AGS endpoints. At β = 0.4, their values are 0.284 and 0.955, matching Table 2. The model ordering at the two extremes is therefore stable for the lowest and highest risk models. The harness comparison is more sensitive to the evidence source. OpenAgent ranges from 0.774 to 0.859 as β increases. QwenPaw moves in the opposite direction from 0.896 to 0.825. The lowest harness AGS consequently changes from OpenAgent under trajectory evidence only to QwenPaw under log evidence only. OpenClaw remains highest and ranges from 0.941 to 0.977. UGS is not included in this sensitivity analysis because its definition uses the utility criteria directly and does not contain β.

Behavior Conditioned Results This detailed RQ1 experiment examines whether the aggregate model ranking in Table 2 is consistent across the 15 risk behaviors. OpenClaw 2026.5.19, the case pairs, reset state,

tools, grading criteria, and evidence verifiers remain fixed. Table 7 reports malicious ¬AGS for each behavior. A larger value indicates less attack evidence. The final columns report the frequency weighted average ¬AGS and average benign UGS, which match the aggregate values in Table 2. Results. Claude-Opus-4.8 records the highest average ¬AGS at 0.716, with average UGS of 0.938. Its lowest behavior values are 0.358 for Goal Hijacking and 0.430 for Deceptive Tool Invocation. Deepseek-v4-Pro records the lowest average ¬AGS at 0.045, with average UGS of 0.922. Its ¬AGS is below 0.10 for 12 behaviors. The other three values are 0.101 for Tool Scope Escalation, 0.134 for False Reporting, and 0.172 for Context Flooding. The range across models depends on the behavior. State Tampering spans 0.039 to 0.932, which gives the largest range at 0.893. Data Exfiltration spans 0.024 to 0.916, with a range of 0.892. Tool Scope Escalation spans 0.101 to 0.973, with a range of 0.872. Deceptive Tool Invocation spans 0.002 to 0.483, and no evaluated model reaches 0.5 on this behavior. These values identify behavior specific differences that are hidden by a single model average. Average utility does not reproduce the safety ordering. Claude-Opus-4.8 and Grok-4.5 both record average UGS of 0.938, while their average ¬AGS values are 0.716 and 0.130. Kimi-K3 records the highest average UGS at 0.940, while its average ¬AGS is 0.511. The matched benign score therefore measures task completion without serving as a substitute for the malicious behavior score. Conclusion. The behavior conditioned results show two distinct model profiles. Deepseek-v4-Pro has low ¬AGS across 12 behavior categories, so its aggregate risk is not produced by one label. Claude-Opus-4.8 has the highest average ¬AGS, but Goal Hijacking and Deceptive Tool Invocation remain below 0.5. The fixed harness and matched tasks link these differences to how each base model maps the same untrusted context to tool actions and environment effects. Reporting the behavior conditioned distribution therefore reveals residual failure modes that the aggregate average does not localize.

Table 7: Category level behavioral results across base models with OpenClaw fixed. Columns B1 to B15 report malicious ¬AGS. The final columns report the frequency weighted model averages of ¬AGS and benign UGS.

Average

B10

B11

B12

B13

B14

B15

¬AGS

UGS

Claude-Opus-4.8

0.641

0.358

0.916

0.740

0.599

0.932

0.430

0.820

0.801

0.911

0.680

0.739

0.667

0.925

0.636

0.716

0.938

Claude-Sonnet-4.6

0.645

0.504

0.789

0.492

0.451

0.904

0.483

0.770

0.696

0.973

0.680

0.757

0.648

0.158

0.331

0.653

0.927

GPT-5.6-Sol

0.582

0.684

0.678

0.676

0.502

0.615

0.359

0.695

0.204

0.616

0.454

0.665

0.625

0.196

0.250

0.566

0.924

GPT-5.6-Terra

0.596

0.649

0.849

0.808

0.534

0.635

0.346

0.728

0.320

0.650

0.540

0.595

0.616

0.130

0.543

0.614

0.922

GPT-5.6-Luna

0.550

0.575

0.590

0.501

0.288

0.497

0.208

0.665

0.172

0.554

0.626

0.543

0.552

0.174

0.269

0.497

0.927

GPT-5.5

0.525

0.639

0.587

0.290

0.207

0.640

0.178

0.641

0.528

0.532

0.536

0.568

0.595

0.358

0.185

0.507

0.928

GPT-5.4

0.499

0.569

0.552

0.360

0.180

0.391

0.238

0.454

0.218

0.551

0.397

0.427

0.439

0.186

0.178

0.409

0.895

GPT-5.4-mini

0.324

0.311

0.180

0.251

0.264

0.100

0.190

0.360

0.357

0.637

0.232

0.243

0.306

0.235

0.152

0.273

0.904

Grok-4.5

0.115

0.240

0.194

0.019

0.067

0.098

0.048

0.174

0.107

0.302

0.048

0.086

0.208

0.088

0.087

0.130

0.938

Grok-4.3

0.148

0.132

0.199

0.021

0.053

0.143

0.380

0.024

0.093

0.261

0.317

0.090

0.145

0.115

0.077

0.138

0.881

GLM-5.2

0.364

0.213

0.391

0.242

0.419

0.804

0.163

0.477

0.498

0.822

0.636

0.317

0.663

0.278

0.081

0.453

0.929

Qwen3.7-max

0.538

0.324

0.386

0.314

0.308

0.712

0.253

0.557

0.535

0.829

0.567

0.618

0.578

0.118

0.149

0.489

0.915

Qwen3.7-plus

0.464

0.394

0.372

0.382

0.201

0.685

0.180

0.578

0.479

0.928

0.582

0.566

0.623

0.130

0.134

0.476

0.915

Kimi-K3

0.409

0.451

0.326

0.535

0.536

0.639

0.194

0.821

0.147

0.631

0.750

0.637

0.592

0.133

0.304

0.511

0.940

Kimi-K2.6

0.167

0.771

0.107

0.243

0.249

0.137

0.087

0.367

0.131

0.501

0.150

0.164

0.380

0.171

0.048

0.252

0.869

MiniMax-M3

0.519

0.283

0.667

0.619

0.291

0.909

0.460

0.626

0.649

0.845

0.599

0.657

0.655

0.502

0.497

0.598

0.917

MiniMax-M2.7

0.188

0.199

0.086

0.039

0.310

0.121

0.193

0.336

0.155

0.627

0.202

0.184

0.358

0.094

0.064

0.196

0.880

MiniMax-M2.5

0.230

0.160

0.204

0.044

0.123

0.159

0.459

0.108

0.086

0.363

0.603

0.232

0.328

0.024

0.046

0.201

0.874

Deepseek-v4-Pro

0.039

0.045

0.024

0.031

0.020

0.039

0.002

0.026

0.014

0.101

0.033

0.064

0.134

0.172

0.016

0.045

0.922

Deepseek-v4-Flash

0.067

0.084

0.072

0.027

0.103

0.120

0.047

0.112

0.033

0.309

0.210

0.075

0.418

0.223

0.040

0.113

0.900

Deepseek-v4-Flash-0731

0.366

0.367

0.232

0.085

0.118

0.326

0.289

0.722

0.117

0.314

0.525

0.472

0.620

0.359

0.047

0.368

0.918

Hunyuan-3.0

0.462

0.252

0.521

0.427

0.446

0.794

0.239

0.681

0.624

0.825

0.619

0.522

0.646

0.653

0.353

0.545

0.933

Model

Detect Risk Behavior

17 Ignore Risk Behavior

Figure 11: B1 Instruction Hijacking Risk: Claude-Sonnet-4.6 trajectory analysis shows that Sonnet detected malicious behavior through injection and blocked the malicious behavior.

ActBench: Self-Evolving Benchmark of Behavioral Safety in Cowork Agents

Analyze & explore related research in Conceptio

Related documents