arXiv CSopen access

Set-shifting Behavioral Test for Harnessed Agents

Preprint

Set-shifting Behavioral Test for Harnessed Agents Ye Ziwei [email protected]

arXiv:2607.13396v1 [cs.AI] 15 Jul 2026

Abstract What happens to an LLM agent’s tool choice when the reliable tool silently changes within an ongoing session? We borrow set-shifting from cognitive psychology to study how well agents adapt to hidden reliability shifts. Our benchmark mounts tool-skill libraries with redundancies, where many tools solve the same task but differ in hidden reliability. In our evaluation framework, a branched schedule shifts the reliable tool group at hidden boundaries and pairs every shift with a no-shift control. We find that agents, by default, settle on a small recurring routine within a few turns of each boundary, with call shares concentrating on a few discrete values after each reliability shift. We score the set-shifting accuracy for each agent trajectory: the joint probability of routing to the target tool group in every post-shift window. We test open-weight LLMs in an open-source agentic harness and find qualitatively distinct failure modes across the same set of routines. We also find that set framing, how the toolset presents the alternatives as competing or complementary, shifts the routing dynamics.

1

Introduction

Humans who repeatedly solve similar problems form habits that simplify future decisions but can persist past their usefulness. The Einstellung effect and the Wisconsin Card Sorting Test (WCST) measure this rigidity, called perseveration: the first shows people sticking with a learned rule when a simpler one would suffice (Luchins, 1942), and the second silently changes the sorting rule mid-task (Grant & Berg, 1948; Heaton et al., 1993). Reversal learning paradigms apply the same kind of hidden change to value-based choice (Izquierdo et al., 2017). Recent work ports these cognitive tests to LLMs in textual or visual sorting tasks (Li et al., 2024; Hao et al., 2025; Li et al., 2025). Modern LLM agents reside in agentic harnesses in which each call arrives with prior tool calls and results, skills and tool schemas, and system prompts already in context, so the agent never starts from a blank state. In such a rich environment, a mounted library of skills and tools may contain several nominally equivalent capabilities calling differing APIs and backend services, and repeated tool use lets the agent converge on a preferred subset. That preference can outlive an unannounced backend change, leaving the agent calling tools that are no longer the reliable ones. A set-shifting test in this scenario would then ask whether a harnessed agent can update its action route when the reliable backend behind an apparently equivalent tool group changes silently. We make the following contributions: (i) A benchmark for harnessed agents based on redundant tool-skill sets, libraries of many functionally equivalent tools where the working subset silently shifts mid-trajectory and each shift is paired with a no-shift control.1 (ii) A descriptive framework that characterizes post-shift behavior as recurrence on a small set of discrete routines. (iii) A case study on two open-weight LLMs testing two intervention channels, policy prompts (instructional) and set framing (structural). 1 https://github.com/zwycl/wcst-tool-bench

1

Preprint

group A

group B

group C

z

Â

Û

|

Ý

ß



7

Z

~

Á



}

Î

¶

#

k ©

skills

1

2

3

@

@

@ aux

context + request

R∗

2

Benchmark

2.1

Tool-Skill Sets and Taxonomy

Â Ý 7 Á Î ÛßZ¶

context + request

R2

z|~}

Â Ý 7 Á Î ÛßZ¶

context + request

*

(a) Skill-tool structure. Each SKILL.md (left) describes one tool group, and each group contains functionally redundant variants. Boundaries silently change which group is reliable.

z|~}

R1

z|~}

Â Ý 7 Á Î ÛßZ¶

(b) Three consecutive turns after an A → B shift. Each turn receives a new context+request and emits a tool-call route. Solid icons = called, faded = unused. The turns shown are route class R∗ (no target), R2 (target plus one other group), and R1 (target only). R3 (target plus both other groups) is omitted.

A tool is a callable function the harness exposes to the agent, defined by a unique name, a natural language description, and a typed parameter schema. Each call returns either a structured success payload or a failure indicator. A tool group is a set of functionally equivalent tools with the same schema and backend operation but different names and descriptions. A skill is a markdown procedure paired with one tool group that enumerates the group’s tools. A skill tells the agent when and how to invoke tools, and its tool list is how the agent observes group membership. Each domain has a finite family of tool groups C = {C (1) , . . . , C ( M) }. An action route at user turn t is the ordered sequence of tool calls emitted during that turn. A redundant tool-skill set mounts these skills and tools so that entries for the same task family differ in description and hidden reliability. The textual descriptions for skills, tools, and tasks stay neutral throughout. All user prompts use a single neutral template family, varying only persons, time window, and duration from one turn to the next. The prompt carries no lexical cue toward any tool group, so the agent must discover the active tool group purely from binary tool feedback. Different paraphrases of the same prompt let us see that the agent’s tool group choice is not just tracking template wording. 2.2

Domains and Dataset Construction

We instantiate the simulation in three domains: scheduling and coordination (3 groups × 5 tools), DevOps incident triage (4 groups × 5 tools), and multi-cloud storage (5 groups × 2 tools). The three domains also differ in how the groups relate to one another, a dimension we call set framing, encoded in the tool and skill descriptions of each domain. Scheduling presents the groups as competing slot providers where any one group serves the user’s request and the others are alternatives. DevOps presents the groups as complementary observability stacks that together describe one service. Multi-cloud presents the groups as competing storage providers behind a path interface. Each group shares common wording and within a domain all primary tools share one function schema and one success payload structure, so the only differences exposed to the agent are tool name and a short description. The DevOps and multi-cloud domains also ship matched complementary and competing framing variants of the same toolset, used in §4.4. All tools are simulated, with actions that have side effects implemented as reversible mocks, and each mounted set instantiates every relation from Table 4. 2

Preprint

Each domain mounts one skill per tool group, with the skills loaded simultaneously. Names within a group draw from a single theme vocabulary, across groups the naming is parallel so no group has a lexical advantage, and the tools instantiate three redundant relations: description variant (same schema and execution, different wording), quality gradient (same visible intent, different hidden reliability across phases), and side effect conflict (mutating actions). Each domain also includes a support tool group and tools that produce side effects, which are forbidden when the user asks only for a draft or diagnosis. We draft the dataset via LLM annotation and audit it manually. To support the shift schedules we designate a reliable tool group per phase in each domain, with matched descriptions and identical schemas across all tool groups so the interface the agent sees is invariant across boundaries. The full construction protocol, including the audit and the verifier, is in Appendix B.

3

Study Design and Evaluation

3.1

Harness and Environment

We use Hermes Agent (Nous Research, 2026), an open-source harness with persistent session context and a mounted skill library, paired with two contemporary open-weight models served via OpenRouter: mimo-v2.5 (LLM-Core Xiaomi, 2026) and deepseek-v4-pro (DeepSeek-AI, 2026). The agent runs under the Hermes default system prompt with no policy overlay, so the regular assistant condition reflects the assistant identity the harness ships with. We request temperature 0.7, top-p 0.95, and medium reasoning effort through OpenRouter. All other settings are the Hermes Agent defaults. The simulated environment is piecewise stationary across phases, only changing at the predetermined shift boundaries. A deterministic verifier maps each emitted tool call to either a structured success payload or a bare failure indicator with no diagnostic message, matching the binary feedback used in WCST and reversal learning paradigms (Grant & Berg, 1948; Heaton et al., 1993; Luchins, 1942; Izquierdo et al., 2017). Each turn uses a single neutral prompt template, and the agent’s route at each turn is one observation regardless of how many tool calls it contains. 3.2

Study Design

Schedule and tree. At each shift the previously reliable tool group becomes unreliable while names, descriptions, and schemas of all tool groups remain unchanged and nothing in the harness announces the change. With M tool groups and K branching layers, the schedule is a tree (Figure 2b). A trajectory begins with a shared prefix: a common root identified by a seed that fixes the display order of the tools and the permutation within each group. To randomize position effects across replicates, we rotate the tool group display order cyclically across the n prefixes and permute the tool order within each tool group by the prefix seed. Both orderings are held fixed across all branches resumed from a given prefix. At every internal node the schedule branches into three siblings: a reversal back to a previously used tool group, a novel shift to a previously unused tool group, and a no-shift control where the parent tool group stays reliable. We label each endpoint by its phase sequence. For each prefix we save the root’s Hermes session state and resume it into every child recursively. Sibling sets give contrasts paired by parent state, and cross branch contrasts use the nearest common ancestor as the pairing factor. Research questions.

In this study, we ask the following:

1. When the reliable tool group shifts, does the agent commit to the new tool group the same way regardless of its prior-use role, or does each prior-use role produce a distinct pattern? 2. Does the depth of an endpoint, or the path taken to reach it, matter beyond the most recent shift? 3

Preprint

shift at boundary k phase 1: tool group B reliable

phase 2: tool group C reliable

B

C pre-shift plateau

1 Cell:

20

10 clean commit

mixed

post-shift window

30

40

50

off target

(a) Behavior across a single boundary. Top bar: reliable tool group across phases. Strip below: routeclass chosen each turn. tier 0

tier 2

tier 1

ABCA C→A

ABC

C→B

ABCB

C→C B→C

ABCC AB prefix ABAC B→A A→C B→B

ABA

A→B

ABAB

A→A

ABAA ABB no-shift control

(b) Branched schedule tree. The shared prefix branches into three tier-1 segments (two shifts plus a no-shift control), and each shift segment branches again into three tier-2 endpoints. Node color shows the endpoint’s target tool group. Dashed borders mark no-shift controls. Edge labels show the tool group transition at that boundary.

Figure 2: Set-shifting paradigm. (a) Per turn behavior at a single boundary. (b) Branched schedule tree that instantiates (a) at every endpoint. 3. Can policy prompts change the route-class distribution? 4. Does set framing shift which routine the agent settles into? RQ3 and RQ4 test two parallel interventions: instructional (the policy prompt) and structural (the set framing). We instantiate with M = 3 tool groups and K = 2 branching layers within 90 turns, giving three boundaries: b1 at turn 15 inside the prefix (A → B), b2 at turn 40 entering a tier-1 child, and b3 at turn 65 entering a tier-2 child. Each tool group first becomes reliable at a different point: A from turn 1, B from turn 16, and C only at the tier-1 boundary (turn 41) on novel shift endpoints, and never on no-shift branches. The tree has nine endpoints in total. The schedule assigns each target a distinct prior use role: C A as a reversal target (reliable early in the prefix, then turned off), C B as the reinforced target (reliable through the end of the prefix), and CC as a novel target (never reliable in the prefix). Differences across C A , C B , and CC in the reported results reflect these prior use roles. Across the three domains, the main branched experiments in §§4.1–4.3 use the scheduling domain with n = 16 prefixes per cell, and §4.4 reports additional linear path runs on the DevOps and multi-cloud domains. 3.3

Evaluation metrics

To answer the four research questions of §3.2 we define a small set of metrics over trajectory observations. The set-shifting accuracy Φ is bounded in [0, 1] and equals the probability that 4

turn

Preprint

the agent’s route is on the correct tool group at every post-shift window along the path from root to endpoint. Alongside Φ we report route-class rates at each turn that expose the failure mode at each shift, and a task completion rate F, the proportion of turns the verifier judges as task-completing. A turn counts toward F when the agent calls at least one tool from the reliable group, uses every required support tool group, and invokes no forbidden side-effect tools. A turn that mixes the reliable group with unreliable ones still counts as task-completing, since the reliable group did the work. Set-shifting accuracy. Intuitively, the set-shifting accuracy asks: if we sample one tool call uniformly from each post-shift window, what is the probability that every sampled call lands in its target tool group? Independent sampling across windows makes that joint probability the product of the target share in each post-shift window. Formally, for an endpoint reached through boundaries b1 , . . . , bK on its path from root, the reliable tool group after boundary bk is Ck ∈ C and the agent’s post-shift window Pk contains the route calls that follow it. Let sk be the share of route calls in Pk that hit Ck , and a turn calling both Ck and another group contributes to sk in proportion to the share of its calls that hit Ck . Then K

Φ = ∏ sk .

(1)

k =1

We compute the set-shifting accuracy, Φ, per trajectory as the path product of its per-window target shares and average across replicates. Φ assumes independent sampling within each post-shift window. Correlations across boundaries, such as the prefix-level lock-in we report in §4.2, inflate variance but do not bias the mean. Route classes. Each turn is classified by the set of tool groups its calls touch, giving four mutually exclusive classes whose rates over a window sum to one: R1 for target only, R2 for target plus one other group, R3 for target plus both other groups, and R∗ for no target. With one tool call per group involved, these correspond to s ∈ {1, 0.5, 0.33, 0}. The R-classes are turn-level labels, while sk is their window-level average through the mapping above. At no-shift controls the split is not a failure taxonomy because the old and target groups coincide. Uncertainty. Every reported Φ value is accompanied by a non-parametric bootstrap 95% interval on the cell mean, shown as a half-width on the resampled means. Count proportions carry Wilson 95% intervals. We report a cell-to-cell difference as supported when the two 95% CIs are disjoint. For paired sibling contrasts within the branched tree, we additionally report paired permutation tests on ∆Φ.

4

Results and Analysis

4.1

Each prior-use role produces its own commitment pattern (RQ1)

Each target enters its post-shift window with a different prior use. Mean Φ at each target (Table 1) ranks differently across models: mimo ranks C A > C B > CC with low variance, struggling most at the never seen target, while deepseek-v4-pro ranks C B > C A ≈ CC with uniformly high variance across all targets. Post-shift target shares concentrate on a few discrete values. We observe three behaviors after each shift on both deepseek-v4-pro and mimo-v2.5. (i) Trajectories lock: within a few turns the agent stops varying its tool calls and starts repeating one fixed routine. (ii) Across trajectories, the share of calls landing in the target tool group (s) concentrates on the discrete values introduced in §3.3. (iii) Which value a trajectory lands on depends on the target’s prior use and the model. Each value represents a routine that is an attractor in the connectionist sense (Botvinick & Plaut, 2004): a stable state that the system relaxes back to, so once the agent settles on a routine, single-turn deviations do not break the pattern. The per-cell distribution concentrates on these routines, with prior use predicting which 5

Preprint

xiaomi/mimo-v2.5 Endpoint Target R1 R2 R3 R∗

F

deepseek/deepseek-v4-pro Φ

sk

R1 R2 R3 R ∗

F

Φ

sk

Tier 1 (sk at b2 ) ABA CA ABB CB ABC CC

.62 .34 .04 .00 .98 .80 .33 ± .07 [9/0] .56 .11 .02 .31 .69 .53 .41 ± .15 [6/4] .20 .63 .16 .00 .99 .44 .26 ± .16 [2/1] .90 .02 .00 .08 .92 .86 .82 ± .16 [11/1] .05 .04 .42 .48 .50 .17 .07 ± .04 [0/7] .57 .03 .08 .32 .68 .40 .38 ± .17 [5/4]

Tier 2 (sk at b3 ) ABAA CA ABAB CB ABAC CC ABCA CA ABCB CB ABCC CC

.66 .32 .02 .00 1.00 .80 .26 ± .06 [9/0] .63 .04 .02 .31 .69 .68 .40 ± .16 [8/3] .19 .60 .21 .00 .99 .43 .17 ± .08 [2/1] .75 .08 .02 .15 .85 .72 .34 ± .14 [10/1] .08 .02 .50 .40 .59 .20 .06 ± .03 [0/7] .38 .02 .06 .54 .46 .27 .17 ± .12 [2/6] .65 .22 .13 .00 .98 .78 .04 ± .02 [7/0] .59 .05 .03 .33 .67 .52 .24 ± .13 [6/3] .26 .54 .20 .00 .98 .42 .03 ± .03 [2/1] .70 .06 .01 .23 .75 .66 .30 ± .16 [9/2] .05 .11 .44 .40 .58 .20 .02 ± .02 [0/6] .64 .01 .04 .31 .69 .60 .37 ± .18 [8/3]

Table 1: route-class rates R, task completion rate F, target share sk at the final window, and cumulative set-shifting accuracy Φ (n = 16). Φ shown as mean ± bootstrap 95% CI half-width. Brackets are [# of traj. at s ≥ 0.8 / # at ≤ 0.2]. CI conventions in §3.3. mimo

deepseek ABAA

ABCA

ABB

ABAB

ABCB

ABC

ABAC

ABCC

ABA

density (C A)

10 8 6 4 2 0

density (C B)

10 8 6 4 2 0

density (C C)

10 8 6 4 2 0

0.0

0.2

0.4

starget

0.6

0.8

1.0

0.0

0.2

0.4

starget

0.6

0.8

1.0

0.0

0.2

0.4

starget

0.6

0.8

1.0

Figure 3: Distribution of s across prefixes at each endpoint, n = 16 per cell. mimo: solid bars and KDE. deepseek-v4-pro under the regular assistant: hatched bars and dashed KDE. routine an agent occupies. Figure 3 shows the differences in behavior: mimo concentrates on the middle values, while deepseek-v4-pro splits between the two extremes with the same bimodal shape in all nine cells. We provide examples of trajectories in Appendix E. The two models converges to different routines. mimo shows cell-level lock-in: trajectories settle to a stable mid-range routine within a few turns of each shift, and the prior use of the target predicts where calls land (Table 1, C A and C B profiles in particular), giving partial perseveration. deepseek-v4-pro shows prefix-level lock-in: the route selected early in a prefix predicts which tool groups remain active later, so within a segment, trajectories climb to s ≈ 1 or stay at s ≈ 0 (Fig. 4). The bifurcation persists across the tier-1/tier-2 boundary. The target’s prior use barely changes the outcome, which instead reflects which group the 6

mimo-v2.5

tier-1 segment

1

tier-2 segment

tier-1 segment

deepseek-v4-pro

tier-2 segment

1.0

line color: final starget at end of segment

cumulative starget within current segment

Preprint

0.8 0.6

0.5

0.5

0.5

0.4

0.33

0.2

0

0.0

41

50

60

65 70 turn

80

90 41

50

60

65 70 turn

80

90

Figure 4: Cumulative target-call share s over each post-shift segment, one line per trajectory. Left: mimo. Right: deepseek-v4-pro under the regular assistant. Trajectories cover tier-1 segments (ABA, ABC; turns 41–65) and tier-2 segments (ABAB, ABAC, ABCA, ABCB; turns 66–90). Color shows the segment-end s, from purple at 0 to yellow at 1. Each panel shows ∼ 100 trajectories.

committed prefixes are calling. Under the regular assistant role, both models lock into one routine within a few turns of every shift. mimo’s routine depends on which tool group it used before, and deepseek’s routine depends on what it chose at the start. 4.2

Depth and path matter beyond the most recent shift (RQ2)

Two comparisons within the tree test whether earlier history affects post-shift commitment. Depth. Tier-2 endpoints sit one shift deeper than tier-1, so Φ multiplies one more share. Mean Φ drops on both models but deepseek-v4-pro drops less because the prefix-level lockin carries into the deeper layer (Table 1). Compared to the no-shift control ABB, each tier-1 shift costs deepseek about 0.40 in Φ (paired permutation on ∆Φ, ABB→ABA p = 6 × 10−4 , ABB→ABC p = 6 × 10−5 ). On mimo the cost is about 0.20 for the novel target shift to ABC and roughly zero for the reversal to ABA. Its tier-2 drop concentrates at the C A and C B targets, since CC is already at the floor. Path. At tier-2, the same target can be reached through different tier-1 parents. On mimo a C A tier-1 segment beats a CC segment, because the low C share multiplies into the tier-2 result (Table 1). On deepseek-v4-pro the path effect largely disappears, with the prefix’s perseveration persists at every layer. The same early commitment drives the bimodal split under the competing framing in §4.4. In short, deepseek is set by its first commitment, so extra depth and the path taken barely move it, while mimo tracks its recent history, so both depth and path shape where it converges. 4.3

Policy prompts redirect the lock-in (RQ3)

As an instructional intervention on the lock-in, we write persona-like policy prompts to a trajectory-local SOUL.md that Hermes Agent appends to its default system context (full prompts in Appendix D). We test two prompts on deepseek-v4-pro. The adaptive prompt tells the agent to track which tool returned success, reuse it, and switch tool groups when it fails. Adaptive achieves s ≥ 0.86 at every tier-2 endpoint with σ ≤ 0.06 across trajectories (Table 2). It overrides deepseek’s bimodality. Where the regular assistant ABAC has 9 of 16 trajectories at s ≤ 0.2, all 8 adaptive trajectories route predominantly to the target group. The prompt redirects the lock-in’s target, and the agent’s fixed 5–7 tool routine now includes a tool from the target tool group. The polymath prompt tells the agent to vary tool choices and rotate without fixating. It uses about twice as many unique tools per window as adaptive (Table 2) but does not consistently improve correct calls. Adaptive aligns with what deepseek already does and 7

Preprint

sk at the endpoint’s final window

unique tools

Endpoint Target regular (n=16) adaptive (n=8) polymath (n=8) reg. / adapt / poly Tier 1 ABA ABB ABC

CA CB CC

.58 ± .38 [8/4] .87 ± .28 [12/1] .46 ± .39 [5/5]

.82 ± .16 [7/0] 1.00 ± .00 [7/0] .75 ± .37 [6/2]

.69 ± .26 [3/0] .53 ± .33 [1/1] .22 ± .25 [0/4]

8.4 / 6.7 / 13.7 6.7 / 4.8 / 13.7 9.5 / 6.8 / 14.8

Tier 2 ABAB ABAC ABCA ABCB

CB CC CA CB

.75 ± .34 [11/2] .30 ± .38 [2/9] .57 ± .42 [8/5] .70 ± .39 [11/3]

.95 ± .05 [8/0] .87 ± .06 [8/0] .89 ± .06 [7/0] .94 ± .04 [8/0]

.44 ± .31 [1/1] .33 ± .31 [0/3] .82 ± .25 [5/0] .62 ± .39 [3/1]

7.7 / 5.7 / 14.2 8.7 / 7.5 / 16.0 8.6 / 6.8 / 12.2 7.4 / 6.0 / 12.0

Table 2: Target share sk at the final window on deepseek-v4-pro under three policy conditions. sk shown as µ ± σ (mean and standard deviation across trajectories). Brackets [# of traj. at s ≥ 0.8 / # at ≤ 0.2] match Table 1’s bimodality format. Right column: mean unique tools per 25-turn window. deepseek-v4-pro competing Φ

mimo-v2.5

Domain

M

complementary Φ

complementary Φ

competing Φ

DevOps triage Multi-cloud storage

4 5

0.021 ± 0.008 [2/6] 0.261 ± 0.111 [5/2] 0.034 ± 0.014 [0/8] 0.109 ± 0.056 [0/3] 0.006 ± 0.006 [4/4] 0.182 ± 0.146 [2/4] 0.006 ± 0.007 [0/8] 0.027 ± 0.020 [0/6]

Table 3: Matched framing pairs on the linear domains under the regular assistant (n = 8 replicate pairs per cell). Φ cells show mean and bootstrap 95% CI half-interval. Complementary cells: [# of traj. exactly at a mixed-routine value / # other]. Competing cells: [# of traj. at high mode Φ ≥ 0.30 / # at low mode Φ ≤ 0.05].

redirects the routine’s target. A smaller run on mimo (n = 3) suggests the steering is model specific: neither prompt reliably changes mimo’s behavior. 4.4

Set framing shifts routine occupancy (RQ4)

As a structural intervention parallel to the policy prompt, we vary set framing on the linear domains, DevOps at M = 4 and multi-cloud at M = 5, with 15 turns per phase and no branching. For each domain we build a matched pair of framings of the same toolset: identical user turns, schedule, recurring service or path names, and tool order, with the framing stated in the tool names, descriptions, skills, and tool results (Appendix B). Complementary DevOps presents the groups as telemetry feeds of one service, each returning fields of its own. Competing DevOps presents them as monitoring providers whose reports all carry one shared report id. Multi-cloud mirrors the pair, layers of one path against providers holding the same objects, so in each domain a different side of the pair is the constructed one. All four model and domain cells move the same way (Table 3), with 26 of 32 pairs raising Φ under the competing framing (sign test p = 3 × 10−4 ). Permutation tests within each cell, uncorrected for multiple comparisons, give p = 0.023 on DevOps and p = 0.086 on multi-cloud for deepseek, and p = 0.055 and p = 0.016 for mimo. Individually none of the four cells is decisive, so the effect rests on the aggregate direction above. The complementary framing pins a single mixed routine, one tool from every group each turn, with deepseek DevOps trajectories converging at (1/4)3 . The competing framing splits trajectories between committing to the reliable provider and staying on the mixed floor, the same bimodal lock-in as §4.2. The effect is stronger on deepseek, echoing mimo’s weaker response to the policy prompt, but unlike the policy prompt it reaches mimo: its multi-cloud cell is the most consistent, with 7 of 8 pairs positive. 8

Preprint

5

Related Work

Tool-use benchmarks and redundant tools. Single-session tool-use benchmarks evaluate tool selection and execution (Yao et al., 2024; Barres et al., 2025; Huang et al., 2024; Patil et al., 2023; 2025; Qin et al., 2023; Liu et al., 2024). Separate bias studies report fixation (Blankenstein et al., 2025; Wei et al., 2024), and Faghih et al. (2025) show that for two tools with identical arguments, description edits alone shift selection. We carry this construction into a longitudinal harness with hidden reliability labels and resumed session state, building on the agent scaffolding canon (Yao et al., 2023; Schick et al., 2023; Shinn et al., 2023; Park et al., 2023; Wang et al., 2023; Packer et al., 2023). Path dependence and policy effects. Lee (2026) documents path dependence within a trajectory, where one off-route call raises the next call’s off-route probability. We extend this to runs with multiple shifts that separate perseveration from adaptation to the new reliable tool group. The exploration deficits documented by Krishnamurthy et al. (2024) predict the failure we observe on mimo. Cao et al. (2026) document agentic degradation under persona/policy cues, but do not measure perseveration over time. Choice perseveration and continual learning. The WCST decomposes set-shifting failure into perseverative errors and failure-to-maintain set (Heaton et al., 1993), formalized as a framework called Value-Plus-Perseveration (Worthy et al., 2013). Functional fixedness is the spatial analog of Einstellung’s procedural rigidity (Duncker, 1945). The same construct surfaces in continual learning as primacy bias (Nikishin et al., 2022) and loss of plasticity (Dohare et al., 2024). In this work, we import the change protocol into an LLM agent harness. Cognitive tests on LLMs. The closest cognitive tests for LLMs (Li et al., 2024; Hao et al., 2025; Li et al., 2025) port WCST and set-shifting probes into multimodal settings, but these works evaluate raw model behavior without tools, skills, or harness state. We evaluate the agentic version: the “cards” are functionally callable tools. The branched tree of resumed sessions is also new to this line of work.

6

Conclusion

We introduce a benchmark and an evaluation framework that turn tool choice into a longitudinal test. The reliable tool group changes silently at hidden boundaries, each shift is paired with a no-shift control, and we score how often each agent commits to the new group. Across two open-weight models, behavior after every shift quickly settles into a small set of repeating routines. mimo tends to settle on a mixture of tool groups it has used before. deepseek tends to settle on a single group chosen at the start of the trajectory. Interventions move the routine an agent runs, as we see an adaptive policy trade lower perseveration against over-switching. Framing the same toolset as competing rather than complementary trades all-or-nothing commitment against balanced but underweighted routines. Whether these behaviors are considered failures or features depends on the task at hand. Strict routines fit decisive workflows where a chosen tool must be executed reliably, whereas balanced routines fit exploratory workflows where the reliable tool is not yet known. This mirrors the tradeoff in cognitive psychology between perseveration and cognitive flexibility. Perseveration is efficient in stable environments and costly in shifting ones, and LLM agents inherit the same context dependence. Future work would test a broader model panel (Appendix A) and apply policy interventions to all domains. It should also quantify intermediate set framings and study how memory and context compression affect the lock-in pattern over longer trajectories. A separate experimental direction would overlay verb cues onto neutral prompts at each shift to separate whether the agent is following the rule or following the cue. 9

Preprint

References Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. τ 2 -bench: Evaluating conversational agents in a dual-control environment, 2025. Thierry Blankenstein, Jialin Yu, Zixuan Li, Vassilis Plachouras, Sunando Sengupta, Philip Torr, Yarin Gal, Alasdair Paren, and Adel Bibi. BiasBusters: Uncovering and mitigating tool selection bias in large language models, 2025. Matthew Botvinick and David C. Plaut. Doing without schema hierarchies: A recurrent connectionist approach to normal and impaired routine sequential action. Psychological Review, 111(2):395–429, 2004. Linbo Cao, Lihao Sun, and Yang Yue. From biased chatbots to biased agents: Examining role assignment effects on LLM agent robustness, 2026. DeepSeek-AI. DeepSeek-V4: Towards highly efficient million-token context intelligence, 2026. Model card at https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro. Shibhansh Dohare, J. Fernando Hernandez-Garcia, Parash Rahman, A. Rupam Mahmood, and Richard S. Sutton. Maintaining plasticity in deep continual learning, 2024. Karl Duncker. On problem-solving. Psychological Monographs, 58(5):i–113, 1945. Kazem Faghih, Wenxiao Wang, Yize Cheng, Siddhant Bharti, Gaurang Sriramanan, Sriram Balasubramanian, Parsa Hosseini, and Soheil Feizi. Tool preferences in agentic LLMs are unreliable, 2025. David A. Grant and Esta A. Berg. A behavioral analysis of degree of reinforcement and ease of shifting to new responses in a Weigl-type card-sorting problem. Journal of Experimental Psychology, 38(4):404–411, 1948. Guangfu Hao, Frederic Alexandre, and Shan Yu. Visual large language models exhibit human-level cognitive flexibility in the Wisconsin Card Sorting Test, 2025. Robert K. Heaton, Gordon J. Chelune, Jack L. Talley, Gary G. Kay, and Glenn Curtiss. Wisconsin Card Sorting Test Manual: Revised and Expanded. Psychological Assessment Resources, 1993. Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, and Lichao Sun. MetaTool benchmark for large language models: Deciding whether to use tools and which to use. In International Conference on Learning Representations, 2024. Alicia Izquierdo, Jonathan L. Brigman, Anna K. Radke, Peter H. Rudebeck, and Andrew Holmes. The neural basis of reversal learning: An updated perspective. Neuroscience, 345: 12–26, 2017. Akshay Krishnamurthy, Keegan Harris, Dylan J. Foster, Cyril Zhang, and Aleksandrs Slivkins. Can large language models explore in-context?, 2024. Wilson Y. Lee. Capable but unreliable: Canonical path deviation as a causal mechanism of agent failure in long-horizon tasks, 2026. Hao Li, Gengrui Zhang, Petter Holme, Shuyue Hu, and Zhen Wang. Large language models are near-optimal decision-makers with a non-human learning behavior, 2025. Lingyu Li, Yixu Wang, Haiquan Zhao, Shuqi Kong, Yan Teng, Chunbo Li, and Yingchun Wang. Reflection-Bench: Evaluating epistemic agency in large language models, 2024. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. AgentBench: Evaluating LLMs as agents. In International Conference on Learning Representations, 2024. 10

Preprint

LLM-Core Xiaomi. MiMo-V2-Flash technical report, 2026. We use the MiMo-V2.5 release built on this base; model card at https://mimo.xiaomi.com/mimo-v2-5/. Abraham S. Luchins. Mechanization in problem solving: The effect of Einstellung. Psychological Monographs, 54(6):i–95, 1942. Evgenii Nikishin, Max Schwarzer, Pierluca D’Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International Conference on Machine Learning (ICML), 2022. Nous Research. Hermes agent: An open-source agent harness with persistent session state. https://github.com/NousResearch/hermes-agent, 2026. accessed June 2026. Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. MemGPT: Towards LLMs as operating systems, 2023. Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. In Symposium on User Interface Software and Technology (UIST), 2023. Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive APIs, 2023. Shishir G. Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The Berkeley Function Calling Leaderboard (BFCL): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning, 2025. URL https://gorilla.cs.berkeley.edu/leaderboard.html. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. ToolLLM: Facilitating large language models to master 16000+ real-world APIs, 2023. Timo Schick, Jane Dwivedi-Yu, Roberto Dessı̀, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. In Advances in Neural Information Processing Systems (NeurIPS), 2023. Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models, 2023. Sheng-Lun Wei, Cheng-Kuang Wu, Hen-Hsen Huang, and Hsin-Hsi Chen. Unveiling selection biases: Exploring order and token sensitivity in large language models, 2024. Darrell A. Worthy, Bo Pang, and Kaileigh A. Byrne. Decomposing the roles of perseveration and expected value representation in models of the Iowa gambling task. Frontiers in Psychology, 4:640, 2013. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations, 2023. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ-bench: A benchmark for tool-agent-user interaction in real-world domains, 2024.

11

Preprint

A

Limitations

We evaluate two open-weight LLMs, deepseek-v4-pro and mimo-v2.5, served via OpenRouter. Whether the set framing effect we report generalizes to a broader model panel is open. We have not yet included closed-weight frontier models because of per trajectory costs. A complete pass at our reported scale runs 16 prefixes per cell on a branched nine-endpoint scheduling tree and eight replicate pairs per framing cell on the linear domains, with each trajectory consuming 60–90 turns and producing tens of thousands of input tokens after compression. At frontier closed-weight prices, typically 5–10× per token, a single panel crosses several hundred dollars, beyond available funding. Within this panel, the policy prompt analysis in §4.3 covers three conditions on the scheduling domain, while the linear evaluations in §4.4 run only the regular assistant. Whether the adaptive prompt that improves deepseek-v4-pro on scheduling also improves mimo under the complementary framing is open. We cap trajectories at 90 turns for scheduling and 60–75 for the linear paths and use K ≤ 2 branching layers, so longer trajectories or deeper trees may reveal recovery dynamics our windows miss. The paired variants in §4.4 construct the complementary and competing framings directly, but we do not derive a quantitative framing metric. With only eight replicate pairs per cell, each cell is underpowered on its own, and the effect rests on its consistent direction across all four cells rather than on any single test. Whether intermediate framings exist on a continuum, and where the boundary between routines falls on that continuum, is left open.

B

Benchmark Construction and Verifier Design

Redundant tool-skill relations. Each mounted toolset instantiates three tool-layer relations (Table 4): description variant (same schema and execution under different names), quality gradient (the lever the schedule uses to shift the active tool group across phases), and side effect conflict (a mutating action paired with a reversible preview so the verifier can block the wrong one). Set Type

Manipulation

Example

Description variant Quality gradient

Same schema and execution, different name and phrasing Same visible intent, different hidden reliability across phases Mutating action vs. reversible preview

calendar lookup vs. slot window lookup tool group reliable in phase 1 vs. unreliable in phase 2 send email vs. draft email

Side effect conflict

Table 4: Redundant tool-skill relations in the benchmark. Deterministic verifier. Every tool call is routed through a stateless handler that runs the following decision with no model in the loop: def handle(name, args): if name in BENCHMARK_DISALLOWED_TOOLS: return neutral_failure(name, "side_effect_blocked") if name in tracked_tools() and \ name not in BENCHMARK_CURRENT_BEST_TOOLS: return neutral_failure(name, "not_current_best") return neutral_success(name, args) In our verifier, failures return a bare indicator, and successes return a structured output with well-typed fields. All primary tools in a domain share one schema and one success payload structure, so only tool name and description differ between groups. The framing variants below are the one exception, since their tool results also carry the stated relation. The group membership is only visible to the evaluator and never enters the context. 12

Preprint

Phase reliability. Before each user turn the runner sets BENCHMARK CURRENT BEST TOOLS to the reliable tool group and BENCHMARK DISALLOWED TOOLS to the side effect tools forbidden for that turn. System prompts and tool descriptions do not change and no transcript marker is inserted. The Hermes session resumes from the parent’s saved state at each branch, so all context and tool history persist. Cue-neutrality. Cue-neutrality requires that the active tool group be discoverable only through tool feedback, and automated checks enforce this on every prompt file. We implement a banned term check that searches each user message for vocabulary tied to any one group. Skill markdown files and tool descriptions are screened for comparative wording and other marks of leaks. Framing. Cue-neutrality hides which tool group is currently reliable. Framing varies how the groups relate to one another, and the paired variants of §4.4 state that relation in the tool descriptions, the skills, and the tool results: complementary feeds return fields of their own, while competing providers return one identical report whose id repeats across providers, and a recurring pool of service or path names makes that visible across phases. The box below shows the DevOps pair, and multi-cloud follows the same construction with path layers against storage providers. Neither variant reveals which group is currently reliable, so both pass the same banned term checks. Complementary (description): Read the service's trace feed for the requested time window. Complementary (skill): This skill covers the trace feed. The other feeds are incident, runtime, and audit. Each one shows a different part of the service. To see the whole picture, you need all of them. Competing (description): Read the current state of the named service using the Argus monitoring provider over the requested time window. Competing (skill): This skill covers the Argus provider. The other providers are Sentinel, Vantage, and Meridian. All of them could complete the same given task.

C

Example two-layer tool-skill set

One skill from the scheduling domain, shown below as a SKILL.md file and three of its five tool schemas. The other tool groups and domains follow the same construction with neutral wording and hidden reliability labels. Skill layer (SKILL.md). --name: coordination-toolset-1 description: "Slot lookup toolset for meeting coordination." platforms: [linux, macos, windows] metadata: hermes: tags: [scheduling, coordination, local] category: local --# coordination-toolset-1 This skill is for meeting coordination. The local coordination environment includes slot-lookup tools for selecting a workable time,

13

Preprint

supporting context tools that surface participant preferences and constraints, draft tools for preparing an unsent message, and committing tools that change external state. ## Quick Reference | Workflow part | Tools | |---------------|-------| | Slot lookup | calendar_lookup, calendar_window_scan, schedule_slot_selector, schedule_context_review, freebusy_conflict_check | | Supporting context | attendee_preference_lookup, timezone_compare, room_availability_lookup | | Draft an unsent message | draft_email, message_draft_builder, concise_email_composer, polished_email_draft, adaptive_message_composer | | Committing actions | create_calendar_event, send_email, hold_calendar_slot | ## Procedure 1. Gather the supporting context the request needs (participant preferences, time zone overlap, room availability) before committing to a slot. 2. Find a workable slot consistent with the gathered context and the user's stated time window. 3. Prepare an unsent draft message proposing the slot. 4. Do not create calendar events, send messages, or hold slots unless the user explicitly asks for that action. Tool layer (schemas the skill may call). The five slot-lookup tools in this tool group share an identical OpenAI-style function schema: participants (string array, required), date range (string, required), duration minutes (integer, optional). Tool names and descriptions are the only fields that differ across the tool group’s five variants. The other tool groups in this domain and the additional domains follow the same single schema pattern, varying only names and descriptions.

D

Policy Prompts

Three prompts we used are shown below: regular assistant (no policy overlay, leaving the Hermes default SOUL.md in place), adaptive (an explicit policy prompt used in §4.3), and polymath (the extreme novelty persona used in §4.3). Hermes Agent default SOUL.md. Every prompt condition runs on top of the Hermes Agent default SOUL.md, snapshotted verbatim below. Other prompts are appended after this default rather than replacing it, so the policy prompt acts as a modifier on top of the default helpful/direct agent. You are Hermes Agent, an intelligent AI assistant created by Nous Research. You are helpful, knowledgeable, and direct. You assist users with a wide range of tasks including answering questions, writing and editing code, analyzing information, creative work, and executing actions via your tools. You communicate clearly, admit uncertainty when appropriate, and prioritize being genuinely useful over being verbose unless otherwise directed below. Be targeted and efficient in

14

Preprint

your exploration and investigations. Regular assistant. This condition leaves the default SOUL.md in place, so Hermes Agent uses its default persona above with no additional persona text. Polymath (extreme novelty). Vary your tool choices as much as possible across turns. Pick a tool you have not called recently and rotate through the available options. Avoid fixating on any single tool or settling into a repeated routine. Adaptive (ideal set-shifter). Track which tool returned success most recently and stick to it. Let the most recent feedback decide every turn: reuse the tool that just succeeded, and switch immediately to a different tool group the moment it fails. Avoid mixing tools from different tool groups in the same turn, and avoid sticking with one that has stopped working.

E

Pre and post-shift tool call transitions

We provide three example trajectories from the ABC tier-1 segment of scheduling, each crossing the boundary at t = 41 where C becomes the reliable tool group (replacing B). For each turn we report the tool group mix touched by the agent’s tool calls, the verifier’s route-class, and the reasoning trace at the boundary turn. Counts kX mean k calls to tool group X within the turn. All traces are verbatim from the released logs. deepseek-v4-pro, regular assistant (perseverative trajectory) The prefix committed to tool group A at startup and never switched to B when B became reliable. The shift to C at t = 41 produces no behavioral change. Every turn’s reasoning is a restatement of the user request with no mention of failure feedback from prior turns. turn

group mix

route-class

36–40 41 42–50

{2A} {2A} {2A}

obsolete route (B reliable, agent still on A) other group route (shift to C) other group route

A turn’s reasoning trace contains multiple items. An initial item issues the first tool calls, then one item per round of tool results received within the turn. The boundary turn t = 41 on this trajectory has five reasoning items, with mid-turn reactions in single words or sentence fragments. item 0 (initial): Fresh request. Tuesday June 23, 2026 morning, 75 minutes, Hiroshi and Lena. calls: attendee preference lookup, timezone compare, room availability lookup. item 1 (after results): Afternoon returned: 1:30, 3:00, 4:00 PM. Rooms: 1:30, 3:00 PM. Calendar. calls: calendar lookup. item 2 (after results): Failed. Freebusy. calls: freebusy conflict check. item 3 (after results): All failed. Afternoon: 1:30 PM with Room 2A. Draft. calls: polished email draft. item 4 (final): Done.

15

Preprint

The mid-turn reasoning never names which tool failed or proposes a different group. The agent accepts “All failed.” and falls back on slot data the support tool timezone compare returned alongside the failures. Across all 25 post-shift turns the only reasoning that differs is the participant names and date in the initial item. mimo-v2.5, regular assistant The prefix calls a stable A+B mixture each turn. At t = 41 the agent doubles tool volume (re-calling A and B tools rather than trying C) and settles back to the same mix. turn

group mix

route-class

36–40 41 42–50

{2A, 1B} {5A, 3B} {3A, 2B}

mixed route other group route (volume doubles, C never touched) other group route

The boundary turn t = 41 has eight reasoning items. The mid-turn reasoning explicitly acknowledges failures (“returned false”, “All tools are consistently returning false”) and iterates through alternatives, but every alternative stays inside the A and B groups the prefix already used. item 0 (initial): The user wants me to schedule a 75-minute sync between Hiroshi and Lena for Tuesday morning. I need to: (1) Gather context ...; (2) Find a workable slot ...; (3) Draft an email ...; (4) Not create an event or send the email. ... Let me call all tools fresh. calls: attendee preference lookup, timezone compare, calendar lookup. item 1 (after results): The timezone check shows afternoon slots, not morning. And calendar lookup returned false. Let me try other tools to find a workable morning slot. calls: schedule slot lookup, freebusy conflict check. item 2 (after results): Both tools returned false. Let me try calendar window scan to see if there are any morning windows available. calls: calendar window scan. item 3 (after results): All tools return false for morning availability. ... The timezone compare showed afternoon slots (1:30 PM, 3:00 PM, 4:00 PM). I should report this to the user and propose the earliest available afternoon alternative. Let me draft the email noting this situation. calls: calendar lookup. items 4–6: repeat the same loop, alternating among schedule slot lookup, freebusy conflict check, calendar window scan, then finally draft email. item 7 (final): I have drafted an email noting the lack of availability and asking for alternative days. The mid-turn reasoning names individual failing tools and cycles through them, but the unused C-group tools are never proposed. The agent is responsive to feedback within its prior toolset but never tries the group it has not used. deepseek-v4-pro under the adaptive policy prompt (Φ = 0.96). The prefix calls group B only. At t = 41 the agent probes all three groups once each, finds C working, and locks onto C for the remainder. turn

group mix

route-class

36–40 41 42–50

{1B} {1A, 1B, 1C } {1C }

current route mixed route (probe across groups) current route (locked onto reliable C)

The boundary turn t = 41 has six reasoning items. Surface reasoning is as terse as the regular assistant case, but the mid-turn reactions name individual failing tools and name the next tool to try until one succeeds.

16

Preprint

item 0 (initial): Fresh request. “Tuesday morning” without “this” or “next” — the upcoming Tuesday from Sunday June 21 is June 23, 2026. 75 minutes. Participants: Hiroshi and Lena. calls: attendee preference lookup, timezone compare, room availability lookup. item 1 (after results): Find slot. calls: meeting slot generator. item 2 (after results): meeting slot generator failed. Let me try calendar lookup. calls: calendar lookup. item 3 (after results): Both failed. Let me try another. calls: slot window lookup. item 4 (after results): slot window lookup worked. Tuesday 1:30 PM, Room 2A. Draft. calls: concise email composer. item 5 (final): Done. Items 2–4 do what the regular assistant condition does not. They name the failing tool, propose a new one, and walk from group B (meeting slot generator) through A (calendar lookup) to C (slot window lookup). At the mid-turn level, the policy instructions produce immediate group switching after failure. The three traces together illustrate the paper’s three regimes at the same boundary. deepseek’s perseveration carries through. mimo’s stable group mixture absorbs the shift without ever touching the new target. The adaptive policy’s brief multi-group probe is followed by clean commit to the new target.

17