arXiv CSopen access

Understanding Agent-Reactive Bugs at the Model-Harness Boundary: An Empirical Study of LLM Agent Issue Reports

Understanding Agent-Reactive Bugs at the Model-Harness Boundary: An Empirical Study of LLM Agent Issue Reports Jingyi Chen, Songqiang Chen, Hengcheng Zhu, Jialun Cao* , Jiasi Shen* , and Shing-Chi Cheung

arXiv:2607.15684v1 [cs.SE] 17 Jul 2026

The Hong Kong University of Science and Technology, Hong Kong, China Guangzhou HKUST Fok Ying Tung Research Institute, Guangzhou, China {jchenix,i9s.chen,hzhuaq}@connect.ust.hk {jcaoap,sjs,scc}@cse.ust.hk

Abstract—LLM agents span command-line interfaces (e.g., Codex) and agent frameworks (e.g., LangChain), integrating backend LLMs with harness code that parses model outputs, controls agent loops, and manages context. Both the harness and LLM-generated responses jointly shape an agent’s execution. This architecture gives rise to bugs that cannot be readily understood by inspecting either component alone, because some bugs occur only when a particular LLM response elicits an abnormal reaction from the agent. Prior empirical studies of agent bugs have largely attributed failures either to limited model capabilities or to harness-side defects, such as outdated APIs and configuration misalignment, without characterizing these AR bugs. We conduct the first empirical study focused on agent-reactive (AR) bugs. Through manual analysis of 255 bug reports from Codex, GeminiCLI, LangChain, and CrewAI, we construct a two-axis taxonomy covering observable symptoms and the LLM behaviors that trigger them. Our findings show that many AR bugs manifest as silent errors without well-defined test oracles, which makes detection difficult. The stochasticity of LLM responses further complicates bug reproduction. We additionally examine fixes proposed by users and implemented by developers. This analysis exposes a mismatch: users frequently advocate harness-side guardrails, whereas developers may attribute the issue to the LLM or respond slowly to user-proposed fixes. These findings point to the need for mechanisms that help users and developers understand the root causes and resolutions of AR bugs. Overall, the study highlights challenges specific to LLM agents and motivates the design of test oracles, reproduction support, and fault-localization techniques for AR bugs. Index Terms—LLM Agent Reliability, Model-Harness Interaction, Tool-Use Failure, Empirical Study

I. I NTRODUCTION Large Language Model (LLM) agents, available in a form ranging from command-line interfaces (CLIs) to programmable agent frameworks, are increasingly being adopted for coding tasks [1], [2], web navigation [3], and customer service [4]. In the rest of this paper, we use agents to refer to LLM agents for brevity. At the core of an agent is a harness scripted to dispatch tool calls, parse model output, execute the agent loop, and manage context, while its backend LLMs are continually prompted to generate answers that can vary across runs. The * Corresponding authors: Jialun Cao and Jiasi Shen.

agent reacts to these answers and performs the assigned task [5]. The highly reactive nature of interactions between the harness and the underlying stochastic LLMs distinguishes agents from traditional software. Evolving quickly, these agents also suffer from various bugs. Users of open-source agents often report the encountered bugs in the agent projects’ GitHub trackers. The bug reports of agents have drawn growing research attention. Many empirical studies have been conducted to propose a taxonomy of agent bugs by mining issues from the repository of agents or agent frameworks [6]–[9]. Studies have also been carried out to evaluate agent capabilities using various benchmarks [1], [4], [10], [11] and explain incorrect LLM responses, such as hallucination, [12]–[18]. These studies help explain the kinds of failure that can occur in agents and the limits of their capabilities. However, they have not examined a unique class of bugs that arise from LLM-harness interactions when the harness does not handle that LLM behavior properly, which we name as agent-reactive (AR) bugs. Our analysis reveals that AR bugs account for 8.4% of all actively discussed issues in four popular agent projects: Codex, Gemini-CLI, LangChain, and CrewAI (Section III). For example, in Codex issue 13491 (shown in Fig. 1), the orchestrator model is requested to spawn a sub-agent with fork_context=true setup. The harness forks the context from the orchestrator and appends handoff instructions informing the sub-agent to treat the next user message as its own task. However, the subagent model ignores the handoff instructions, acts as another orchestrator, and spawns subagents recursively. Such AR bugs pose particular challenges to diagnose compared with both LLM-side performance problems and harness-side defects, like outdated APIs and configuration misalignment, because AR bugs cannot be identified by examining the harness or the LLM alone. The stochastic nature of LLMs also makes AR bugs hard to reproduce. Thus, understanding the symptoms and fixing strategies of the AR bugs is crucial to improving the reliability of agents. To fill the gap, we conduct the first empirical study of AR bugs. We collect agent bugs from two popular agent CLIs (Codex and Gemini-CLI) and two agent frameworks

LLM (Orchestrator)

LLM (Sub-Agent)

Context of the orchestrator: User Message: spawn a subagent... Action: call spawn agent tools with fork_context: true

Action: call spawn agent tools with fork_context: true

Harness Context forked from orchestrator: User Message: spawn a subagent... Action: call spawn agent tools... Context appended by harness: User Message: You are the newly spawned agent. Treat the next user message as your new task...

Spawn sub-agents recursively Regard as the instruction to sub-agent

Ignore

Fig. 1. Codex issue 13491: a forked worker inherits parent intent and attempts recursive delegation instead of following its assigned task.

• We conduct the first empirical study dedicated to agent

reactive (AR) bugs, whose manifestation depends on both a specific LLM behavior and the harness that handles it. • We manually annotate a dataset of 255 AR bugs from two agent CLIs (Codex and Gemini-CLI) and two agent frameworks (LangChain and CrewAI). • We build a two-axis taxonomy that pairs the symptoms of these bugs with the LLM behaviors that trigger them, and use it to analyze how a behavior turns into a symptom. We find the new challenges raised from AR bugs in detection, reproduction, and fix localization, motivating new research opportunities. • We study how users and developers currently act to fix these bugs, revealing a mismatch between them over fault attribution in agents and inspiring how to handle users’ complaints in the future.

II. P RELIMINARIES (LangChain and CrewAI). These four projects allow users to report agent failures as GitHub issues, where reporters, other users, and developers discuss symptoms, reproduction attempts, A. LLM Agents workarounds, and possible fixes. Based on the issues of the four Large language models (LLMs) have shown strong capability projects, we obtain 255 AR bugs after rule-based filtering to obtain bug reports with LLM invocation and manual annotation in software engineering [22], including code generation [23], to collect bugs that depend on specific LLM responses. We [24], code translation [25], and test generation [26]. To move analyze the symptoms observed by users (RQ1) and the LLM beyond answering a single prompt, LLMs are wrapped in an behaviors that trigger them (RQ2), which motivates the design agent loop and connected to external tools, yielding an LLM of test oracles and reproduction techniques. We study a follow- agent that can plan and carry out multi-step tasks [2], [27]. up question: when did these failures surface, and where were In common designs, the loop alternates between behaviors their fixes applied—in the agent harness or at the backend generated by the model and results returned by tools or LLM? We answer this question by analyzing user proposals environments [2], [28]. Such agents are deployed as clients and developer actions in issue discussions and related linked (e.g., OpenAI Codex and Gemini-CLI) and frameworks for customized agents (e.g., LangChain and CrewAI). Following pull requests (RQ3). We identified five categories of AR bug symptoms and prior work [2], [27], [29], we view an agent as two interacting eight LLM behaviors that trigger AR bugs. Our findings components. Backend LLM. Given the current context, the backend reveal several challenges for detection, reproduction, and fix LLM generates the next response or action. Its output may localization. First, many symptoms lack a rigorous oracle [19], vary across runs even with an identical initial prompt [5]. [20], especially silent errors, where a plausible but incorrect Harness. It provides an agentic wrapping of the backend result is hard to tell from a correct one. This suggests that LLM [29]. It renders prompts, parses model output, performs agent users should carefully monitor and audit the LLMthe agent loop, manages context, and so on. The formats or harness interactions on critical tasks, as well as motivates constraints that the LLM should follow and that the harness is research opportunities for designing effective test oracles to expected to handle are defined in a piece of harness code. automatically detect AR bugs. Second, we found that the occurrence of AR bugs is affected by specific LLM responses, The harness reacts to the LLM. It sends the LLM a context context length, and the state of the workspace [5], [21], making with task instructions and previous responses. The LLM returns them difficult to reproduce. Thus, developers are recommended either a final response or an action, such as a tool invocation. to instruct users to properly record and share the context to The harness parses the output, performs the action if needed, ease debugging of AR bugs. This also leaves auto-reproduction appends the result to the context, and continues until the of AR bugs a significant research problem. Third, we reveal task ends. Unlike traditional software, whose control flow is that the diagnosis of AR bugs can be controversial: users predefined, an agent’s control flow is determined on-the-fly by may propose harness-side guardrails, while developers may the interaction between the harness and the LLM. For example, attribute failure to poor LLM capability. This poses the need in the agent shown in Fig. 1, the LLM in the sub-agent treated for a guideline for fault attribution in agents, which can help the forked orchestration history as an instruction for itself and developers and users align on the root cause of AR bugs and invoked the spawn-agent tool. The control flow of the whole improve the reliability of agents. system therefore shifts from the intended sub-agent task to a In summary, our study makes four contributions: recursive delegation loop.

and then to 11,288 LLM-reachable bug issues (column LLMUsers of agents routinely encounter bugs and report them reachable). We treat a bug as LLM-reachable if the reported on project GitHub trackers, sometimes with reproduction steps, reproduction steps contain the invocation of the backend LLM. workarounds, or proposed fixes. Many empirical studies analyze Finally, we keep only issues with sufficient maintainer or these reports through taxonomies of bug types, symptoms, and community engagement so that annotators have enough context root causes [6]–[9]. These studies often attribute a failure to to analyse. The four projects differ in user mix and discussion one of the two components: limited backend-LLM capability, density: Codex and Gemini-CLI receive many user reports and such as hallucination [12], [15], or harness-side defects, such comments but rarely have linked PRs, while LangChain and CrewAI attract fewer but technically deeper discussions. We as outdated APIs and configuration misalignment [7]–[9]. In this work, we focus on bugs at the interface between the therefore adapt the engagement threshold per project. Codex two components, which we call AR bugs. Such bugs manifest and Gemini-CLI require a linked PR or at least three nononly when the backend LLM exhibits particular behaviors, author commenters; LangChain requires a linked PR and at such as generating unexpected tool arguments, ignoring task least three non-author commenters; CrewAI requires a linked instructions, or fabricating claims, and the harness handles that PR and at least one non-author commenter, relaxed due to its behavior in a way that produces a user-visible failure. These smaller corpus. After all three filters, 3,037 issues (column bugs can be hard to detect when their symptoms are inaccurate Engaged) are forwarded to manual annotation. For every issue in the engagement-filtered pool, we ask outputs rather than thrown exceptions. They can be hard to two questions: (i) does the bug’s manifestation depend on a reproduce because the triggering behavior is generated by the specific LLM response? and, if so, (ii) which LLM behavior LLM and may depend on long context, workspace state, and triggers it and how does the bug manifest? The first question even model version. Whether the bugs should be fixed in the yields a binary AR / non-AR label; the second yields the harness or in the backend LLM is often controversial. two-axis taxonomy (triggering behavior × symptom) used in Section IV-B. Two authors annotated the issues following a III. DATA C OLLECTION two-phase protocol: To study AR bugs in agents, we mine the issues of four Phase 1 — Initial taxonomy. Two authors independently widely used open-source agent projects. Through automated sampled 5% of the engagement-filtered pool and labeled the filtering and manual annotation, we identify the subset of bug issues in an open-coding fashion, naming each triggering reports whose manifestation depends on a specific response behavior and symptom category as it arose. If an issue does from the backend LLM. The pipeline is summarised in Table I. not depend on a specific LLM response (e.g., the issue is due to network or data streaming problems), we assign a special A. Subject Collection label for it. They then merged their category sets and refined We select four projects that cover two common forms the definitions through discussion to produce a shared initial of agents: agent CLIs and agent-construction frameworks. taxonomy. Each selected project is open source, has a public GitHub Phase 2 — Annotation and reconciliation. Then the two issue tracker, contains enough issue discussion for manual authors partitioned the remaining 95%, and each labelled analysis, and implements or supports agent workflows with their portion against the shared taxonomy. New categories model calls, tool use, and so on. The two clients, Codex encountered during this phase were temporarily added and (openai/codex) and Gemini-CLI (google/gemini-cli), are were merged and refined after the annotation. Conflicting or applications that users can run directly. The two frame- uncertain labels were then discussed until both annotators works, LangChain (langchain-ai/langchain) and CrewAI reached a consensus. (crewai/crewai), provide programmable abstractions for users Overall, the two authors each spent about 150 hours on to construct customized agents. This split lets us examine phase 1 and phase 2. Of the 3,037 engagement-filtered issues, whether the symptoms and triggering LLM behaviors of 255 are AR bugs and 2,782 are non-AR bugs (Table I, column agent bugs differ between agent CLIs and agent-construction AR bugs). For these 255 AR bugs, we further inspect issue frameworks. For each project, we collect all issues opened discussions and linked pull requests to analyse user proposals from the start of its issue tracker until April 2026. and actions by project developers about where the fix should live; the detailed rubric is presented in Section IV-C. B. Issue Mining and Annotation IV. S TUDY R ESULTS AND A NALYSIS For each subject project, we collect all its issues through the GitHub REST API, yielding a starting corpus of 32,373 A. RQ1: Symptoms issues across the four projects (Table I, column Raw). To answer the first question, we analyze the symptoms, We then use keyword-based filtering to discard non-bug i.e., what users observe when failures manifest, of each issue issues (e.g., feature requests, questions, documentation sug- in our corpus. For each symptom, we give its definition, a gestions) and bugs whose reproduction cannot reach an LLM representative issue, and its distribution across the four projects. call (e.g., missing-module errors, package-installation failures), We also summarize either users’ attitudes or the difficulties reducing the corpus to 18,161 bug issues (Table I, column Bugs) they face for each symptom. B. Agent Bugs

Raw

Bugs

LLMreachable

Engaged

AR bugs

Codex (CLI) Gemini-CLI LangChain CrewAI

8,523 12,662 9,314 1,874

6,290 4,434 6,365 1,072

2,428 2,802 5,308 750

963 1,201 545 328

82 90 47 36

Total

32,373

18,161

11,288

3,037

255

Project

LLM Before Compaction Context of the LLM: Task: developing a Java project. (Reasoning and Planning...) Action: edit pom.xml (x3) (...more actions...)

Harness Auto-compaction: Compact the long context to a short memento.

Long context accumulated

TABLE I I SSUE COUNTS AT EACH STAGE OF THE DATA - COLLECTION PIPELINE . Raw: ISSUES CRAWLED FROM G IT H UB . Bugs: ISSUES THAT REPORT A BUG . LLM-reachable: BUG ISSUES WHOSE REPRODUCTION STEPS CAN REACH AN LLM CALL . Engaged: LLM- REACHABLE BUG ISSUES WITH SUFFICIENT COMMUNITY ENGAGEMENT ( SEE S ECTION III-B). AR bugs: ISSUES THAT HUMAN ANNOTATORS CONFIRMED AS AGENT- REACTIVE .

LLM After Compaction

Expected: Observed: Summing the bottom Tot row across the four panels of Continue from prior Generate a plan and wait Table II gives the symptom totals, and we therefore introduce edits and run tests. for user intervention. the symptoms from most to least frequent. We abbreviate symptom categories as follows: SE = silent errors, Cr = crash, EIO = error in the output, RL = retry loop, and H = hang. Fig. 2. Codex issue 5957: after auto-compaction, the agent should continue The rows in the table represent triggering behaviors, which we from prior edits but instead generates a new plan and waits for the user. introduce in RQ2. Silent Errors (108): The agent appears to run normally and leaving no exception or warning; the user only notices the produces a fluent answer, action trace, or workspace change, discrepancy if they manually inspect the working trajectory (as but the result is incorrect and no exception, warning, or retry in Codex 5957, illustrated in Fig. 2) or compare the agent’s loop alerts the user. Unlike silent errors in traditional software, narrated steps against the record of actual tool calls (as in Codex this symptom in an agent is often difficult to judge from the 6562). These cases can also be hard to reproduce because final response alone: the evidence needed by the oracle may lie reproduction may rely on long context, complex workspace in the tool log, workspace diff, or long interaction trajectory. state, and a particular sequence of LLM behaviors. Crash (71): Exceptions are raised, and the whole agent We highlight two representative cases of such issues. In Codex issue 5957 [30] (as shown in Fig. 2), the agent worked session terminates or enters an unrecoverable error state, and on a Java repository and was expected to continue editing files users cannot continue working with the session. We illustrate until the task was complete. After the agent had edited multiple the crash symptom with two representative cases that span the files, the harness automatically compacted the long context situation faced by both agent SDK users and agent product into a short memento that omitted those edits. The expected CLI users. In LangChain issue 1358 [32], the user initializes behavior was to continue from the previous task state after compaction. Instead, the agent behaved as if it had just started: a ConversationChatAgent with HuggingFace’s it forgot the previous edits, generated a fresh plan, and waited flan-t5-xl, then the framework raises ValueError: for user intervention. The symptom is silent because no crash Could not parse LLM output: ‘‘Assistant, or warning tells the user that the task state was lost; the user how can I help you today?’’. Another example must inspect the chat history and workspace state to distinguish is Gemini-CLI issue 13292 [33], where a long agent session crashes mid-response with a backend 400 this premature ending from normal task completion. ‘‘Please ensure that the number Another example is Codex issue 6562 [31], in which the stating agent claimed that it performed a sequence of verification steps: of function response parts is equal to running pnpm dev, opening DevTools, clearing the network the number of function call parts of the log, and observing the request payload. However, the user function call turn.’’. Once the error fires, every never granted it permission to access the relevant tools, and the subsequent prompt returns the same wall of 400 messages, and harness had no record of such tool calls. The LLM’s response the user reports there is no way to continue the conversation. is fluent and structured as a successful audit trail, so neither Although the agent program does not terminate, we still the harness nor the user receives any signal that the work never consider this issue a crash because the session cannot continue, happened. To notice this failure, the user cannot rely on the producing an effect similar to a crash in traditional software. final response alone; they must compare the agent’s claimed Compared with agent products (Gemini-CLI and Codex), actions with the configuration and tool-call log. customized agents built with LangChain and CrewAI suffer The SE column of Table II contains 108 issues (56 in Codex, from the Crash symptom more frequently in proportion (as 33 in Gemini-CLI, 14 in CrewAI, and 5 in LangChain), making shown in Table II). Crash accounts for 79% of LangChain silent errors the largest symptom in our corpus. They are hard issues (37/47) and 25% of CrewAI issues (9/36), versus 26% to detect because the agent’s reply is fluent and self-consistent, of Gemini-CLI issues (23/90) and only 2% of Codex issues

TABLE II T RIGGERING BEHAVIORS BY SYMPTOM ACROSS PROJECTS . E ACH PANEL IS ONE PROJECT: ROWS ARE TRIGGERING BEHAVIORS , COLUMNS ARE SYMPTOMS , DARKER CELLS INDICATE MORE ISSUES , BLANK CELLS INDICATE ZERO , AND THE RIGHTMOST AND BOTTOM TOTALS SUMMARIZE TRIGGERING - BEHAVIOR AND SYMPTOM COUNTS .

Codex

Gemini-CLI

LangChain

CrewAI

Trigger SE Cr EIO RL H Total

SE Cr EIO RL H Total

SE Cr EIO RL H Total

SE Cr EIO RL H Total

INC UTA MTC IHI SF EmpR CO TH

31 4 7 1 10

Total

56

1 3

4 6 1

2

1 5

3

1 2 2

6

11 7

36 14 7 9 10 0 4 2

19 3 1

82

33 23

1 7 13

7 2 1

1

1 1

24 29 3 14 8 6 3 3

14

18 2

90

6 2 1

3 1 13 1

4 2

LLM LLM calls a ask_question tool but omits the coworker argument from the call.

Harness Harness bind the argument coworker=None Source code of the ask_question tool: def ask_question(self, question: str, context: str, coworker: Union[str, None] = None, **kwargs): coworker = coworker or kwargs.get("co_worker") or kwargs.get("co-worker") is_list = coworker.startswith("[") and coworker.endswith("]")

↑ raises AttributeError when coworker is None

User Observation Error: 'NoneType' object has no attribute 'startswith'

Fig. 3. CrewAI issue 668: an omitted tool argument becomes an error in the output.

(2/82). Crash occurs in both traditional software projects and agents and is usually easy to detect. Some crash issues also attracted high engagement of users. LangChain issue 1358 accumulates 124 thumbs-up reactions and 81 comments, which is the highest-engagement issue in our corpus. Gemini-CLI 13292 is also the canonical issue for many duplicate reports, indicating that a crash is a significant symptom in agents. Error in the Output (34): The session is still working, but it displays an error message instead of a normal AI message or tool message. Unlike silent errors, this symptom is directly noticeable because the error message can be distinguished from normal output. For example, in CrewAI issue 668 [34] (Fig. 3), the LLM calls the ask_question tool but omits the coworker argument. The harness binds this missing value as None; after the tool code calls startswith on it, the user only receives ’NoneType’ object has no attribute ’startswith’. Such a raw error message is not a useful signal for recovery and is confusing for users to debug.

1 1 1 11 3 18

5 37

1 1

1 1 1

1 2 4 2

3 0

4 12 23 0 1 1 2 4

3 1 8

47

14

1 1

1 3

5 4

3 2

1

1

2 9

12

1 0

3 7 16 0 1 5 2 2 36

The EIO column of Table II contains 34 issues, concentrated in Gemini-CLI (14) and CrewAI (12), with fewer cases in Codex (6) and LangChain (2). EIO is easy to detect because the user sees a raw error message in place of normal output, but it is harder to reproduce because the occurrences are context-dependent and environment-dependent. We found in the comments of CrewAI issue 668 that the workarounds shared in the issue threads tend to help some users while failing for others, suggesting the uncertainty of this bug. Retry Loop (33): The agent repeatedly takes the same or semantically equivalent action until it reaches an iteration/time limit or the user intervenes. Unlike a hang, the agent is still active but unproductive. For example, in Gemini-CLI issue 5629 [35], the user asks Gemini to edit a file, and the agent invokes the replace tool with an old_string that does not exactly match any substring on disk. The tool returns “Failed to edit, 0 occurrences found for old string”, after which the agent re-reads the file, retries an edit with the same problem, fails again, and remains in this read-fail-retry cycle until the iteration limit is reached or the user closes the session. This differs from infinite loops in traditional software systems. On one hand, retry loops in agents waste user tokens without producing useful results, which makes users less tolerant of this symptom. On the other hand, the retry loop pollutes the agent’s context, thus affecting subsequent work. The RL column of Table II contains 33 issues, concentrated in Gemini-CLI (18) and Codex (11), with only 3 in LangChain and 1 in CrewAI. This distribution shows that retry loops are much more common in the two CLI products than in the two agent frameworks in our corpus. Detection of RL is straightforward when the LLM repeats the same action verbatim, but can be challenging when it repeats semantically similar actions. Hang (9): The agent becomes unresponsive or remains in a working/waiting state without visible progress. Unlike a retry loop, a hang does not repeatedly execute the same actions and may produce no action at all. The H column of Table II contains only 9 issues, all from the two CLI products: 7 in Codex and 2 in Gemini-CLI. However,

it is still worth separating from other symptoms because the control flow, such as causing repeated delegation in Codex user’s session is blocked without visible progress. For example, issue 13491 [37]. in Codex issue 4337 [36] (Fig. 4), the agent stays in the Unexpected Tool Arguments (62): Invoking tools to working state after a command timeout instead of moving to a complete tasks is a crucial feature of agents, and the tool failure state and trying other commands. As Fig. 4 shows, after execution starts from the tool arguments provided from LLMs. timeout, the wrapper kills the main process, but a surviving UTA occurs when the model calls valid tools with existing child process keeps running and hold stdout/stderr pipes open. arguments, but the argument value is in unexpected format or However, the harness still waits in the working state. We explain datatype, or the agent framework fail to handle the argument. the cause of this hang in RQ2. If the model uses a wrong tool name or argument name, RQ1 Finding: Agent failures are not limited to explicit crashes we label it as TH instead. For example, in CrewAI issue or raw errors. The most common symptom is silent errors, 668 [34] (Fig. 3), the LLM invokes the ask_question tool where users must inspect external evidence such as tool logs, but omits the required coworker value. The figure shows workspace changes, or long trajectories to notice the failure. the harness binds the missing value as None, dispatches Crash and EIO are easier to observe but still leave localization it to the tool body, and the tool dereferences it through difficult, while retry loops and hangs waste tokens, time, or coworker.startswith(...), producing a raw Python an entire session. These symptoms motivate RQ2, where we AttributeError. Another example is Gemini-CLI issue 3037 [38], where the argument value produced by the LLM analyze the LLM behaviors that trigger them. is a commit message containing backticks. When the harness passes it to git commit through the shell, the shell interprets B. RQ2: Triggering Behaviors Noticed by Users the backticks as command substitution and drops the wrapped RQ2 asks which LLM behaviors expose the user-visible text. In this case, the argument value is a valid string, but the symptoms described in RQ1. The behavior itself is not failure comes from how the harness handles it. necessarily faulty; an AR bug manifests when the harness Unlike INC, UTA has no dominant symptom: crash (19), parses, executes, or compacts that behavior in a way that retry loop (19), and EIO (14) each account for a substantial produces a symptom. Thus, this subsection focuses on the fraction of the 62 cases. The split tracks the harness’s toolinteraction between model decisions and harness control flow, dispatch design. In Table II, the LangChain panel’s UTA row rather than assigning the fault solely to either the backend has 11 cases in the Cr column out of 12 UTA cases, because LLM or the harness. LangChain’s wrappers often propagate the underlying Python We introduce these behaviors from more to less frequent exception. In the Gemini-CLI panel, the UTA row has 13 cases in Table II, which abbreviates them as follows: INC = task in the RL column out of 29 UTA cases, because Gemini-CLI instruction non-compliance, UTA = unexpected tool arguments, often returns the tool error to the model as an observation and MTC = message template conflict, IHI = immature harness allows repair attempts. In the CrewAI panel, the UTA row has 5 interface, SF = statement fabrication, EmpR = empty response, cases in the EIO column out of 7 UTA cases, because CrewAI CO = context overflow, and TH = tool hallucination. For each often surfaces the raw Python error as the tool output. The behavior, we describe the triggering behavior noticed by users, same class of LLM behaviors therefore manifests as different use a representative issue to show how the harness turns it into symptoms depending on the layer of the harness that catches a symptom, and summarize its distribution across symptoms. the unexpected arguments. Task Instruction Non-compliance (67): We label a behavior Message Template Conflict (49): MTC occurs when the as task instruction non-compliance when the model ignores model’s output does not match the message or action template task instructions or system prompts while still producing that the harness parser expects. This differs from INC because syntactically acceptable output. If the output conflicts with the key failure is not simply disobeying a task instruction, a required template, we label it as message template conflict but crossing a parser boundary. For example, in LangChain (MTC), which we discuss below. For example, in Codex issue issue 1358 [32], a ConversationChatAgent using Hug13491 [37] (Fig. 1), an orchestrator calls the spawn-agent tool gingFace’s flan-t5-xl responds with conversational text with fork_context=true. As Fig. 1 shows, the harness instead of the ReAct-style markers that the parser expects, builds the sub-agent context by forking the orchestrator context such as Thought:, Action:, and Action Input:. The and appending a handoff message that tells the sub-agent to parser cannot extract the next action and raises ValueError: treat the next user message as its new task. However, the sub- Could not parse LLM output. agent treats the inherited context as an instruction for itself, The case illustrates how a parser contract designed around ignores the appended handoff, and attempts to spawn sub-agents a particular action format can fail when users plug in a model recursively. Instruction non-compliance usually manifests as that emits a different conversational format. MTC maps to silent errors, as the INC row of Table II shows that 54 of 67 symptoms according to the strictness of parser and recovery issues (81%) surface as SE. This is because when a model policy in the harness. Across the corpus, it manifests as ignores an instruction, it can still return fluent and well-formed crash (21), silent errors (19), EIO (7), and retry loop (2). text that satisfies the harness’s surface contract. The remaining This behavior is concentrated in LangChain and CrewAI, non-SE cases arise only when the ignored instruction changes where users combine framework parsers with diverse backend

workspace state. SF overwhelmingly leads to silent errors: 18 of 20 cases Still working... (90%) manifest as SE. The reason is that the fabricated Codex statement is itself a fluent answer or progress report, so no exception is produced. The two non-SE cases become Harness retry loops when the fabricated statement triggers subsequent Run bash -lc + command A listener that waits for EOF repeated actions. from stdout/stderr pipes. Empty Response (12): EmpR denotes literally empty model Spawn Terminal responses, such as empty content or empty decision, not merely low-quality or unhelpful content. In Gemini-CLI issue Wrapper Process Main Process 7223 [40], the backend model returns an empty completion, Kill No EOF often after a ReadManyFiles call that loads a large set Child Process Timeout! of files into context. Before a harness-side workaround, the Gemini-CLI silently drops the user’s most recent prompt and Fig. 4. Codex issue 4337: after timeout, the wrapper kills the main process, continues as if it had received no model content. After the but a child process keeps running and leaves Codex still working. workaround, the harness automatically retries the request. If all retries fail, it responds the error ’’The model failed to models to build their customized agents, which may not have respond after multiple attempts.’’ to the user. parsers and recovery policies that are robust to formats of all The symptom of EmpR depends on how the harness handles LLMs. In LangChain, 18 of 23 MTC cases manifest as crash the empty payload. Across the corpus, EmpR manifests as EIO because parser failures are raised as exceptions. In CrewAI, (5), crash (4), and silent errors (3). CrewAI and LangChain the symptoms triggered by MTC spread across SE, crash, EIO, usually treat an empty response as an unrecoverable value and RL, indicating that different layers can catch or expose the error, and produce the crash symptom. Gemini-CLI more often mismatch differently. For agent CLIs, Codex and Gemini-CLI shows the corresponding error message to the user. show fewer MTC cases in our corpus. One plausible reason Context Overflow (11): CO covers failures triggered by long is that their harnesses and default model families (i.e., GPT memory content. Accumulated long messages generated by and Gemini series) are co-designed around similar interaction LLMs or overly large tool results can lead to a long memory formats. content. CO manifests mainly as crash (6) or silent errors Immature Harness Interface (23): IHI captures cases (4). For example, Codex issue 5957 [30] (Fig. 2) mentioned where the LLM takes a valid action, but the harness interface in RQ1 falls into silent errors. In this issue, long context cannot coordinate, terminate, or recover from that action triggers automatic compaction, but the compacted memento correctly. These cases often involve gaps between what the omits prior file edits, so the agent generates a new plan instead model attempts to do and what the surrounding agent runtime of continuing from the prior edits. However, in CrewAI issue can safely support. For example, in Codex issue 4337 [36] 3843 [41], the user asks for a single Elasticsearch record, but (Fig. 4), the LLM invokes a shell tool, and the harness runs the agent retrieves ten thousand records; once the tool output the command through its standard bash -lc wrapper. The is appended to context, the next model request exceeds the command spawns a sub-process and then times out. The token window and the agent crashes. wrapper kills the main process, but leaves a child process alive This is a case where the same LLM behavior can become and keeps stdout or stderr pipes open. The harness therefore either an explicit failure or a plausible but wrong output, waits for an EOF that never arrives, and the agent is stuck in indicating that the behavior-symptom mapping is useful not the “working” state. Here the triggering action is an ordinary only for root-cause analysis, but also for designing robust shell invocation, but the symptom comes from the process- harness. management interface in the harness. Tool Hallucination (11): TH occurs when the model use a Statement Fabrication (20): SF occurs when the model tool or tool argument that the tool registry does not define, or claims an observation or state that is inconsistent with facts refers to an existing tool or argument under an incorrect name. on the user’s device. For exmaple, in Codex issue 6562 [31], This category concerns names, not argument values. the agent fabricated that it performed verification steps using The 11 TH cases mainly manifest as crash (6) and EIO (3). DevTools, but the user had never granted access to the relevant The same incorrect name can crash when the harness treats it tools and the harness had no corresponding tool-call record. as an unresolved registry lookup (in LangChain and Codex), Another exmaple is CrewAI issue 3154 [39], an agent config- or appear as EIO when the harness returns an error message ured with a WebSearchTool returned content that looks like as output (in Gemini-CLI and CrewAI). web search results, yet the WebSearchTool.run() was RQ2 Finding: The most common triggering behaviors are never invoked. Actually, the LLM fabricates the web search INC (67), UTA (62), and MTC (49). They expose two different results directly instead of calling a real tool. These claims are failure paths. INC and SF often preserve the harness’s surface difficult to distinguish because users must compare the model’s contract, so they mostly surface as silent errors. Other triggering narration with configurations, tool logs, network activity, or behaviors, in contrast, usually violate interaction rules defined LLM

LLM calls a shell tool to run a command that spawns child processes.

User Observation

in the harness. Their symptoms therefore depend on how the harness validates, propagates, retries, or recovers from the behavior. Thus, an AR bug cannot be understood from the LLM behavior alone: the same behavior can surface as different symptoms under different harness handling. This interaction between LLM and harness makes repair-target attribution difficult, because the triggering behavior is produced by the LLM while the observed symptom is shaped by the harness. C. RQ3: Attitudes from Users and Developers Toward Bug Repair

TABLE III D EVELOPER ACTION BY TRIGGERING BEHAVIOR . H-PR = MERGED HARNESS PR; LLM = DEVELOPER ATTRIBUTES THE ISSUE TO THE BACKEND LLM; A MBIG . = DEVELOPER ENGAGES WITHOUT A CLEAR REPAIR TARGET; N O E NG . = NO DEVELOPER ENGAGEMENT.

Trigger

H-PR

LLM

Ambig.

No Eng.

Total

INC UTA MTC IHI SF EmpR CO TH

11 22 18 2 1 1 2 5

22 5 5 0 5 1 1 0

24 28 17 18 11 10 3 5

10 7 9 3 3 0 5 1

67 62 49 23 20 12 11 11

To answer RQ3, we investigate which component each issue discussion targets for repair: the agent harness, the backend LLM, or both. We infer this target from repository activity, including pull requests (PRs) and comments by users and Total 62 39 116 38 255 project developers. We categorize these actions into three groups: harness-side Developer User fixes, model-side fixes, and ambiguous or no engagement. Each issue receives one developer-side code and one userH-PR (62) side code; when multiple signals appear, we apply the rules H-PR (114) in the order described below. For developers, an issue with LLM (39) a linked non-bot PR merged into the project is marked as C-Both (5) D-HarnessPR (abbreviated as H-PR), because a merged PR indicates that the developers modified the harness. Otherwise, C-Harness (49) Ambig. (116) if a developer explicitly mentioned in the issue comments C-LLM (7) that the issue will be handled by improving the backend No Pos. (80) LLM, the issue is marked as D-LLM (abbreviated as LLM). No Eng. (38) An issue with developer participation yet without a clear fix location is marked as D-Ambiguous (abbreviated as Fig. 5. Repair target codes by user and developer. Ribbon width indicates Ambig.); an issue without human developer participation issue count. H-PR = harness PR; C-Both = both layers; C-Harness = harness; is marked as D-NoEngagement (abbreviated as No Eng.). C-LLM = LLM; No Pos. = no user position; Ambig. = ambiguous developer We similarly extract user actions or suggestions from user- position; No Eng. = no developer engagement. created PRs, issue bodies, and comments. We mark issues with user-created PRs as U-HarnessPR (abbreviated as the harness (U-ComplainHarness), 7 (3%) complain about H-PR). If there is no user-created PR, we mark the user the backend LLM (U-ComplainLLM), and 5 (2%) complain complaints that point to the backend LLM, the harness, or about both layers (U-ComplainBoth); the remaining 80 both layers as U-ComplainLLM (abbreviated as C-LLM), issues (31%) describe only the symptom without taking a U-ComplainHarness (abbreviated as C-Harness), or position (U-NoPosition). On the developer axis, 62 issues U-ComplainBoth (abbreviated as C-Both). Finally, issue (24%) end with a merged harness PR (D-HarnessPR), 39 reports that only describe observations without expressing a (15%) have a developer explicitly attributing the failure to clear position are marked as U-NoPosition (abbreviated the model (D-LLM), 116 (46%) have developers engaged as No Pos.). As in RQ1 and RQ2, two human authors by requesting more information or closing a PR, instead of independently annotated the 255 issues following this guidance committing to a fix location (D-Ambiguous), and 38 (15%) and resolved disagreements through discussion. Bots are not have no developer engagement (D-NoEngagement). Overall, treated as developers or users, and PRs or comments created in our corpus, user-side actions or complaints most often target the harness: 168 of 255 issues (66%) include a user-created PR or merged only by bots are excluded from these rules. The Sankey diagram in Figure 5 connects each issue’s user- or a harness-side complaint, while the largest developer-side side actions with its developer-side actions. Each ribbon shows group is ambiguous (116 of 255 issues, 46%). whether a user-side position leads to a consistent or mismatched Considering that user-created PRs clearly indicate a harnessdeveloper-side position, and whether user-engaged issues get side action, we further inspect the user-side H-PR branch in responses from developers. We organize the result in this Fig. 5. Among the 114 issues in which users contributed PRs, way to facilitate RQ3 to analyze each issue from a two-sided only 43 (38%) have been merged, and another 4 (4%) drew perspective, not only the distributions. explicit attribution to the LLM by developers. Only these User and developer positions. Figure 5 shows that, on the 47 issues (41%) reached a clear repair outcome under our user axis, 114 of 255 issues (45%) have user-created PRs for rubric. The remaining issues stall in different ways: developers the agent harness (U-HarnessPR), 49 (19%) complain about may hold a PR open pending more information or further

TABLE IV U SER – DEVELOPER MISMATCHED ISSUES BY TRIGGERING BEHAVIOR .

Trigger

Mismatch

Total

Rate

INC UTA MTC IHI SF EmpR CO TH

10 3 3 0 2 1 0 0

67 62 49 23 20 12 11 11

15% 5% 6% 0% 10% 8% 0% 0%

Total

19

255

7%

improvements, close stale PRs, or leave the issue managed by Bots. As discussed in RQ1, reproducing agent bugs and designing test oracles can be difficult, which may increase the pressure on developers to handle PRs. This partially explains why a harness-side contribution does not always lead to an accepted repair. Mismatches between user and developer. Figure 5 indicates a clear mismatch between users’ suggestions and developers’ repair targets, i.e., a non-trivial number of issues flow from U-HarnessPR or U-ComplainHarness to D-LLM. It suggests a phenomenon that even when users have suggested harness-side repair solutions, developers sometimes still explicitly state that they will resolve the issue by improving the LLM. Table IV further shows that the mismatch concentrates on Task Instruction Non-compliance (INC) and Statement Fabrication (SF), which have a mismatched issue rate greater than 10%. Therefore, we further conduct a case study to understand the users’ and developers’ attitudes in both categories. For INC, Codex 5807 [42] reports that Codex overwrites the user’s changes to a file without requesting permission. The user proposed harness-side suggestions, such as recording and checking the version of files before applying patches from the LLM. However, the OpenAI developer responded that “We’ve tried to do what you’ve proposed... This needs to be addressed at the model level.” Although the user proposed more harnessside workarounds afterward, the developer has not responded as of the time of this paper’s completion. SF also leads to a high mismatching rate. Codex 10828 [43] is a representative issue, during which the agent claimed it was still working on the task, for example, “I’m continuing now..., and will return results only once complete”. But the turn had silently ended when the LLM was generating the response, and subsequent work was never performed. A user proposed a harness-side fix, asking the CLI to notify the user whenever a run stops unexpectedly, while the developer left a comment, stating that there is not much they can do except improve the backend LLM via training. Merged PRs reveal harness-side mitigation patterns. Beyond how developers respond, we examine how merged harness PRs mitigate these behaviors in our corpus. As shown in Table III, merged harness PRs are concentrated in Unexpected

Tool Arguments (22), Message Template Conflict (18), Task Instruction Non-compliance (11), and Tool Hallucination (5). We also conduct a further case study on these four behaviors to summarize the representative mitigation patterns. Unexpected Tool Arguments: The UTA row in Table III contains the largest number of merged harness PRs, and these fixes commonly validate the model’s tool calls before execution. For example, the merged fix for the read_file crash in Gemini-CLI 533 [44] rejects an undefined path. However, when tool contracts are specified through natural-language descriptions or built-in types in the host programming language, it remains challenging to validate the arguments provided by LLMs before execution. Message Template Conflict: The MTC row in Table III also shows many merged harness PRs, often through more tolerant parsing or prompt steering. For example, LangChain 12077 [45] strips the spurious SQLQuery: prefix the model prepends before the query runs, and LangChain 5163 [46] appends a reminder that steers the router to wrap its output in a JSON code block. Task Instruction Non-compliance: The INC row in Table III includes 11 merged harness PRs but also 22 explicit model-side attributions. Its merged harness fixes span three mechanisms. When the model disobeys an instruction related to the user’s customized output format, tolerant parsing mitigates it. For example, in LangChain 11408 [47], where the model was asked for a YES or NO answer but replied with words with similar meanings, the harness loosens the boolean parser to accept “Not relevant (NO)”. When the model violates a mode or policy, the PRs strengthen the prompt. When INC behavior shows up as looping or repeated work, the PRs add runtime guards, such as the infinite-loop protection merged in Gemini-CLI 1484 [48]. For other INC behaviors, the fix is still contested between users and developers, as discussed above. Tool Hallucination: For TH, Table III shows 5 merged harness PRs among 11 issues, and the direct mitigation is to check tool names against the registry. LangChain 34910 [49] raises a clear exception when the model calls a tool that does not exist. As the mismatch cases above show, the fix for statement fabrication is also contested. There are too few merged PRs for the rest behaviors to summarize a representative fixing method. RQ3 Finding. Among our collected issue reports, users prefer taking harness-directed actions or complain about the harness, while developers often respond without a clear repair target or attribute some behaviors to the backend model. Accepted harness PRs mitigate AR bugs through validation, tolerant parsing, prompt steering, registry checks, and runtime guards. However, INC and SF still expose contested repair targets. The results of RQ3 suggest that maintaining agents would benefit from fault-localization support and reproducible tests that distinguish model limitations from harness robustness gaps.

V. D ISCUSSION A. Practical Advice for Users and Developers Our taxonomy may serve as a checklist for the maintenance of agent products. When users report a failure, the symptom taxonomy may guide them to systematically describe what they observed: silent errors, a crash, an error in the output, a retry loop, or a hang. The triggering-behavior taxonomy then helps users and project developers collect the evidence needed for reproduction, such as the model output, tool arguments, context state, or tool-call log that exposed the failure. Finally, the repairtarget taxonomy helps project developers decide whether the issue record supports a harness-side fix, a backend-model fix, or both. In this way, the taxonomy turns vague reports such as “the agent failed” into structured maintenance evidence: what failed, which LLM behavior exposed the failure, and where the repair is justified. Agent project maintainers can build their issue report template and bug-repairing workflow around these three RQs, and users can use the taxonomy to provide more actionable bug reports. B. Research Opportunities for Oracles and Reproduction

and PRs. We cannot validate how users and developers detect, locate, and fix bugs without such information. VII. R ELATED W ORK A. LLM Agents and Harness Engineering Early LLM applications mainly used a model as a singleturn chat or completion engine. Prompting techniques improve this interaction by providing examples [51], teaching models to follow instructions [52], or eliciting step-by-step reasoning [53]. However, many tasks require coordination beyond one response. Multi-agent systems therefore assign different roles to LLMs. AutoGen supports conversable-agent programming [54]. MetaGPT encodes standard operating procedures [55]. ChatDev organizes software-development roles through chat chains [56]. These systems still need an execution substrate when agents must use tools, observe results, and continue working. Loopbased agents provide this substrate. ReAct interleaves reasoning and acting [28]. ToolLLM studies API use [14]. SWE-agent builds an agent-computer interface for software engineering [2]. In such agents, the harness renders prompts, parses actions, dispatches tools, manages context, and coordinates recovery [2], [27], [29]. Our work studies the failure introduced by this model-harness architecture.

The challenges of oracle design and reproduction observed in our study also suggest research opportunities. A test oracle for agents cannot solely rely on the final response, especially B. Testing and Debugging of AI Systems for silent errors and statement fabrication symptoms. Future AI-based systems create maintenance problems that differ work can design trace-based oracles that check whether an from traditional software. ML pipelines accumulate hidden agent’s claimed behavior or state is supported by real trajectory technical debt [57]. Production ML systems require new engirecords. Reproducing AR bugs also needs new support. Existing neering practices [58]. Deep-learning frameworks also expose work on structural testing of agents suggests using traces and distinctive bug patterns [59], [60]. Testing work responds with mocks to make agent behavior reproducible and simplify the input generation [61], cross-backend validation [62], and behavtest environment [50]. AR bugs are suitable for this direction ioral tests [63]. For LLMs, existing work evaluates instruction because their manifestations depend on specific LLM behaviors. following [52], factuality [64], [65], hallucination [66], [67], Beyond replaying user inputs, tests should preserve or mock and faithfulness [68]. These studies motivate our focus on the triggering behavior of LLMs, including the specific tool model behaviors and observable symptoms. Our study further arguments, template-breaking message, or long context that asks where the repair should happen when such behavior is exposed the failure. Our triggering-behavior taxonomy can wrapped by an agent harness. This question also connects to guide which LLM response or interaction state should be bug-report research on report quality [69], bug assignment [70], mocked, so developers can evaluate harness repairs without and misclassification [71]. Agent bugs complicate this line of depending on the backend model to regenerate the bug- work because the repair target may be the backend model or triggering response. the harness code. VI. T HREATS TO VALIDITY First, our study involves manual inspection of bug reports. These subjective steps may be biased because we manually infer symptoms, triggering LLM behaviors, and repair targets from issue reports, which can be incomplete and interpreted by annotators differently. To reduce this threat, two annotators worked independently and resolved conflicting cases through discussion until consensus was reached. Second, it is unclear to what extent our findings can be generalized to other agents. We study 255 issues from four agent projects that are popular and publicly disclose issue trackers and source code of the harness. However, they may not cover agents with different users or harness designs. Expanding the corpus is difficult because many popular agents, such as Claude Code, are closed-source or lack public trackers of issues

VIII. C ONCLUSION This paper studies Agent-Reactive (AR) bugs in agents, i.e., failures whose manifestation depends on both a specific LLM behavior and how the harness handles it. From issues across Codex, Gemini-CLI, LangChain, and CrewAI, we collect the issue reports of 255 AR bugs and analyze their symptoms, triggering LLM behaviors, and repair-target discussions by examining the bug reports and related communications. We summarized 5 symptoms, 8 triggering behaviors of AR bugs, and repair strategies or discussions for each triggering behavior from both the user and developer perspectives. Overall, our findings shed light on reporting, reproducing, diagnosing, and repairing AR bugs, and motivate agent-specific support for test oracles, behavior-preserving reproduction, and harness-versusmodel fault localization.

R EFERENCES [1] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, “Swe-bench: Can language models resolve realworld github issues?” in The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. [2] J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “Swe-agent: Agent-computer interfaces enable automated software engineering,” in Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. [3] S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y. Bisk, D. Fried, U. Alon, and G. Neubig, “Webarena: A realistic web environment for building autonomous agents,” in The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. [4] S. Yao, N. Shinn, P. Razavi, and K. Narasimhan, “τ -bench: A benchmark for tool-agent-user interaction in real-world domains,” CoRR, vol. abs/2406.12045, 2024. [5] S. Ouyang, J. M. Zhang, M. Harman, and M. Wang, “An empirical study of the non-determinism of chatgpt in code generation,” ACM Trans. Softw. Eng. Methodol., vol. 34, no. 2, pp. 42:1–42:28, 2025. [6] N. Islam, R. S. Ayon, D. Thomas, S. Ahmed, and M. Wardat, “When agents fail: A comprehensive study of bugs in LLM agents with automated labeling,” CoRR, vol. abs/2601.15232, 2026. [7] X. Zhu, J. Wu, X. Zhang, T. Li, Y. Mu, J. Zhai, C. Shen, C. Fang, and Y. Liu, “An empirical study of bugs in modern LLM agent frameworks,” CoRR, vol. abs/2602.21806, 2026. [8] M. B. Shah, M. M. Morovati, M. M. Rahman, and F. Khomh, “Characterizing faults in agentic AI: A taxonomy of types, symptoms, and root causes,” CoRR, vol. abs/2603.06847, 2026. [9] X. Zhang, H. Zhang, and S. H. Tan, “Dissecting bug triggers and failure modes in modern agentic frameworks: An empirical study,” CoRR, vol. abs/2604.08906, 2026. [10] X. Liu, H. Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang, S. Zhang, X. Deng, A. Zeng, Z. Du, C. Zhang, S. Shen, T. Zhang, Y. Su, H. Sun, M. Huang, Y. Dong, and J. Tang, “Agentbench: Evaluating llms as agents,” in The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. [11] G. Mialon, C. Fourrier, T. Wolf, Y. LeCun, and T. Scialom, “GAIA: a benchmark for general AI assistants,” in The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. [12] L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, and T. Liu, “A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,” ACM Trans. Inf. Syst., vol. 43, no. 2, pp. 42:1–42:55, 2025. [13] S. Kokane, M. Zhu, T. Awalgaonkar, J. Zhang, T. Hoang, A. Prabhakar, Z. Liu, T. Lan, L. Yang, J. Tan, R. Murthy, W. Yao, Z. Liu, J. C. Niebles, H. Wang, S. Heinecke, C. Xiong, and S. Savarese, “Spectool: A benchmark for characterizing errors in tool-use llms,” CoRR, vol. abs/2411.13547, 2024. [14] Y. Qin, S. Liang, Y. Ye, K. Zhu, L. Yan, Y. Lu, Y. Lin, X. Cong, X. Tang, B. Qian, S. Zhao, L. Hong, R. Tian, R. Xie, J. Zhou, M. Gerstein, D. Li, Z. Liu, and M. Sun, “Toolllm: Facilitating large language models to master 16000+ real-world apis,” in The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. [15] H. Xu, Z. Zhu, L. Pan, Z. Wang, S. Zhu, D. Ma, R. Cao, L. Chen, and K. Yu, “Reducing tool hallucination via reliability alignment,” in Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025, ser. Proceedings of Machine Learning Research, vol. 267. PMLR / OpenReview.net, 2025. [16] J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou, “Instruction-following evaluation for large language models,” CoRR, vol. abs/2311.07911, 2023. [17] N. Skripko, “Instruction-following evaluation in function calling for large language models,” CoRR, vol. abs/2509.18420, 2025. [18] N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, “Lost in the middle: How language models use long contexts,” Trans. Assoc. Comput. Linguistics, vol. 12, pp. 157–173, 2024.

[19] E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, “The oracle problem in software testing: A survey,” IEEE Trans. Software Eng., vol. 41, no. 5, pp. 507–525, 2015. [20] F. Tambon, A. Nikanjam, L. An, F. Khomh, and G. Antoniol, “Silent bugs in deep learning frameworks: an empirical study of keras and tensorflow,” Empir. Softw. Eng., vol. 29, no. 1, p. 10, 2024. [21] Q. Luo, F. Hariri, L. Eloussi, and D. Marinov, “An empirical analysis of flaky tests,” in Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, (FSE-22), Hong Kong, China, November 16 - 22, 2014. ACM, 2014, pp. 643–653. [22] X. Hou, Y. Zhao, Y. Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engineering: A systematic literature review,” ACM Trans. Softw. Eng. Methodol., vol. 33, no. 8, pp. 220:1–220:79, 2024. [23] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, and et al., “Evaluating large language models trained on code,” CoRR, vol. abs/2107.03374, 2021. [24] J. Chen, S. Chen, J. Cao, J. Shen, and S. Cheung, “When llms meet API documentation: Can retrieval augmentation aid code generation just as it helps developers?” CoRR, vol. abs/2503.15231, 2025. [25] S. Chen, C. Xu, J. Chen, J. Cao, J. Wu, and S. Cheung, “Can emulating semantic translation help llms with code translation? A study based on pseudocode,” CoRR, vol. abs/2510.00920, 2025. [26] M. Schäfer, S. Nadi, A. Eghbali, and F. Tip, “An empirical evaluation of using large language models for automated unit test generation,” IEEE Trans. Software Eng., vol. 50, no. 1, pp. 85–105, 2024. [27] Z. Xi, W. Chen, X. Guo, W. He, Y. Ding, and B. H. et al., “The rise and potential of large language model based agents: a survey,” Sci. China Inf. Sci., vol. 68, no. 2, 2025. [28] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao, “React: Synergizing reasoning and acting in language models,” in The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. [29] Q. Meng, Y. Wang, L. Chen, W. Wu, Y. Li, W. Jiang, Q. Wang, C. Lu, Y. Gao, Y. Wu, and Y. Hu, “Agent harness for large language model agents: A survey,” Preprints, 2026, preprint, DOI 10.20944/preprints202604.0428. [30] “OpenAI Codex, issue #5957,” https://github.com/openai/codex/issues/ 5957, gitHub issue. [31] “OpenAI Codex, issue #6562,” https://github.com/openai/codex/issues/ 6562, gitHub issue. [32] “LangChain, issue #1358,” https://github.com/langchain-ai/langchain/ issues/1358, gitHub issue. [33] “Gemini CLI, issue #13292,” https://github.com/google-gemini/ gemini-cli/issues/13292, gitHub issue. [34] “CrewAI, issue #668,” https://github.com/crewAIInc/crewAI/issues/668, gitHub issue. [35] “Gemini CLI, issue #5629,” https://github.com/google-gemini/gemini-cli/ issues/5629, gitHub issue. [36] “OpenAI Codex, issue #4337,” https://github.com/openai/codex/issues/ 4337, gitHub issue. [37] “OpenAI Codex, issue #13491,” https://github.com/openai/codex/issues/ 13491, gitHub issue. [38] “Gemini CLI, issue #3037,” https://github.com/google-gemini/gemini-cli/ issues/3037, gitHub issue. [39] “CrewAI, issue #3154,” https://github.com/crewAIInc/crewAI/issues/ 3154, gitHub issue. [40] “Gemini CLI, issue #7223,” https://github.com/google-gemini/gemini-cli/ issues/7223, gitHub issue. [41] “CrewAI, issue #3843,” https://github.com/crewAIInc/crewAI/issues/ 3843, gitHub issue. [42] “OpenAI Codex, issue #5807,” https://github.com/openai/codex/issues/ 5807, gitHub issue. [43] “OpenAI Codex, issue #10828,” https://github.com/openai/codex/issues/ 10828, gitHub issue. [44] “Gemini CLI, issue #533,” https://github.com/google-gemini/gemini-cli/ issues/533, gitHub issue. [45] “LangChain, issue #12077,” https://github.com/langchain-ai/langchain/ issues/12077, gitHub issue. [46] “LangChain, issue #5163,” https://github.com/langchain-ai/langchain/ issues/5163, gitHub issue. [47] “LangChain, issue #11408,” https://github.com/langchain-ai/langchain/ issues/11408, gitHub issue. [48] “Gemini CLI, issue #1484,” https://github.com/google-gemini/gemini-cli/ issues/1484, gitHub issue.

[49] “LangChain, issue #34910,” https://github.com/langchain-ai/langchain/ issues/34910, gitHub issue. [50] J. Kohl, O. Kruse, Y. Mostafa, A. Luckow, K. Schroer, T. Riedl, R. French, D. Katz, M. P. Luitz, T. Takher, K. E. Friedl, and C. Laurent-Winter, “Automated structural testing of llm-based agents: Methods, framework, and case studies,” in IEEE International Conference on Big Data, BigData 2025, Macau, China, December 8-11, 2025. IEEE, 2025, pp. 1847–1856. [51] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, and P. D. et al., “Language models are few-shot learners,” in Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. [52] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe, “Training language models to follow instructions with human feedback,” in Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, 2022. [53] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V. Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” in Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. [54] Q. Wu, G. Bansal, J. Zhang, Y. Wu, S. Zhang, E. Zhu, B. Li, L. Jiang, X. Zhang, and C. Wang, “Autogen: Enabling next-gen LLM applications via multi-agent conversation framework,” CoRR, vol. abs/2308.08155, 2023. [55] S. Hong, M. Zhuge, J. Chen, X. Zheng, Y. Cheng, J. Wang, C. Zhang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou, C. Ran, L. Xiao, C. Wu, and J. Schmidhuber, “Metagpt: Meta programming for A multi-agent collaborative framework,” in The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. [56] C. Qian, W. Liu, H. Liu, N. Chen, Y. Dang, J. Li, C. Yang, W. Chen, Y. Su, X. Cong, J. Xu, D. Li, Z. Liu, and M. Sun, “Chatdev: Communicative agents for software development,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024. Association for Computational Linguistics, 2024, pp. 15 174–15 186. [57] D. Sculley, G. Holt, D. Golovin, E. Davydov, T. Phillips, D. Ebner, V. Chaudhary, M. Young, J. Crespo, and D. Dennison, “Hidden technical debt in machine learning systems,” in Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, 2015, pp. 2503–2511. [58] S. Amershi, A. Begel, C. Bird, R. DeLine, H. C. Gall, E. Kamar, N. Nagappan, B. Nushi, and T. Zimmermann, “Software engineering for machine learning: a case study,” in Proceedings of the 41st International Conference on Software Engineering: Software Engineering in Practice, ICSE (SEIP) 2019, Montreal, QC, Canada, May 25-31, 2019. IEEE / ACM, 2019, pp. 291–300. [59] M. J. Islam, G. Nguyen, R. Pan, and H. Rajan, “A comprehensive study on deep learning bug characteristics,” in Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT FSE 2019, Tallinn, Estonia, August 26-30, 2019. ACM, 2019, pp. 510–520. [60] N. Humbatova, G. Jahangirova, G. Bavota, V. Riccio, A. Stocco, and P. Tonella, “Taxonomy of real faults in deep learning systems,” in ICSE ’20: 42nd International Conference on Software Engineering, Seoul, South Korea, 27 June - 19 July, 2020. ACM, 2020, pp. 1110–1121. [61] K. Pei, Y. Cao, J. Yang, and S. Jana, “Deepxplore: Automated whitebox testing of deep learning systems,” in Proceedings of the 26th Symposium on Operating Systems Principles, Shanghai, China, October 28-31, 2017. ACM, 2017, pp. 1–18. [62] H. V. Pham, T. Lutellier, W. Qi, and L. Tan, “CRADLE: cross-backend validation to detect and localize bugs in deep learning libraries,” in Proceedings of the 41st International Conference on Software Engineering, ICSE 2019, Montreal, QC, Canada, May 25-31, 2019. IEEE / ACM, 2019, pp. 1027–1038. [63] M. T. Ribeiro, T. Wu, C. Guestrin, and S. Singh, “Beyond accuracy: Behavioral testing of NLP models with checklist,” in Proceedings of the

58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020. Association for Computational Linguistics, 2020, pp. 4902–4912. [64] S. Lin, J. Hilton, and O. Evans, “Truthfulqa: Measuring how models mimic human falsehoods,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022. Association for Computational Linguistics, 2022, pp. 3214–3252. [65] S. Min, K. Krishna, X. Lyu, M. Lewis, W. Yih, P. W. Koh, M. Iyyer, L. Zettlemoyer, and H. Hajishirzi, “Factscore: Fine-grained atomic evaluation of factual precision in long form text generation,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023. Association for Computational Linguistics, 2023, pp. 12 076–12 100. [66] P. Manakul, A. Liusie, and M. J. F. Gales, “Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023. Association for Computational Linguistics, 2023, pp. 9004–9017. [67] J. Li, X. Cheng, X. Zhao, J. Nie, and J. Wen, “Halueval: A largescale hallucination evaluation benchmark for large language models,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023. Association for Computational Linguistics, 2023, pp. 6449–6464. [68] J. Maynez, S. Narayan, B. Bohnet, and R. T. McDonald, “On faithfulness and factuality in abstractive summarization,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020. Association for Computational Linguistics, 2020, pp. 1906–1919. [69] N. Bettenburg, S. Just, A. Schröter, C. Weiss, R. Premraj, and T. Zimmermann, “What makes a good bug report?” in Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of Software Engineering, 2008, Atlanta, Georgia, USA, November 9-14, 2008. ACM, 2008, pp. 308–318. [70] J. Anvik, L. Hiew, and G. C. Murphy, “Who should fix this bug?” in 28th International Conference on Software Engineering (ICSE 2006), Shanghai, China, May 20-28, 2006. ACM, 2006, pp. 361–370. [71] K. Herzig, S. Just, and A. Zeller, “It’s not a bug, it’s a feature: how misclassification impacts bug prediction,” in 35th International Conference on Software Engineering, ICSE ’13, San Francisco, CA, USA, May 18-26, 2013. IEEE Computer Society, 2013, pp. 392–401.