arXiv:2607.04974v1 [cs.SE] 6 Jul 2026
A Comprehensive Study of Implementation Bugs in Multi-modal Agents Suwan Li
Lei Bu
Shangqing Liu
Department of Computer Science Nanjing University Nanjing, China [email protected]
Department of Computer Science Nanjing University Nanjing, China
Department of Software Engineering Nanjing University Nanjing, China
Yile Wang
Guangdong Bai
Department of Computer Science Nanjing University Nanjing, China
Department of Computer Science The City University of Hongkong Hongkong, China
Fuman Xie
Kai Chen
School of Electrical Engineering and Computer Science The University of Queensland Brisbane, Australia
Institute of Information Engineering Chinese Academy of Science Beijing, China
Chang Yue Institute of Information Engineering Chinese Academy of Science Beijing, China
Abstract—Multi-Modal Agents (M-agents), empowered by Large Language Models (LLMs), excel in various complex, open-world scenarios such as autonomous driving and robotics. However, their unique requirements to interact with dynamic and diverse multi-modal environments introduce novel implementation challenges beyond those faced by traditional agents. Outdated perception, untrustworthy planning and inapplicable execution could cause traffic accident and financial loss. Despite growing study on agent issues, there has not been a systematic study focusing on M-agent-specific implementation bugs. To address this gap, we conducted the first systematic study of implementation bugs in M-agents. We collected 34 representative M-agents from diverse sources and, through meticulous filtering, identified 158 M-agent-specific bugs from 1,268 issue reports. Using a top-down strategy, we developed a comprehensive taxonomy that classifies bugs by global symptoms, functionality component-level symptoms, and root causes. We then implemented MATester, an automatic proofof-concept bug identifier by analyzing runtime inter-component outputs. When applied to 12 extra M-agents, MATester successfully covered 61.4% of known open issues and discovered 31 additional bugs, demonstrating the practical usefulness of our study. Our work provides a comprehensive reference and guideline for classification, prevention and fix of M-agent bugs. Index Terms—Multi-modal Agent, Large Language Model, Implementation Bug, Empirical Study
I. Introduction Owing to the rapid advancement of large language models (LLMs), intelligent agents have emerged as a prominent paradigm with capabilities including reasoning [1],
[2], program synthesis [3]–[5], and counseling [6]. Multimodal agents (M-agents) further extend this paradigm by interacting with high-dimensional, real-time and heterogeneous environments, enabling deployment in openworld safety-critical scenarios such as autonomous driving [7]–[9], robotics [10]–[13], and GUI automation [14]–[17]. However, implementation flaws in M-agents’ functional components may lead to severe consequences. For instance, outdated environmental perception in autonomous driving can result in traffic accidents [18]. Unconstrained execution of unverified plans in GUI automation manifests as unexpected behaviors, potentially leading to financial losses [19] and privacy violations [20]. Nevertheless, existing empirical studies primarily focus on single-modal agents, emphasizing aspects such as security [21], compliance [22], code-level implementation defects [23] and general module-level issues [24]. Many works concentrate on specific application domains, like software engineering [25], [26], code generation [27] and search [28], or agent architectures like multi-agents [29] and platform-orchestrated agent [30]. Although they study bugs from multiple perspectives, there is still a lack of systematic investigation dedicated to M-agents, particularly with respect to multi-modal environment interaction bugs. Compared with single-modal agents, M-agents exhibit three distinctive characteristics: (1) perception: M-agents must distill and fuse real-time multi-modal information for LLM processing, whereas single-modal agents handle text
directly; (2) execution: M-agents dynamically adapt action types and parameters to heterogeneous environments, while single-modal agents execute relatively fixed actions; (3) multi-modality: representations across modalities may introduce cross-modal conflicts. These features substantially complicate M-agents and motivate us to conduct the first systematic study of M-agent-specific implementation bugs. Our study proceeds in three stages. 1) To construct a representative dataset, we initially identified 86 Magents from GitHub, top-tier publications, and surveys. We then performed a coarse-grained text-based filtering by examining associated papers and project documentation, followed by a fine-grained manual code inspection to retain only M-agents with complete functionality components. The manual process is conducted by at least two authors separately to mitigate bias. After that, we collected 1,268 raw reports and manually identified 130 M-agent-specific reports covering 158 distinct bugs. 2) We then classified bugs using a top-down strategy across three dimensions: 6 categories of end-user-observable global symptoms, 16 categories of developer-oriented functionality-componentlevel symptoms, and 7 categories of root causes. Inconsistent perception and inapplicable actions caused by mishandling object dynamism, object informativeness and tool application scenarios are reported in 14.7% and 50% of M-agents, respectively. Cross-modal issues span multiple components. 3) Finally, to demonstrate the practical utility of our taxonomy, we implemented MATester, a preliminary bug detector. MATester automatically identifies both symptoms by analyzing runtime inter-component outputs. We applied MATester to 12 extra M-agents, where it successfully uncovered 61.4% of existing open issues and identified 31 unreported bugs. These results validate the effectiveness of MATester and highlight the practical value of our proposed taxonomy. To sum up, this paper has the following three contributions. • To the best of our knowledge, we present the first systematic study of agent-specific implementation bugs in M-agents. Using a step-by-step filtering strategy, we collect 34 M-agents with 1,268 raw issue reports and identify 158 M-agent-specific implementation bugs. This dataset offers a benchmark for future M-agent testing. • We propose a taxonomy of M-agent implementation bugs using a top-down analysis strategy. Specifically, the bugs are categorized along three dimensions: globallevel symptoms, functionality-component-level symptoms, and root causes. The taxonomy provides a guideline for understanding, prevention and fix of M-agent bugs. • We implement MATester, an automated bug detector for M-agents, to validate our proposed concepts. Based on our findings, MATester automatically detects symptoms in 12 extra M-agents. It successfully covers 61.4% of existing open issues and uncovers 31 previously
task
snapshoti
Perceptor
plani
Planner M-agent
environmenti
Multi-modal environment
Executor actioni
environmenti+1
Fig. 1: The structure and workflow of M-agents.
unknown bugs, which demonstrate the applicability of our taxonomy. II. Multi-modal LLM Agents M-agents are built upon LLMs, leverage external tools to accomplish tasks, and interact with multi-modal environments. Owing to the advanced reasoning, crossmodal processing, and tool-use capabilities, they have been deployed in complex real-world scenarios, including autonomous driving [7]–[9], robotics [10]–[13], and GUI automation [14]–[17]. Fig. 1 illustrates the architecture and workflow of a typical M-agent. Multi-modal environments often contain dynamic and diverse information that LLMs cannot directly process, so the Perceptor firstly distills it into compact snapshot. The snapshot is forwarded to the Planner, which generates plans. These plans are subsequently transferred into concrete actions and executed by the Executor. This procedure continues iteratively until the task is complete. Because the workflow is sequential, failures in any components can propagate downstream, ultimately causing global failure. For clarity of presentation, we denote the intercomponent outputs as snapshot, plan, and action, using subscripts to distinguish interaction rounds. Specifically, snapshoti denotes the Perceptor’s output in round i. The labels environmenti and environmenti+1 denote the environment state before and after executing actioni , respectively. III. Study Design A. Overview The overview of this study is shown in Fig. 2. To investigate M-agent bugs, we curated an up-to-date collection of M-agents, gathered their issue reports, and focused on agent-specific bugs (Section III-B). We constructed a top-down taxonomy analyzing symptoms from a global (end-user) and a functionality-component (developer) perspective, and identified root causes to guide future development (Section III-C). Finally, we formulated research questions on the distribution, relationships and applicability of the taxonomy, and implemented MATester to validate our findings (Section III-D).
Data collection (3.2)
Issue Taxonomy (3.3) Crash
Github 34 M-agents
Misb.
Planner
Misinf.
Publication
Survey
Perceptor
412 issuerelated reports
……
MATester
Corn-. Error-.
instrumented M-agents
Conc. Executor
Glob.
Func.
…… Cause
expanded datasets
Task Runner Glob. Analyzer
RQ2: Relationship between symptoms and root causes
R1: It claims to be implemented on a Large Language Model. R2: It should be able to process multi-modal environmental information. R3: It is an agent, at least taking actions to achieve its goal.
Func. Analyzer
Research Questions (3.4) RQ1: Distribution of symptoms and root causes
TABLE II: Basic requirements of a M-agent. Requirements of a M-agent
RQ3: Applicability of the taxonomy
Fig. 2: Overview of the analysis. TABLE I: Keywords used for searching. Type
Large Language Model
Multi-Modal
Agent
Keywords
LLM, Large Language Model, Language Model, ChatGPT, AI
Multi-modal, Multi modal, Visual, Vision, Audio, Speech
Agent, Embodied, Embody, Robot
B. Data Collection and Processing To the best of our knowledge, there is currently no comprehensive and up-to-date list of M-agents. Prior to studying bugs, we therefore conducted a systematic search to curate a representative set of M-agents from multiple sources. We then collected issue reports from their code repositories using automated keyword-based searches, followed by rigorous manual inspection and labeling to identify true issue reports. 1) M-agents Collection: To collect M-agents developed in both industrial and academic settings, we gathered candidates from three sources: GitHub repositories, toptier publications, and recent survey papers. In particular, the survey by [31] provides a comprehensive list of Magents published prior to February 2024. Given that more than two years have elapsed since the publication of this survey, we further expanded the collection by systematically searching recent publications in leading conferences and journals, including ACL, CHI, CVPR, ICCV, NeurIPS, ICLR, ICSE, FSE, ISSTA, TPAMI, JMLR, TSE and TOSEM, covering the period from 2023 to 2025. All results were collected before December 30, 2025. Following prior studies [23], [32], we employed the Quasi-Gold Standard (QGS) approach [33] via automated keyword searches, snowballing, and manual screening. As shown in Table I, keywords on LLMs, multi-modality, and agents were used for initial selection from GitHub and top-tier venues. GitHub repositories with less than 5 stars or forks were excluded. For publications, we applied both backward and forward snowballing [34] to capture overlooked related work. After automated screening, we performed coarsegrained text-based manual filtering by examining topics, descriptions, and README files of GitHub repositories, and titles, keywords, and abstracts of papers, against the criteria in Table II. Specifically, we excluded: (1) tools that
evaluate M-agents rather than implement an M-agent; (2) claimed conversational agents, functioning similarly to LLMs; and (3) agents that accept only text input. This yielded 84 M-agents: 25 from GitHub, 31 from surveys, and 30 from top-tier conferences. Finally, we conducted fine-grained code-based manual filtering to retain runnable M-agents with complete functionality modules. We excluded candidates: (1) lacking an implementation artifact; (2) missing essential files affecting execution; (3) relying on unavailable dependencies; (4) lacking core functionality modules; or (5) whose code semantics violate Table II. This yielded 46 M-agents. Three-fourths (34) of M-agents were used to construct the taxonomy, while the remaining one-fourth (12) was used to evaluate its extensibility. 2) Bugs Collection: Following prior empirical studies [35]–[37], we collected closed issues and merged pull requests from selected M-agents’ GitHub repositories. To balance issue numbers across repositories with varying activity levels, we applied differentiated strategies. For recently created repositories with fewer than 50 closed reports, or repositories inactive for over a year, we also included open reports. For actively maintained repositories, we collected reports within a 15-month window [37]. This yielded a raw dataset of 1,268 items. To ensure that the issues that we collected are true bugs, we adopted a semi-automatic two-step filtering process. We first applied an automated filter by searching for bugrelevant keywords, namely “error”, “fix”, “bug”, “issue”, “fault”, “wrong”, “mistake” and “fail”, within issue or pull request titles and tags. This step yielded 263 candidate issues. To complement the automated filtering, a manual inspection of the remaining issues is conducted. Two authors independently reviewed the titles and contents of these issues and reached agreement on 96.5% of them. Disagreements were subsequently resolved through discussion to reach consensus. Through this process, we identified an additional 149 bug-related issues, resulting in a total of 412 bug-related issues for further analysis. 3) Classification and Labeling: We classified bugs in two stages. First, we categorized each issue by bug type, then annotated agent-specific implementation bugs with global- and functionality-component-level symptoms and root causes, using prior studies [35]–[40] as an initial coding framework. The first author reviewed 11.0% of issues to validate and refine the taxonomy. All authors then met to consolidate the taxonomy. A second author independently re-labeled the same 11.0% subset, achieving an inter-rater agreement of 0.42, measured using Cohen’s Kappa [41]. After resolving disagreements, both authors
labeled the remainder independently (Kappa = 0.84), and a final consensus meeting resolved remaining discrepancies. The labeling produced 130 implementation issue reports covering 158 unique bugs, 6 global-level symptoms, 16 functionality-component-level symptoms, and 7 root causes. C. Bug Taxonomy 1) Global Level Symptoms: We summarized the globallevel symptoms, which characterize the observable behaviors of M-agents from an end-user perspective when bugs occur. The identified global-level symptoms are described below. • Report error and crash (Crash). This symptom characterizes the unexpected termination of M-agents during task execution, typically accompanied by error reports. • Misbehave during execution without crash (Misbehave). This symptom describes incorrect agent behavior without causing a system crash. Specifically, the agent performs actions that are irrelevant to the given task or inapplicable to the environment, ultimately resulting in task failure. • Can not respond (Unrespond). This symptom indicates that M-agents become unresponsive for an extended period. • Mis-notify the exit (Mis-signal). This symptom characterizes the proactive exit of M-agents before the task completion. • Behave inconsistently under the same setting (Diverge). This symptom characterizes inconsistent behavior, whereby an M-agent exhibits different execution outcomes when performing the same task under identical environmental conditions. Notably, this is a resultoriented rather than an action-oriented symptom: if an M-agent follows different actions to complete the same task but lead to identical outcomes, the behavior is not considered inconsistent. • User interface provides wrong information (Misinform). This symptom denotes inconsistency between the Magent’s behavior and the feedback it provides to users. 2) Functionality Component Level Symptoms: Since an M-agent comprises three core functionality components, functionality-component-level symptoms are correspondingly categorized according to these components. Perceptor. The Perceptor transforms the multi-modal environment information into a snapshot for the Planner’s LLM. When it fails, the following symptoms arise. • The snapshot does not exist (Missing-S). • The snapshot has wrong format (Malformed-S). Magents’ snapshots are typically multi-modal or nontextual. This inherent complexity makes them more susceptible to formatting issues. Moreover, non-textual snapshots are typically stored as files. Therefore, bugs related to file names and file paths may also lead to this symptom.
The snapshot’s content is wrong or incomplete compared with the environment (Wrong-S). The environmental heterogeneity may affect the Perceptor’s ability to recognize specific objects (incomplete), while the dynamic feature may cause outdated snapshots (wrong). We further split it into two sub-symptoms “same (SM) / different (DM)” according to modality consistency of snapshots and environments. • Available objects are labeled in the snapshot, but labels are wrong or incomplete (Mislabeled-S). Due to the diversity of objects in multi-modal environments, Perceptors normally label accessible objects to assist the LLMs in identifying them. However, these labels may be inaccurate: accessible objects are missed (incomplete), or environmental features, like the background, are marked (wrong). • Two environments are the same, but their snapshots are different (Inconsistent-S). Environments are the same if they contain the same objects with the same position, outlook and available actions. Due to the dynamic environment and instable agent behavior, the Perceptor may generate different snapshots for the same environment. We further split it into two sub-symptoms “same (SM) / different (DM)” according to modality consistency of snapshots and environments. •
Planner. The Planner constructs a prompt from the snapshot and the task, then queries the LLM for the next plan. Bugs may arise in both prompt construction and plan generation. The prompt misses necessary information (Wrong-PR). A complete prompt should include at least the following information: environmental information (the snapshot), task specification, available tools and expected output formats. • The plan does not exist (Missing-P). • The plan has wrong format (Malformed-P). The plan should satisfy the output format specified in the prompt. • The plan is internally contradictory (Conflict-P). It mainly exhibits two types of forms. The LLM may “act” (i.e., invoke available tools) differently from it “thinks” (i.e., expresses its thought in natural language). Tool invocations may also exhibit parameter or behavior conflicts. • The plan does not satisfy constraints given by the snapshot (Inapplicable-P). The snapshots are often multi-modal or non-textual, which brings challenges to object identification and action extraction. Therefore, LLM-generated plans may contain wrong actions or unsupported parameters. We further split it into sub-symptoms “same (SM) /different (DM)” based on modality consistency of plans and snapshots. • The plan is not related to the task (Irrelevant-P). The plan should facilitate task completion. We further classify it into two types, representing “when” (WrongStop) and “what” (Wrong-Step) decisions respectively. •
- The plan cannot correctly decide the task termination - The plan cannot lead to task completion • The tasks, snapshots and prompts are nearly the same, but their plans are different and finally result in different execution results (Inconsistent-P). Multi-modal inputs could amplify the inherent uncertainty of LLMs [42], making them likely to produce different plans given the same input. However, the randomness of the plan should not affect the execution results. This symptom is classified into sub-symptoms “same (SM) /different (DM)” based on modality consistency of plans and snapshots. Executor. The Executor component translates textual plans into concrete actions by invoking off-the-shelf or custom-defined APIs to advance task execution. It is the Executor’s responsibility to decide when and how to interact with the multi-modal and dynamic environment. When the Executor encounters bugs, they manifest as the following symptoms. • The action does not exist (Missing-A). This symptom is identified if M-agents do nothing following correct plans. • The action is different from the plan (Wrong-A). Given a correct plan, the action should follow its instruction. • The action is not applicable to the environment (Inapplicable-A). Since the environment is constantly changing, plans made on outdated snapshots may not be applicable to the current environment. It is identified if the action follows the plan’s instruction, but is not applicable to the environment. It is classified into two types. - The plan is wrong, the action sticks to the plan and causes crashes (Wrong-P&A). Normally, an action following a wrong plan is not identified as an Executorrelated bug unless the M-agent crashes, because only the wrong action is the direct cause of Crash. - The plan and the invoked tool are correct, but the tool runs unexpectedly (Wrong-API). • The reflection on the action’s result is wrong (WrongR). This symptom is identified if the reflection of the action’s influence on the environment is different from the fact. 3) Root Cause: Root causes constitute another category of M-agent analysis in this work. Below, we only introduce the major categories that are independent of the scenario. The specific root causes will be introduced in Section IV. • Concurrency. It refers to situations in which multiple components access shared resources without proper synchronization. We observed two representative scenarios. The first is a data race, where the environment undergoes internal changes (i.e., a write to the environment) after snapshoti is captured (i.e., a read of the environment), rendering actioni inapplicable to the updated environment. In this case, no happens-before relationship exists between the environment write and
the snapshot read. The second scenario is an atomicity violation, in which snapshoti+1 is captured after actioni is executed, but the environment has not yet transitioned to a stable state environmenti+1 under the effect of actioni . Consequently, snapshoti+1 does not accurately represent environmenti+1 , thereby adversely affecting subsequent stages of the agent’s workflow. • Corner-case gaps. It arises when functionality modules fail to correctly handle edge cases, which typically stem from rare yet valid inputs. Based on the locations of corner cases, we further categorize the overlooked factors into task processing, plan comprehension, snapshot generation, LLM query, and tool understanding and invocation. For example, the Perceptor may ignore “dynamic objects”, and the Executor may invoke tools with limited “applicable scenarios” and operate on unauthorized objects. The detailed classification is introduced and analyzed in Section IV. • Error-case gaps. It refers to situations in which functionality modules lack robustness to minor errors, allowing small deviations to propagate into more severe failures. Similar to corner-case gaps, the overlooked factors are also divided under different phases: plan comprehension, snapshot generation, LLM query, and tool understanding and invocation. For example, facing an “unexpected environment”, the invoked tool may access unavailable objects or perform unsupported actions, causing Misbehave or Crash. The detailed classification is presented in Section IV. • Lack of persistent memories for task execution experience (No persistent memory). It captures bugs arising from the absence of a memory mechanism in M-agents that conduct user-specified tasks, which adversely affects behavioral consistency. When equipped with memory, M-agents can leverage prior successes and failures to inform subsequent decision making. In contrast, a taskoriented M-agent treats repeated instances of the same task as entirely new, independent executions. Moreover, LLMs inherently exhibit a degree of stochasticity, so the lack of memory further affects the reproduction of previously successful behaviors. • LLM’s limitation. LLMs exhibit inherent limitations, including hallucination, inconsistency, and instability. In M-LLMs, such inconsistency may manifest across different modalities. • Unsuitable LLM parameters (Bad parameters). The behavior of the underlying LLM is highly sensitive to configuration parameters, including prompt design, temperature setting, and maximum token limits. Prompts that omit critical information or exceed model constraints can significantly impair plan generation. An inappropriate temperature setting may lead to overly rigid or constraint-violating behaviors, while an insufficient maximum token limit can result in truncated or incomplete outputs. • Incorrect code semantics (Incorrect semantics). We ob-
TABLE III: Distribution of bug types. The first type is reported in the format of “issue number / bug number”.
TABLE IV: Distribution of global level symptoms and percentage of M-agents with these symptoms.
Impl. bug (specif.)
Impl. bug (irrel.)
Depend -ency bug
Cross -system bug
Crash
Misb.
Unresp.
Mis-sig.
Diverge
Usage issue
Version bug
Import bug
Not a bug
Glob.
Config. bug
Num.
83
58
5
3
5
4
130 / 158
28
57
28
45
12
24
25
63
Perc.
38.2%
35.3%
8.8%
8.8%
8.8%
8.8%
served two representative forms of such errors. Variables associated with the i-th interaction round are incorrectly updated for the (i + 1)-th round. In anther case, the implemented code semantics deviate substantially from the intended behavior implied by comments or function names. This root cause does not follow a systematic pattern and is likely attributable to careless or erroneous implementations. D. Research Questions Our study aims to answer the following research questions. RQ1: What is the proportion and the distributions of agent-specific implementation bugs? It characterizes the reliability landscape of M-agents by quantifying how frequently agent-specific bugs occur and how they are distributed, revealing that such problems are systematic rather than isolated. RQ2: What is the relationship between these symptoms? It analyzes how agent-specific bugs manifest and propagate across functionality components, revealing symptom cooccurrence and causal relationships to enable more effective diagnosis and repairing strategies. RQ3: Is our taxonomy applicable to undiscovered bugs? It evaluates whether the taxonomy generalizes to undiscovered bugs, validating it as a predictive and diagnostic tool rather than a purely descriptive framework. IV. Experimental Results A. RQ1: Distribution We collected 412 reports and classified them into various bug types. The agent-specific implementation bugs are further analyzed for global symptoms, functionality component-level symptoms, and root causes. Distribution of bug types. Table III shows the distribution of bug types. Implementation bugs account for the largest proportion of about 38.3%. They are further classified into agent-specific and irrelevant bugs. Irrelevant bugs refer to bugs of assistive functionalities, such as dataset processing and log management. We observe that the number of issues containing agent-specific implementation bugs is 4.6 times that of irrelevant implementation bugs. The first column also reports that there are 158 bugs found in agent-specific implementation issue reports. Dependency, configuration, version, and other nonimplementation-bugs account for the remaining categories and are not directly linked to M-agent behavior. Symptoms and root causes are thus identified solely from agentspecific implementation bugs.
Misinf.
FINDING 1: Agent-specific implementation bugs account for the largest proportion among all the bug types. They are directly related to M-agent’s behavior, and potentially reveal the issues in its functionality components.
Distribution of global level symptoms of agent-specific implementation bugs. Table IV shows the distribution of global level symptoms. Crash is the most common, accounting for over half of all bugs. Misbehave follows at 36.7%. Runtime symptoms (Unrespond, Mis-signal) and functionality symptoms (Diverge, Misinform) are also observed. Compared with single-modal agents, M-agents are more deployed in real-life scenarios, causing more severe consequences. Crash directly interrupts tasks and is observed in autonomous driving, GUI automation, and robotics, leading to outcomes like traffic accidents. Misbehave can cause financial leakage or privacy violations (e.g., inserting wrong account IDs or phone numbers into text boxes). Diverge and Misinform indicate unstable or untrustworthy agent behavior. FINDING 2: Crash and Misbehave are the most common global symptoms, accounting for 52.5% and 36.7% of Magent-specific bugs, respectively. Crash causes traffic accidents in autonomous driving. Misbehave manifests as unexpected tokens inserted into text boxes of mobile apps, further causing financial or privacy leakage.
Distribution of functionality component level symptoms in agent-specific implementation bugs. Table V shows the distribution of functionality component level symptoms classified by different functionality components, along with the percentage of M-agents with specific symptoms. The multifacetedness, complexity and dynamism of the multi-modal environments pose challenges to acquire complete, accurate and real-time snapshots and identify interactive objects, resulting in the prevalence of Wrong-S and Mislabeled-S. Wrong-S is discovered in 9 cases and revealed in 14.7% of M-agents. Mislabeled-S is found in 10 cases. Among the two subcategories of Wrong-S, DM is supported by more cases than SM because cross-modality environmental perception is more challenging. In addition, even SM perception in M-agents is more prone to errors due to the involvement of non-textual modality object identification and understanding. Identifying and labeling objects in M-agents are necessary especially when the environments contain complex backgrounds or distracting elements, making Mislabeled-S a unique M-agent symptom. Its prevalence stems from M-agents’ failure to recognize dynamically generated objects, which are common in multi-modal environments. M-agents that percept based
TABLE V: Distribution of functionality component level symptoms and percentage of M-agents with these symptoms. The values in Wrong-S, Inconsistent-S, InapplicableP and Inconsistent-P are in the form of total(SM, DM). The values in Irrelevant-P and Inapplicable-A are total(Wrong-Stop, Wrong-Stop) and total(Wrong-P&A, Wrong-API), respectively. Symp.
Perceptor Missing-S
Malformed-S
Wrong-S
Mislabeled-S
Inconsistent-S
10
4 (2, 2)
5.9%
5.9%
Num.
6
8
9 (4, 5)
Perc.
8.8%
8.8%
14.7%
Wrong-PR
Missing-P
Malformed-P
Conflict-P
Inapplicable-P
Irrelevant-P
Inconsistent-P
Num.
5
15
15
2
14 (9, 5)
14 (3, 11)
3 (2, 1)
Perc.
2.9%
14.7%
11.8%
2.9%
23.5%
26.5%
8.8%
Symp.
Planner
Symp.
Executor Missing-A
Wrong-A
Inapplicable-A
Wrong-R
Num.
5
4
40 (19, 21)
4
Perc.
14.7%
11.8%
50.0%
11.8%
solely on static layouts often miss dynamically generated objects. FINDING 3-1: Wrong-S is the most severe Perceptor symptom. Its two subcategories are evenly distributed, indicating that both cross-modality and non-textual intra-modality environment perception are challenging. Due to the diversity of the multi-modal environments, Wrong-S is also common as Perceptors could overlook specific object types.
In the Planner, Missing-P, Malformed-P, InapplicableP, and Irrelevant-P each appear in over 14 cases, with Inapplicable-P and Irrelevant-P affecting over 20% of Magents. The popularity of Inapplicable-P demonstrates the difficulty of understanding the constraints implied in the snapshots. Among the subcategories, SM has more cases than DM, but we also observe that SM planners are more common than DM planners. LLMs in the DM planner must identify operable objects, locate positions, predict action ranges, and generate correct tool-calling parameters. However, the operable objects, positions and action ranges are not explicitly given as prompts, instead, they are implied in the non-textual modality snapshot, bringing challenges to DM planning. LLMs in the SM planner also perform action range prediction and parameter generation despite the summarization of operable objects in the snapshot. Among the subcategories of Irrelevant-P, Wrong-Step is more frequent than WrongStop. Its popularity reflects the struggling of M-agents to decompose the general task into multiple unit steps and identify the task end. FINDING 3-2: Inapplicable-P and Irrelevant-P are the most common symptoms in the Planner. Their popularity demonstrate the difficulty in figuring out the constraints implied in the non-textual modality snapshots, decomposing the task into small steps and judging the termination of the task.
Inapplicable-A dominates the Executor symptoms, ac-
counting for about 76.9%. The popularity of InapplicableA indicates that generating effective and correct actions under complex and dynamic multi-modal environments is difficult. Inapplicable-A is further classified into WrongP&A and Wrong-API. They are almost equally distributed. The popularity of Wrong-P illustrates that many M-agents are designed to follow LLM-generated plans without extra correctness checking. On the other hand, the commonality of wrong-API symptoms arises from the mismatch between APIs and specific objects. Multi-modal environments are heterogeneous, meaning that objects may have different permissions or interaction interfaces. Wrong-APIs may work on specific, common objects and fail on others. FINDING 3-3: Inapplicable-A is the major Executor symptom. Its popularity implies the difficulty to generate environmentcompatible actions. Its subcategory Wrong-P&A reflects that M-agents follow LLM-generated plans without extra checking. Wrong-API demonstrates that self-defined APIs may fail on special objects.
Distribution of root causes in agent-specific implementation bugs. Table VI displays the distribution of root causes and percentage of M-agents with specific root causes. Corner-case gaps and Error-case gaps are the major root causes, exhibited in 40 and 38 reports, respectively. To figure out the major factors that are overlooked by M-agent developers, Fig. 3 further presents the distribution of factors related to Corner-cases and Error-cases, shown by 3a and 3b, respectively. In Corner-case gaps, snapshot generation dominates the distribution, and its common subclasses include file path, object dynamism, file size, and object informativeness. Unlike textual snapshots, non-textual snapshots are mostly saved in files, causing problems when tracking relative file paths or handling LLM-incompatible file sizes. Additionally, while textual snapshots explicitly list operable objects, operable objects in non-textual snapshots are embedded into the background. Perceptors are required but may fail to identify specific objects, such as dynamically generated or running objects, or objects with few functional descriptions. Further more, plan comprehension and tool invocation are the most common components where Errorcases occur, followed by LLM query. Wrong format and unexpected environments are the most frequent factors. Ignorance of wrong format suggests the lacking of format checkers to verify the validness of the LLM generated outputs. The unexpected environments involve various objects with different action ranges, access permissions and subsequent behavior, making it difficult to select appropriate tools and set correct parameters at first try. However, certain tools fail immediately when executed in incompatible environments, yet many M-agents lack explicit mechanisms for handling and analyzing such errors. LLM’s limitation (29.4%) and Incorrect semantics