MCPEvol-Bench: Benchmarking LLM Agent Performance Across Dynamic Evolutions of MCP Servers
Huanxi Liu1,2,3 Kun Hu1 Jiaqi Liao1,2,3 Qiang Wang1,2,3 Pengfei Qian1,2,3 YuanZhao Zhai1,2,3 Dawei Feng1,2,3∗ Bo Ding1,2,3 Huaimin Wang1,2,3 1
College of Computer Science and Technology, National University of Defense Technology 2 State Key Laboratory of Complex & Critical Software Environment 3 National Key Laboratory of Parallel and Distributed Computing
arXiv:2607.14642v1 [cs.AI] 16 Jul 2026
Abstract As Model Context Protocol (MCP) servers emerge as the core infrastructure for connecting LLMs with external tools, existing benchmarks leverage real-world MCP servers to evaluate LLM agents’ tool-using capabilities. However, these benchmarks overlook the continuous evolution of tool interfaces and functionalities within MCP servers, resulting in flawed assessments that fail to capture the agent’s adaptability in changing tool landscapes. To bridge this gap, we introduce MCPEvol-Bench, a novel benchmark for evaluating the task-solving capabilities of LLM agents under dynamic toolset evolution. Inspired by large-scale empirical study, we propose 11 mutation operators to simulate realistic tool evolution within 123 MCP servers. We benchmark 12 state-of-the-art LLMs on multiple versions of MCP servers, revealing that even frontier models struggle to adapt to evolving tools. For instance, GPT-5.4 and Claude-Sonnet-4-6 exhibit performance declines of 13.7% and 14.4% in evolved MCP servers, respectively, accompanied by substantial increases in planning and reasoning errors. These findings highlight the vulnerability of LLM-driven workflows, establishing MCPEvol-Bench as a standard for evaluating agent adaptability in dynamic tool environments.
1
Introduction
Recent advances in large language models (LLMs) have inspired widespread efforts to develop toolusing agents capable of comprehending natural language instructions, planning multi-step workflows, and interacting with external tools to solve complex tasks [1, 2, 3]. These agents have been widely adopted in various real-world domains, such as software development [4], system automation [5], and scientific research [6], where LLM agents address user tasks by chaining multiple tools and orchestrating interdependent operations. Despite significant progress, existing benchmarks for evaluating LLM agents’ tool-using capabilities remain fundamentally limited. Early efforts [7, 8, 9] provided unstable evaluations by assessing agents through interactions with real-world APIs, which suffered from frequent endpoint changes and service deprecations [10]. Recently, the Model Context Protocol (MCP) [11] has emerged as a unified standard for connecting LLMs with external tools, leveraging dynamic server discovery to overcome the inflexibility of prior API-based integration. However, as summarized in Table 1, current MCP-based benchmarks overlook the continuous evolution of tool interfaces and functionalities within MCP servers. Consequently, these benchmarks cannot accurately evaluate the adaptability of LLM agents in preserving the integrity of original workflow under real-world tool changes. To bridge this gap, we introduce MCPEvol-Bench, a novel benchmark for evaluating the task-solving capabilities of LLM agents under dynamic toolset evolution, as illustrated in Figure 1. We first ∗ Correspondence to: [email protected]
40th Conference on Neural Information Processing Systems (NeurIPS 2026).
Figure 1: The framework of MCPEvol-Bench. (A) Empirical Study of MCP Server Evolution: Validates the prevalence of evolution and identifies predominant evolution patterns. (B) LLM-Driven MCP Server Evolution: Simulates developer behavior to adaptively select mutation operators for source code modification, generating evolved toolsets. (C) MCP Agent Evolution Evaluation: Assesses the task performance of LLM agents across multiple versions of MCP servers. conduct a comprehensive empirical study of MCP servers by analyzing temporal changes in remotely hosted servers and version histories of public code repositories. This study reveals the prevalence and patterns of MCP server evolution, with 20.7% of remote servers unavailable and 54.6% of tools in repository deleted or replaced. From the collected repositories, we curate 123 MCP servers with 1,272 tools across diverse domains (e.g., software development, data analytics), and automatically generate cross-server tasks via an LLM-based pipeline. While historical versions of MCP servers provide rich evolution data, frequent failures due to broken dependencies make them unsuitable for benchmarking. To address this, we extract 11 mutation operators covering the Tool, Parameter, and Description levels from observed evolution patterns. Leveraging these operators as prompt instructions, we implement LLM-Driven MCP Server Evolution, where LLMs automatically select appropriate modifications and iteratively modify tools, thereby generating a series of evolved MCP servers. Finally, in MCP Agent Evolution Evaluation, LLM agents execute tasks across multiple versions of MCP servers, measuring their task performance under dynamic toolset evolution. In this research, we evaluate 12 representative LLMs on the benchmark, revealing that most widely used frontier models exhibit significant performance degradation on the evolved MCP servers. Notably, GPT-5.4 and Claude-Sonnet-4-6 suffer 13.7% and 14.4% drops in task fulfillment, respectively. Analysis of agent trajectories shows that evolution primarily increases planning errors by 34.1% and reasoning errors by 35.6%. By analyzing mutation operators, we find that the evolution involving tool additions or modifications severely impair agent performance, whereas removing redundant tools or parameters has negligible effect on original workflow execution. The reliability of our benchmark is validated by the high semantic similarity between simulated evolutions and real-world version updates, along with the strong agreement with human experts in evaluations. Further experiments demonstrate that incorporating reflection, planning and memory modules enhances the adaptability of LLM agent systems, which highlights the critical role of cognitive components in evolving scenarios. Our contributions can be summarized as follows: • We conduct a large-scale empirical study that validates the prevalence of MCP server evolution and uncovers its dominant patterns. Inspired by these patterns, we propose 11 mutation operators that empower LLMs to autonomously and iteratively perform precise MCP server evolution. • We introduce MCPEvol-Bench, a novel benchmark comprising 123 multi-version MCP servers, which evaluates the task-solving capabilities of LLM agents under dynamic toolset evolution. Table 1: Comparison of representative tool-using benchmarks. Benchmark ToolBench [7] SHORTCUTSBENCH [9] APIGen [12] MCP-Flow [13] MCPEval [14] MCP-Bench [15] MCPEvol-Bench (Ours)
# Domain
# Tool/API
MCP Ecosystem
Real-World Toolset
Multi-Tool Collaboration
Dynamic Evolution
19 50 49 10 5 28 9
3451 426 3673 11536 19 250 1272
× × × ✓ ✓ ✓ ✓
✓ ✓ × ✓ ✓ ✓ ✓
✓ ✓ ✓ × ✓ ✓ ✓
× × × × × × ✓
2
Table 2: An example from MCPEvol-BENCH, comparing original and evolved server configurations. Task
Original Servers & Tools
Evolved Servers & Tools
I need to plan a business trip from Beijing to Shanghai for next Wednesday. Please check the availability of high-speed train tickets in the morning, and then add them to my travel planning list in work.md.
Servers: mcp-tasks, 12306-mcp Tools: • tasks_setup ... • tasks_add ... • get-current-date ... • get-station-code-of-citys ... • get-tickets: {“fromStation”: the name or the station_code of the departure location; ...} • ......
Servers: mcp-tasks, 12306-mcp Tools: • tasks_setup (unchanged); • tasks_add (unchanged); • get-current-date (unchanged); • get-station-code-of-citys (unchanged); • get-tickets (changed): {“fromStation”: the name or the station_code, which can be queried by the get-station-code-of-citys... , “minPrice”(new): ... }; • ......
• Our extensive evaluation of 12 state-of-the-art LLMs across 201 challenging tasks reveals persistent weaknesses in dynamically evolving tool-using scenarios, highlighting the necessity of cognitive components to enhance LLM agent adaptability.
2
Related Work
2.1
Tool-Use Benchmarks
Integrating LLMs with external tools has emerged as a critical research direction for extending model capabilities beyond parametric knowledge. Early tool-use benchmarks, such as APIGEN [12] and ToolACE [16], address API data scarcity through automated synthesis, while ToolBench [7] and SHORTCUTSBENCH [9] aggregate numerous APIs from the web to enhance evaluation realism. However, real-world APIs are prone to frequent endpoint modifications and service deprecations, compromising evaluation stability [10]. The emergence of the MCP provides a standardized invocation schema and stable service interfaces for building toolsets, facilitating the development of recent benchmarks like MCP-Bench [15], MCPEval [14], and MCP-Flow [13]. Despite this progress, existing MCP-based benchmarks assume a static tool environment, overlooking the dynamic nature of MCP servers, which undergo continuous evolution through version iterations. This limitation necessitates tool-use benchmarks that evaluate model performance under evolving server states. 2.2
Evaluating LLM Agent Capability
LLM agents are autonomous systems that leverage LLMs as their cognition core for decisionmaking, engaging in multi-turn tool interactions with external environments to accomplish complex user tasks [1]. Existing evaluations typically target distinct agent capabilities: MCP-Bench [15] assesses the handling of ambiguous instructions, ToolBench [7] focuses on multi-step reasoning, and BFCL v4 [17] evaluates proficiency in multi-turn interactions. To assess domain-specific abilities, representative benchmarks such as WebArena [18], SWE-bench [19], and OSWorld [20] evaluate agent performance in web navigation, software development, and operating system environments, respectively. However, these benchmarks operate in static environments, contrasting sharply with the dynamic and evolving nature of real-world settings. This discrepancy creates a critical gap in evaluating the adaptability of LLM agents to continuous environmental changes.
3
Empirical Study of MCP Server Evolution
To validate the existence of real-world MCP server evolution and characterize its dynamics, we conducted a large-scale empirical study from two dimensions: the temporal dynamics of remotely deployed servers and the evolutionary history of publicly released versions. 3.1
MCP Server Collection
Our data acquisition process comprises two parallel streams: remote server monitoring and released repository collection. For remote services, we used Smithery [21], a mainstream MCP server hosting platform as our data source. We used functional keywords such as “browser-automation” to identify 1,869 repositories hosting remotely accessible MCP servers. Each service endpoint was subjected to continuous availability monitoring for three months, with weekly testing intervals. For 3
Figure 2: Availability of remote MCP servers.
Figure 3: The distribution of evolution patterns.
code repositories, we aggregated server names from Smithery, GitHub [22], and ModelScope [23] and traced their publication histories via the NPM package manager [24]. After validation through actual deployment, we curated a dataset of 515 independent MCP servers, containing 9,273 historical versions with 6,436 distinct tools. Detailed collection process is provided in Appendix E.1. 3.2
Analysis of MCP Server Evolution
Remote Service Availability Decay. Figure 2 illustrates the validity of remote MCP servers decreased significantly from 72.7% (week 1) to 52.0% (week 12) over three months. This decline is primarily driven by Bad Request (18.9%) and Internal Server Error (16.7%), both attributable to deployment failures arising from compatibility conflicts in the evolution process. Growth of Toolset Complexity. Following the release sequence, we analyzed the evolution trends of MCP servers in terms of average tool count, parameter count, and description length. As shown in Figure 8, all three metrics exhibit an overall upward trend across version iterations, showing an expansion in toolset scale and increasing structural complexity. Additionally, we find that 54.6% of the initial tools were either modified (32.5%) or deprecated (22.1%) in the latest versions, reflecting the intense evolutionary activity of MCP servers. Analysis of Evolution Patterns. We analyzed tool evolution across version iterations in NPMpublished MCP servers and along the temporal timeline in Smithery-deployed servers. Figure 3 presents the distribution of these evolution patterns, with types such as P1 (TOOL-DESC_CHANGE) for tool description modifications and P2 (TOOL-ADD) for new tool additions; full definition of these patterns are available in Appendix E.3. We observe two main characteristics: 1) Consistency, where the top six patterns (P1–P6) are identical across both data sources, suggesting that these evolution patterns are platform-independent and accurately reflect the prevailing evolution dynamics; and 2) Divergence, where Smithery-deployed servers prefer parameter-level adjustments (P3, P4), in contrast to the tool-level modifications (P1, P2) dominant in NPM repositories. This divergence likely stems from minimizing tool-level changes to ensure remote service continuity. In summary, our findings reveal the current landscape of MCP server evolution and also highlight the critical need for benchmarks that evaluate LLM agents’ adaptability in evolving environments.
4
MCPEvol-Bench Formalization
The agent task with environment feedback is formalized as a Partially Observable Markov Decision Process (POMDP). Each task in our benchmark is defined by a POMDP tuple (U, S, A, O, T, R, Σ), where: U denotes the task instruction space; S is the state space; A is the action space including both reasoning content and tool invocations; O is the observation space containing tool execution results and internal signals; T : S × A → S × O is the state transition function; R : S → [0, 1] is the reward function; Σ = {v1 , v2 , . . . , vn } is the set of available MCP servers. Unlike most benchmarks [15, 25, 14] that rely on dynamic tool retrieval, we maintain fixed MCP server selection for each task to isolate and evaluate the impact of server evolution on agent Sn workflows. Each server vi ∈ Σ exposes a set of tools Ti , defining the complete toolset T = i=1 Ti . A structured tool invocation is written as atool = ⟨vi , tool_name, parameters⟩. The full action space is A = Areasoning ∪ Atools , and the observation space is O = Otools ∪ Ostate . For the workflow of the agent, we adopt a multi-round decision process [26], as detailed in Algorithm 1. The agent initializes its execution trajectory and initial state s0 with a server description function 4
Algorithm 1 Multi-turn Tool Invocation and Observation Require: Task instruction u, maximum steps Tmax , MCP server cluster Σ Ensure: Final answer answer, execution trajectory trajectory 1: function MULTITURNEXECUTE(u, Tmax , Σ) 2: trajectory ← {}, s0 ← U PDATE(u, Φ(Σ)) ▷ Initialize initial trajectory and state 3: for t = 0 to Tmax do 4: (continuet , at ) ← π(st ) ▷ Generate current tool invocation 5: if continuet = False then 6: answer ← at ▷ Obtain the final answer from agent 7: break ▷ Stop if agent signals termination 8: end if 9: ot ← Σ(at ) ▷ Execute tools in MCP servers 10: trajectory ← trajectory ∪ {(at , ot )} ▷ Log tool invocation and observation 11: st+1 ← U PDATE(st , ot ) ▷ Update agent internal state 12: end for 13: return (answer, trajectory) 14: end function Φ : Σ → D. The description space D includes essential metadata such as tool input schemas and semantic descriptions. At each step t, the agent’s policy π generates an action at that integrates reasoning with multiple tool calls, based on the current state st (Line 4). Upon detecting a completion signal in at , the termination signal continuet is set False, and the reasoning content at is returned as the final answer (Lines 5-8). Otherwise, the MCP server cluster Σ executes the action at to obtain the tool output as observation ot (Line 9). This observation is logged into trajectory, and the agent state st is updated accordingly (Lines 10-11). The loop continues Tmax is reached or the task is completed. Finally, the full execution trajectory and answer are returned for evaluation. The prompt used for the agent execution can be found in Appendix H.3.
5
MCPEvol-Bench Construction
We present MCPEvol-Bench, a benchmark designed to evaluate the task-solving capabilities of LLM agents under dynamic toolset evolution, as illustrate in Figure 1. In this section, we first detail our task construction process. Then, we describe how LLMs simulate human developers modifying and updating MCP toolsets. Finally, through multi-round iterative evolution, we generate multi-version MCP servers, thereby establishing a realistic and dynamic environment for evaluating LLM agents. 5.1
Task Synthesis.
MCP Servers. The majority of MCP servers rely on proprietary API keys or third-party integrations, hindering plug-and-play deployment. To address this, we filtered our empirically collected MCP servers with an LLM-assisted process to exclude key-dependent instances. Besides accessibility, we ensure the toolset’s representativeness through structured curation and expert annotation. The resulting collection comprises 123 MCP servers providing 1,272 tools and is taxonomically organized into nine functional categories: Knowledge (13.8%), Research (7.3%), Software Development (19.5%), UI Design (6.5%), Media & Documentation (16.3%), Data & Analytics (8.1%), Business & Commerce (10.6%), AI & Machine Learning (14.6%), Cloud & Infrastructure (3.3%). To ensure reproducibility, we bundled the MCP servers into an NPM package, thereby providing a stable tooling environment. Task Instructions. Constructing high-quality benchmarks for tool-using agents requires transforming real-world MCP servers into realistic and solvable tasks. To this end, we propose an automated task synthesis pipeline. First, we employed random sampling to select 2–5 MCP servers from the same category, simulating cross-server scenarios. Based on these servers, the LLM generated initial task instructions requiring multi-tool collaboration, without explicitly specifying the required servers and tools. Following the methodology of MCP-Bench [15], each task underwent a rigorous twodimensional quality assessment: 1) Solvability, which evaluates whether the task can be successfully completed using the available tools; and 2) Practical Utility, which determines whether the task addresses genuine user needs rather than contrived scenarios. Tasks failing to meet strict thresholds (Solvability ≥ 9.0/10; Utility ≥ 6.0/10) were discarded. Finally, the remaining tasks were validated through LLM agent rollouts, where execution trajectories were assessed via both LLM judging and 5
Figure 4: The 11 evolution mutation operators are categorized into “TOOL”, “PARAM”, and “DESC” levels. These operators have a blue background, with associated evolution patterns (Figure 3) in gray. Arrows denote a hierarchical relationship, where higher-level mutations encompass lower-level ones. human review. Leveraging DeepSeek-Chat [27] as the task synthesis LLM, we constructed a final dataset of 201 high-quality tasks. On average, each sample involves 25.90 distinct tools within its task context, necessitating 4.37 tool invocations and the utilization of 3.76 servers for successful task completion. The prompt used for task synthesis can be found in Appendix H.2. Table 2 presents an example of a synthesized task and its associated MCP servers. 5.2
MCP Server Evolution
Evolution Mutation Operators. Figure 4 presents the 11 MCP server evolution operators derived from our empirical study, organized into three hierarchical levels: TOOL (tool), PARAM (parameter), and DESC (description). These operators are encapsulated as specialized prompts to guide LLMs in performing precise source code modifications for MCP tools. The evolution targets for each level are detailed below: 1) “TOOL”: This level includes four operators, namely Tool Addition, Tool Replacement, Tool Deletion, and Tool Integration. These operators implement functional changes by adding, removing, or substituting tools, and by simultaneously coordinating the functional logic of existing tools to ensure consistency. 2) “PARAM”: Comprising Flexible Expansion, Constraint Mutation, Parameter Pruning, and Interface Refactoring, these operators optimize parameter structures, constraints, data types, and required attributes, while preserving the core functionality of tools. 3) “DESC”: three operators function at this level, which are Tool Description Update, Parameter Description Update, and Joint Description Update. They serve to refine natural language descriptions, eliminating ambiguity and clarifying functional boundaries to ensure efficient agent invocation. The numerical labels (e.g., P1, P2) within each operator correspond to the evolution patterns identified in our empirical study (Figure 3). Collectively, these operators cover all identified evolution patterns and can be composed to model more complex evolutionary scenarios, thereby showing the completeness of our approach. Definitions and prompts of operators are detailed in the Appendix F.3 and H.1. LLM-Driven Mutation Mechanism. Real-world legacy MCP servers often quickly become nonfunctional due to invalid URLs and breaking changes in external dependencies or runtime environments. To address this, we propose an LLM-driven evolution mechanism that simulate realistic MCP server evolution by directly modifying source code repositories. This mechanism follows a structured workflow, starting with Abstract Syntax Tree (AST)-based code anchoring (Appendix F.2), which utilizes the registration syntax (server.tool) to extract snippets of tool definitions and implementations, effectively filtering out irrelevant contextual noise. Based on these anchored segments, an LLM (Claude-Opus-4-5) autonomously selects appropriate mutation operators and mutates the source code, preserving backward compatibility and Figure 5: An example of LLM-Driven MCP original functionality. Upon passing syntax checks, Server Evolution. the mutated code repository is deployed as an MCP server for Functional Integrity Validation. In this phase, an LLM (DeepSeek-Chat) generates multiple test cases to perform comprehensive coverage testing on the mutated tools, validating their compliance with the mutation requirements 6
Table 3: Performance comparison of LLM agents across three evolution stages of MCP configuration, the early stage with the original servers, the middle stage after three evolution iterations, and the late stage after five evolution iterations. Early Stage
Model Qwen3.5-9B Llama-3.3-70B Qwen3.5-27B GPT-4o Gemini-2.5-pro o3 GPT-5.1 Gemma-4-31B-it Claude-Sonnet-4-5-thinking GPT-5.4 Claude-Sonnet-4-6 Claude-Opus-4-6
Middle Stage
Late Stage
x Task Fulfillment
x Planning Effectiveness
x Task Fulfillment
x Planning Effectiveness
x Task Fulfillment
x Planning Effectiveness
3.38 3.84 4.46 5.24 5.73 5.79 6.28 5.05 6.40 7.23 7.22 7.15
4.89 3.56 5.57 2.85 4.62 3.79 4.35 6.67 6.17 5.71 6.76 6.70
3.21 3.88 4.55 5.03 5.20 5.23 5.94 5.19 6.13 6.74 6.61 6.93
4.45 3.74 5.97 2.74 4.03 3.21 3.43 6.69 5.56 5.52 6.42 6.18
3.41 3.93 4.13 4.43 4.96 5.07 5.32 5.16 5.72 6.24 6.18 6.77
4.86 3.94 5.12 2.78 3.71 3.10 3.50 6.68 4.98 3.87 6.17 6.03
ECS ↑ 3.20 3.56 3.78 3.80 3.84 4.24 4.32 4.45 4.52 5.20 5.22 6.09
and ensuring functional correctness. This evolution process can be iterated over multiple rounds, with a single mutation operator applied per round, ultimately generating multi-version MCP servers. Figure 5 illustrates this process, where the LLM applies Operator 5 (O5) to augment the get-tickets tool with minPrice and maxPrice parameters. This enhancement enables users to filter results by fare range and is subsequently verified through multi-case validation.
6
Experiments
6.1
Experimental Setup
Benchmark Configurations. To evaluate LLM agents’ task-solving capabilities under tool evolution, MCPEvol-Bench pairs each task with three MCP server configurations representing different evolution stages: Early Stage (original, non-evolved server), Middle Stage (evolved for 3 rounds), and Late Stage (evolved for 5 rounds). Prior studies [7, 25] show that the primary source for agent failures is retrieving wrong task-required tools. To analyze the impact of server evolution, we bypass the dynamic retrieval process and employ a set of fixed candidate MCP servers to support task completion. Evaluated Models. We evaluated 12 representative LLMs, including commercial models GPT5.4 [28], GPT-5.1 [29], GPT-4o [30], Claude-Opus-4-6 [31], Claude-Sonnet-4-6 [32], and Gemini-2.5Pro [33]; long-reasoning models o3 [34] and Claude-Sonnet-4-5-thinking [35]; as well as open-source models Llama-3.3-70B [36], Gemma-4-31B-it [37], Qwen3.5-27B, and Qwen3.5-9B[38]. Evaluation Method and Metrics. We adopt the evaluation framework of MCP-Bench [15], leveraging the robustness of rubric-based LLM judges [39, 40, 15]. LLM agents are assessed on a 1–10 scale with two dimensions: Task Fulfillment and Planning Effectiveness. Evaluations are grounded solely in observable evidence from the task instruction, final answer, and execution trajectory. By default, the judge model used here is DeepSeek-Chat [27]. To further evaluate the model’s adaptability, we propose the Evolutionary Competency Score (ECS), which explicitly integrates performance with TF N cross-version stability. Specifically, for each task u ∈ U, let {Su,i }i=1 be the Task Fulfillment scores at i-th version. The ECS is defined as: 1 X TF N TF N ECS = µ({Su,i }i=1 ) − σ({Su,i }i=1 ) (1) |U| u∈U
where N = 3, σ and µ denote the standard deviation and mean, respectively. The effectiveness of ECS metric and other evaluation details are provided in Appendix D.1. 6.2
Main Results
Table 3 reports the task performance of LLM agents across three MCP server evolution stages. The evolution of MCP servers induces substantial performance degradation and reduced execution efficiency across advanced LLMs. For instance, the task fulfillment scores for GPT-5.4 and Claude-Sonnet-4-6 exhibited a consistent decline from 7.23 and 7.22 in the early stage to 6.24 and 6.18 in the late stage, corresponding to degradation of 13.7% and 14.4%, respectively. Concurrently, 7
(b) Evolution Operator Impact. (a) Error Distribution. Figure 6: Diagnostic Analysis of MCP Server Dynamic Evolution. (a) Universal increase in error types across evolution stages. (b) The impact of each evolution operator on task fulfillment metric. the execution efficiency of LLM agents decreases universally; notably, Claude outperformed the GPT series, which suffered a sharp drop in planning effectiveness, as evidenced by the fall in GPT-5.4’s score from the initial 5.71 to 3.87. Cross-version performance stability is crucial for the reliability of LLM agents. Although ClaudeOpus-4-6, Claude-Sonnet-4-6, and GPT-5.4 exhibited comparable scores on the original server, only Claude-Opus maintained robust workflow execution during server evolution, achieving the highest ECS of 6.09. In contrast, the other two models showed significant performance degradation on the evolved MCP servers. Similarly, while GPT-5.1 started with a higher task fulfillment score of 6.28 compared to Gemma-4-31B-it’s 5.06, Gemma exhibited superior stability throughout subsequent evolution rounds, ultimately surpassing GPT-5.1 with an ECS of 4.45 versus 4.32. Models with limited capabilities are insensitive to environmental changes. For instance, Qwen3.59B and Llama-3.3-70B exhibited consistently low task fulfillment scores (<4.0), reflecting a failure to execute expected workflows. The ECS metric effectively captures this by penalizing low performance despite minimal variance, clearly separating true adaptability from mere invariance. 6.3
In-depth Analysis
Our analysis focus on three frontier models: Claude-Opus-4-6, Claude-Sonnet-4-6, and GPT-5.4. Impact of Evolution on Agent Behavior. We assessed the impact of evolution on agent behavior by analyzing the error distribution in task trajectories. Building upon prior work [7, 41, 42], we identified six primary categories of failures: 1) Syntax Violation: violations of the tool-calling protocol, such as invalid parameter names or schema violations; 2) Tool Misalignment: selection of inappropriate tools that fail to align with the user’s intent or current sub-task; 3) Execution Error: provision of semantically incorrect parameter values; 4) Planning Error: errors in multi-step workflow orchestration, such as missing preconditions, infinite loops, or premature termination; 5) Reasoning Error: failure to correctly interpret or react to tool outputs; and 6) Redundancy: inefficient execution paths by repeated identical calls or unnecessary actions. Figure 6 (a) shows the distribution of error types on MCP servers at various stages of evolution. It is observed that MCP server evolution leads to increased erroneous behaviors in LLM agents. Specifically, the increase in agent errors was concentrated in reasoning and planning errors, which rely heavily on context understanding, rising by 34.1% and 35.6%, respectively. Conversely, errors from tool misalignment and syntax violations did not increase significantly. This suggests that the context drift introduced by the evolution of MCP servers did not impair the model’s intrinsic tool-calling capability, but rather disrupted its previously stable workflow, causing LLM agents to generate many redundant errors through trial-and-error. Impact of Evolution Mutation Operators. We recorded the evolution operators selected by LLM developers for MCP servers over multiple rounds of evolution. By comparing task fulfillment scores of the same task at different evolution stages, we analyzed each operator’s impact on task completion. Figure 6 (b) presents the operator-specific contributions, revealing three key insights: 1) Evolution operations predominantly degrade LLM agents’ performance. The majority of operators (9 out of 11) exhibited a negative impact, showing that the evolution process tends to disrupt established workflows. 2) Modifying tool descriptions and adding new tools resulted in the most negative effect. Operators such as O1 (Tool Addition, -0.96), O4 (Tool Integration, -0.90), and O9 (Tool Description Update, -0.81) substantially lowered scores. Such extensive changes often disrupt the agent’s tool 8
Task: I need to plan a business trip from Beijing to Shanghai for next Wednesday. Please check the availability of high-speed train tickets in the morning, and then add them to my travel planning list in work.md.
Success with Original MCP Servers
Fail with Evolved MCP Servers
Action 1: get-current-date(Wednesday) Observation 1: Time: 2026-04-15 Action 2: get-tickets(fromStation=Beijing, toStation= Shanghai, date=2026-04-15, ...) Tickets of Observation 2: Available Trains: All Stations ✓ • G547 (06:18–12:11): VNP → AOH ... • G1 (06:30–11:24): VNP → AOH ... • G3 (06:52–11:33): VNP → SSH ... • ...... Action 3: ...... Observation 3: (Task Fulfillment: 10)
(same Action 1 and Observation 1) Action 2: get-station-code-of-citys([Beijing, Shanghai]) Observation 2: [Beijing: [VNP, BJP, BXP, VAP], Shanghai:[SHH, AOH, SNH]] Action 3: get-tickets(fromStation= VNP , toStation= SHH , date=2026-04-15, ...) Tickets of Observation 3: Available Trains: Two Stations ✗ • G3 (06:52–11:33): VNP → SSH ... • G5 (07:42–12:32): VNP → SSH ... • ......
(Task Fulfillment: 5)
Figure 7: Case Study: Impact of MCP Server Evolution. Table 2 lists the original and evolved toolset configurations. Areas highlighted in yellow show changes compared to the original workflow. orchestration logic, thereby hindering execution accuracy. 3) Subtractive operations are relatively neutral. Operators for removal or pruning had minimal negative or positive effects, as seen in O2 (Tool Replacement, +0.10), O7 (Parameters Pruning, +0.08), and O3 (Tool Removal, -0.13). When executing these updates, developers usually eliminate redundancies while preserving core MCP server functionalities, which reduces the workflow complexity and facilitates task completion. Experiments on real evolution and agent modules. We curated a subset of 50 historical versions of real-world MCP servers, covering 86 tasks, by excluding undeployable or non-functional instances. Table 5 shows apparent declines in task fulfillment across all models. This confirms that our Table 4: Performance gains from agent modules. x x simulated evolution-induced issues also persist in Planning Task Method Effectiveness Fulfillment real-world scenarios. Furthermore, we enhanced GPT-5.4 with agent modules, including reflection, Vanilla 6.24 3.87 + Reflection [43] 6.60 4.12 planning, and memory, and tested them on the late + Plan [44] 6.52 4.23 stage of MCP Servers. Table 4 demonstrates that + Memory [45] 6.71 5.04 these methods effectively alleviated performance drops due to tool evolution, outperforming the vanilla model in both evaluation metrics. Reliability of MCPEvol-Bench. We compared the cosine similarity of code change embeddings for a single mutation or version update under three settings: Evol vs. Real, which measures the similarity between our simulated evolutions and real-world updates; Real vs. Real, which compares consecutive real-world versions; and Random, a baseline established by randomly pairing updates from different servers. As shown in Table 6, while Evol vs. Real scored slightly lower than Real vs. Real on the general BGE-M3 model, it surpassed Real vs. Real on the code-specific CodeT5 and StarCoder2 models. This demonstrates that our method effectively simulates human developer behavior. Furthermore, as shown in Table 8, the high consistency between human experts and LLMjudges in ranking trajectories across evolution stages validated the effectiveness of our assessment. Table 5: Performance drop on historical MCP server.
6.4
Table 6: Semantic similarity of code changes.
Model
Current
Historical
Setting
GPT-5.4 Claude-Sonnet-4-6 Claude-Opus-4-6
7.56 7.33 7.40
6.63 -12.3% 6.47 -11.7% 7.10 -4.1%
Random Real vs. Real Evol vs. Real
BGE-M3 [46]
CodeT5 [47]
StarCoder2 [48]
0.42 0.71 0.63
0.32 0.46 0.52
0.30 0.45 0.53
Case Study
Figure 7 present a case study on workflow degradation from server evolution. The original workflow (green) achieved a perfect score by directly calling get-tickets for all train tickets data between cities. Evolved parameter description in tool get-tickets (Table 2) indicate first to retrieve station codes by calling get-station-code-of-citys. This perturbation resulted in the evolved workflow (red) only querying tickets for two stations, VNP and SHH, causing the score to degrade to 5. 9
7
Conclusion
In this paper, we introduced MCPEvol-Bench, a novel benchmark for evaluating the task-solving capabilities of LLM agents under dynamic toolset evolution. This benchmark bridges the gap in existing static evaluations that fail to capture agent adaptability in evolving tool environments. Guided by our empirical study on MCP server evolution, we developed 11 mutation operators enabling LLMs to automatically and iteratively modify tools within 123 real-world MCP servers, thereby simulating realistic server evolution. Experiments on 12 state-of-the-art LLMs reveal that even frontier models, such as GPT-5.4 and Claude-Sonnet-4-6 suffer significant performance degradation, highlighting the critical need for LLM agents capable of constructing stable workflows in dynamic environments.
10
References [1] Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345, 2024. [2] Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-rong Wen. Tool learning with large language models: a survey. Frontiers of Computer Science, 19(8), 2025. [3] Aske Plaat, Max van Duijn, Niki Van Stein, Mike Preuss, Peter van der Putten, and Kees Joost Batenburg. Agentic large language models, a survey. Journal of Artificial Intelligence Research, 84, 2025. [4] Huacan Wang, Ziyi Ni, Shuo Zhang, Shuo Lu, Sen Hu, Ziyang He, Chen Hu, Jiaye Lin, Yifu Guo, Yuntao Du, et al. Repomaster: Autonomous exploration and understanding of github repositories for complex task solving. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. [5] Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, and Yu Qiao. OS-ATLAS: foundation action model for generalist GUI agents. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. [6] Xiangru Tang, Tianyu Hu, Muyang Ye, Daniel Shao, Xunjian Yin, Siru Ouyang, Wangchunshu Zhou, Pan Lu, Zhuosheng Zhang, Yilun Zhao, Arman Cohan, and Mark Gerstein. Chemagent: Self-updating memories in large language models improves chemical reasoning. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu, editors, International Conference on Learning Representations, volume 2025, pages 100922–100958, 2025. [7] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to master 16000+ real-world apis. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. [8] Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, and Le Sun. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases. CoRR, abs/2306.05301, 2023. [9] SHEN Haiyang, Yue Li, Desong Meng, Dongqi Cai, Sheng Qi, Li Zhang, Mengwei Xu, and Yun Ma. Shortcutsbench: A large-scale real-world benchmark for api-based agents. In The Thirteenth International Conference on Learning Representations, 2025. [10] Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. Stabletoolbench: Towards stable large-scale benchmarking on tool learning of large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 11143–11156, 2024. [11] Anthropic. Model context protocol (mcp). specification, 2024. Accessed: modelcontextprotocol/specification.
https://github.com/modelcontextprotocol/ 2026-04-23. URL: https://github.com/
[12] Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Shirley Kokane, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, Rithesh Murthy, Liangwei Yang, Silvio Savarese, Juan Carlos Niebles, Huan Wang, Shelby Heinecke, and Caiming Xiong. Apigen: Automated pipeline for generating verifiable and diverse function-calling datasets. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 54463–54482. Curran Associates, Inc., 2024. [13] Wenhao Wang, Peizhi Niu, Zhao Xu, Zhaoyu Chen, Jian Du, Yaxin Du, Xianghe Pang, Keduan Huang, Yanfeng Wang, Qiang Yan, and Siheng Chen. Mcp-flow: Facilitating llm agents to master real-world, diverse and scaling mcp tools, 2025. [14] Zhiwei Liu, Jielin Qiu, Shiyu Wang, Jianguo Zhang, Zuxin Liu, Roshan Ram, Haolin Chen, Weiran Yao, Shelby Heinecke, Silvio Savarese, and others. Mcpeval: Automatic mcp-based deep evaluation for ai agent models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 373–402. [15] Zhenting Wang, Qi Chang, Hemani Patel, Shashank Biju, Cheng-En Wu, Quan Liu, Aolin Ding, Alireza Rezazadeh, Ankit Shah, Yujia Bao, and others. Mcp-bench: Benchmarking tool-using llm agents with complex real-world tasks via mcp servers. In International Conference on Learning Representations (ICLR), 2025.
11
[16] Weiwen Liu, Xu Huang, Xingshan Zeng, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Zhengying Liu, Yuanqing Yu, Zezhong WANG, et al. Toolace: Winning the points of llm function calling. In The Thirteenth International Conference on Learning Representations, 2025. [17] Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E Gonzalez. The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models. In International Conference on Machine Learning, pages 48371–48392. PMLR, 2025. [18] Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. [19] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. [20] Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems, 37:52040–52094, 2024. [21] Paul Pajo. Smithery. ai: A model context protocol for enhanced large language model integration and cross-industry applications. 2025. URL: https://smithery.ai. [22] GitHub, Inc. Github, 2026. Accessed: 2026-04-19. URL: https://github.com/. [23] Alibaba Group. Modelscope platform, 2026. Accessed: 2026-04-19. URL: https://www.modelscope. cn/. [24] npm, Inc. npm registry, 2026. Accessed: 2026-04-19. URL: https://www.npmjs.com/. [25] Guozhao Mo, Wenliang Zhong, Jiawei Chen, Xuanang Chen, Yaojie Lu, Hongyu Lin, Ben He, Xianpei Han, and Le Sun. Livemcpbench: Can agents navigate an ocean of mcp tools? [26] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023. [27] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and others. Deepseek-v3 technical report. [28] OpenAI. Introducing gpt-5.4, 2026. Accessed: 2026-04-19. URL: https://openai.com/index/ introducing-gpt-5-4/. [29] OpenAI. Introducing gpt-5, 2026. Accessed: 2026-04-19. URL: https://openai.com/gpt-5/. [30] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. [31] Anthropic. Introducing claude opus 4.6, 2026. Accessed: 2026-04-19. URL: https://www.anthropic. com/news/claude-opus-4-6. [32] Anthropic. Introducing claude sonnet 4.6, 2026. Accessed: 2026-04-19. URL: https://www.anthropic. com/news/claude-sonnet-4-6. [33] Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025. [34] OpenAI. Introducing openai o3 and o4-mini, 2026. Accessed: 2026-04-19. URL: https://openai.com/ index/introducing-o3-and-o4-mini/. [35] Anthropic. Introducing claude sonnet 4.5, 2026. Accessed: 2026-04-19. URL: https://www.anthropic. com/news/claude-sonnet-4-5. [36] AI@Meta. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
12
[37] Google DeepMind. Gemma 4, 2026. Accessed: 2026-04-19. URL: https://deepmind.google/ models/gemma/gemma-4/. [38] Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. Accessed: 2026-04-19. URL: https://qwen.ai/blog?id=qwen3.5. [39] Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y Tang, Alejandro Cuadron, Chenguang Wang, Raluca Ada Popa, and Ion Stoica. Judgebench: A benchmark for evaluating llm-based judges. In 13th International Conference on Learning Representations, ICLR 2025, pages 37400–37426. International Conference on Learning Representations, ICLR, 2025. [40] Zhenting Wang, Shuming Hu, Shiyu Zhao, Xiaowen Lin, Felix Juefei-Xu, Zhuowei Li, Ligong Han, Harihar Subramanyam, Li Chen, Jianfa Chen, et al. Mllm-as-a-judge for image safety without human labeling. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 14657–14666, 2025. [41] Shraddha Barke, Arnav Goyal, Alind Khare, Avaljot Singh, Suman Nath, and Chetan Bansal. Agentrx: Diagnosing ai agent failures from execution trajectories. arXiv preprint arXiv:2602.02475, 2026. [42] Yibing Liu, Chong Zhang, Zhongyi Han, Hansong Liu, Yong Wang, Yang Yu, Xiaoyan Wang, and Yilong Yin. Trajad: Trajectory anomaly detection for trustworthy llm agents. arXiv preprint arXiv:2602.06443, 2026. [43] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in neural information processing systems, 36:8634–8652, 2023. [44] Lutfi Eren Erdogan, Nicholas Lee, Sehoon Kim, Suhong Moon, Hiroki Furuta, Gopala Anumanchipalli, Kurt Keutzer, and Amir Gholami. Plan-and-act: Improving planning of agents for long-horizon tasks. In International Conference on Machine Learning, pages 15419–15462. PMLR, 2025. [45] Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. Agent workflow memory. In International Conference on Machine Learning, pages 63897–63911. PMLR, 2025. [46] Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation, 2024. [47] Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 conference on empirical methods in natural language processing, pages 8696–8708, 2021. [48] Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173, 2024. [49] Hervé Abdi. Coefficient of variation. Encyclopedia of research design, 1(5):169–171, 2010. [50] Philip Sedgwick. Spearman’s rank correlation coefficient. Bmj, 349, 2014.
13
Appendix A
Impact Statement
MCPEvol-Bench highlights the vulnerability of LLM agents to evolving toolsets. By exposing how performance drops when MCP servers change (e.g., updated parameters or descriptions), our work drives the development of more adaptive and robust agents. This paves the way for more stable automation workflows, reducing errors and boosting productivity in software development and daily tasks. Furthermore, by focusing on the open-source Model Context Protocol (MCP), we support a more open and interoperable AI ecosystem. However, as agents integrate more seamlessly with external tools, vulnerabilities or misleading inputs could lead to the unintended leakage of sensitive data (such as financial records or codebases). Additionally, frequent interface changes may create security blind spots that bypass traditional static checks, requiring greater attention to data privacy and security in dynamic environments.
B
Safeguarding Statement
MCPEvol-Bench instructs LLM agents to operate within isolated directories, though potential risks of harmful operations on the host system remain. We mitigate this by enforcing strict file permissions to prevent unauthorized modification or deletion of critical files during evaluation. None of the 123 MCP servers included rely on proprietary API keys, ensuring no personal information is exposed. Furthermore, evolution is simulated via LLM-driven code mutations on locally stored NPM packages within a closed, sandboxed environment, rather than interacting with live production services. This eliminates the risk of disrupting real-world software services. These measures ensure the benchmark can be safely deployed for research without posing security or privacy threats.
C
Limitation and Future Work
The MCP servers included in our benchmark are constrained. To ensure plug-and-play deployment, we excluded instances requiring external API key authorization. These servers usually have more frequent updates due to maintenance by dedicated companies or institutions. However, this exclusion does not compromise the validity of our benchmark or the generalizability of our conclusions. Specifically, the observed inadequacy of current models in adapting to dynamically evolving toolusing scenarios. In future work, we plan to incorporate additional officially maintained MCP servers with greater practical utility and functionality, such as those for Hugging Face and Google Maps.
D
Details of Experiments
D.1
Effectiveness of Evaluation Metrics
In our experiments, we adopted the evaluation metrics Task Fulfillment and Planning Effectiveness from MCP-Bench [15]. Both metrics are evaluated using rubric-based LLM judges, with prompt shuffling employed to ensure the robustness of assessments (prompts detailed in H.3). The main experimental results in Table 3 are averaged over five independent evaluations. The corresponding variances are reported in Table 7. The observed low variance (all < 0.2) shows the stability and effectiveness of these two metrics. The proposed Evolutionary Competency Score (ECS) effectively evaluates agent adaptability during the evolution of MCP Server tools. Let U denote the set of evaluation tasks. For each task TF TF u ∈ U, let Su = {Su,1 , . . . , Su,N } be the sequence of Task Fulfillment scores across N MCP server versions. We define the mean performance µu and standard deviation σu as: v u N N u 1 X X 1 TF t TF − µ 2 . µu = Su,i , σu = Su,i (2) u N i=1 N − 1 i=1 Although the Coefficient of Variation [49] (CVu = σu /µu ) is frequently used to measure stability, it is insufficient for assessing adaptability as it decouples stability from absolute performance magnitude 14
Table 7: Variance (σ 2 ) of performance metrics across three evolution stages. Early Stage
Model Qwen3.5-9B Llama-3.3-70B Qwen3.5-27B GPT-4o Gemini-2.5-pro o3 GPT-5.1 Gemma-4-31B-it Claude-Sonnet-4-5-thinking GPT-5.4 Claude-Sonnet-4-6 Claude-Opus-4-6
Middle Stage
Late Stage
Task σ2 Fulfillment
Planning σ2 Effectiveness
Task σ2 Fulfillment
Planning σ2 Effectiveness
Task σ2 Fulfillment
Planning σ2 Effectiveness
0.04 0.15 0.07 0.03 0.12 0.09 0.18 0.06 0.11 0.05 0.13 0.10
0.12 0.02 0.16 0.11 0.05 0.14 0.08 0.13 0.04 0.10 0.07 0.15
0.08 0.09 0.11 0.17 0.06 0.03 0.12 0.15 0.10 0.14 0.04 0.08
0.03 0.14 0.06 0.04 0.13 0.10 0.05 0.09 0.17 0.08 0.11 0.06
0.11 0.05 0.13 0.08 0.10 0.16 0.04 0.07 0.06 0.13 0.15 0.12
0.07 0.18 0.09 0.15 0.02 0.07 0.11 0.14 0.12 0.03 0.09 0.16
(e.g., a low-performing stable model may have the same CVu as a high-performing one). To construct a comprehensive adaptability metric, we formulate a utility function Ju that maximizes mean performance µu while penalizing relative instability. By applying a linear penalty factor k to CVu , we obtain Ju = µu (1 − k · CVu ). Substituting the definition of CVu yields Ju = µu − kσu . This derivation establishes that penalizing relative volatility is mathematically equivalent to subtracting a scaled absolute standard deviation from the mean. The ECS simplifies this relationship by setting k = 1: 1 X ECS = (µu − σu ) (3) |U| u∈U
The efficacy of ECS in assessing adaptability stems from its sensitivity to performance regression. In our benchmark (N = 3), consider two models on task u: • Model A (Low adaptability): Scores {0.9, 0.5, 0.9} yield µA ≈ 0.77 and σA ≈ 0.23, resulting in ECSA ≈ 0.54. • Model B (High adaptability): Scores {0.8, 0.8, 0.8} yield µB = 0.80 and σB = 0.00, resulting in ECSB = 0.80. Despite similar mean performance, the single performance drop in Model A significantly penalizes its ECS, resulting in a score substantially √ lower than that of Model B. For N = 3 with integer scores in [1, 10], the ECS ranges from 4 − 27 ≈ −1.20 to 10. D.2
Reliability Analysis of Benchmark
In Section 6.3, we compare the semantic similarity between code modifications generated by simulated evolution and those from real-world MCP server version updates. Results presented in Table 6 demonstrate the effectiveness of our evolution. Specifically, for real-world version updates (Real vs. Real), we analyze the code differences between consecutive versions (e.g., between v0.3.2 and v0.3.1, or v0.3.1 and v0.3.0). For encoder model BGE-M3 [46], we directly compute the cosine similarity of word embedding vectors. For the other two decoder-only models, we extract the feature vectors from the final layer for cosine similarity calculation. While human experts often struggle with absolute scoring, they excel at relative ranking. Leveraging this insight, we evaluated the alignment between LLM-based evaluations and human expert judgments. We analyzed task trajectories generated by three advanced models: Claude-Sonnet-4-6, Claude-Opus4-6, and GPT-5.4. Let U denote the set of all task instances. For each task u ∈ U, we established a H H H ground truth ranking vector RH u = [ru,early , ru,middle , ru,late ] by averaging the ranks assigned by three human experts, where r ∈ [1, 2, 3]. For instance, if RH u = [1, 3, 2], it implies that the trajectory in early stage is ranked best, and that in late stage is second. Simultaneously, we employed three LLM evaluators (DeepSeek-Chat, GPT-4o-mini, and Gemini-2.5-pro) to compute Task Fulfillment scores. These scores were converted into rank vectors RE u . The consistency between each LLM evaluator and the human ground truth was quantified using the average of spearman’s rank correlation coefficient ρ̄ [50]: X X 6d⊤ u du ρ̄ = ρu /|U| = (1 − )/|U| (4) N (N 2 − 1) u∈U u∈U H E where du = Ru − Ru is the difference in ranks in task u, and N = 3.
15
Table 8 presents the correlation results. The high correlation scores across all evaluators demonstrate that our evaluations closely align with human perception of trajectory quality. Table 8: ρ̄ between LLM evaluators and human expert rankings. The ranked items are task trajectories generated by Claude-Sonnet-4-6, Claude-Opus-4-6, and GPT-5.4. Trajectory
LLM Evaluator DeepSeek-Chat Gemini-2.5-pro GPT-4o-mini
D.3
Claude-Sonnet-4-6
Claude-Opus-4-6
GPT-5.4
0.82 0.80 0.79
0.73 0.75 0.75
0.85 0.85 0.84
Implementation Details
During the benchmark construction phase, we primarily employed the DeepSeek-Chat model for task synthesis, executing these tasks via the corresponding MCP Server to generate multi-tool invocation trajectories. For the evolution mutation of the MCP Server, we utilized the Claude-Opus-4-5 model, while test case generation continued to rely on DeepSeek-Chat. To ensure workflow stability and strict adherence to tool invocation formatting standards, the temperature parameter for all model evaluations was set to 0.2. The file cleanup script will automatically run after each evaluation to ensure that previous runs do not affect subsequent results. All experiments were conducted on a server equipped with dual Intel(R) Xeon(R) Platinum 8358P CPUs @ 2.60GHz (totaling 64 logical cores), running CentOS 7.9. The software environment was built upon Python 3.12.12, Node.js 24.13.0, and npm 11.6.2. The cost for a single evaluation is approximately 0.26$, calculated using DeepSeek-V3.2 pricing rates (0.14$ per million input tokens and 0.28$ per million output tokens). For the experiments involving Figure 6(b), an MCP server may be subject to multiple operators simultaneously. We evenly distribute the resulting impact on the score among each operator to calculate their contribution scores.
E
Details of the Empirical Study
E.1
Details of Empirical Data Collection
The empirical analysis draws on two primary sources: 1) deployment metadata of MCP servers periodically crawled from Smithery platform, and 2) versioned code repositories published on NPM. We crawled remotely deployed MCP servers from Smithery platform by combining the keywords listed in Table 9 with the filter is:remote. After deduplication, we obtained 1,869 unique endpoints of MCP servers. To complement the Smithery dataset, we also aggregated MCP servers from three additional sources. • ModelScope (12 keywords): browser-automation, search, communication, developer -tools, entertainment-and-media, file-systems, finance, knowledge-and-memory, location-services, art-and-culture, research-and-data, calendar-management. Table 9: Functional keywords of MCP servers crawled from Smithery. Keywords Aggregators Bio Command Line Data Platforms Embedded System Knowledge & Memory Monitoring Security Translation Services Workplace & Productivity
Art & Culture Cloud Platforms Communication Delivery File Systems Legal Multimedia Process Social Media Text-to-Speech Other Tools & Integrations
16
Architecture & Design Code Execution Customer Data Platforms Developer Tools Finance & FinTech Location Services Research Sports Travel & Transportation
Browser Automation Coding Agents Databases Data Science Tools Gaming Marketing Search Support & Service Mgmt. Version Control
• GitHub (2 repositories): https://github.com/punkpeye/awesome-mcp-servers, https://github.com/wong2/awesome-mcp-servers. • NPM (14 keywords): mcp server, model context protocol, modelcontext -protocol, mcp-server, mcp_server, create mcp server, mcp, mcp ai, mcp agent, @modelcontextprotocol, @modelcontextprotocol/server, mcp tools, mcp ai agent, mcp protocol implementation. We initially crawled 906, 1,040, and 1,912 servers from GitHub, Smithery, and ModelScope, respectively. By matching these names against the NPM registry, we download 1,461 repositories with their historical versions. After the removal of undeployable instances, 515 servers with 9273 historical versions remained for analysis. E.2
Analysis of Tool Evolution Stages
To analyze tool evolution trends, we sorted the versions of the 515 collected MCP servers by release date and calculated their current evolution stage via normalization (e.g., version 8 out of 10 corresponds to 80%). As shown in Figure 8, all three metrics exhibit significant growth.
Figure 9: Function classification of MCP servers.
Figure 8: Growth of toolset complexity.
E.3
Analysis of Evolution Patterns
The identified patterns capture the diverse ways in which MCP servers evolve over time, as shown in Table 10. These changes range from simple semantic updates, such as modifying tool or parameter descriptions (P1, P3), to structural modifications like adding, removing, or replacing tools and parameters (P2, P4, P5, P10–P11, P14, P17). Additionally, servers often undergo complex composite changes (P6–P9, P12, P15–P16, P18) that simultaneously alter interface structures and semantic constraints. Understanding these patterns is crucial for developing agents that can robustly adapt to the dynamic nature of real-world tool environments. Table 10: Taxonomy of MCP Server Evolution Patterns ID
Pattern Name
Description
P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P14 P15 P16 P17 P18
TOOL-DESC_CHANGE TOOL-ADD PARAM-DESC_CHANGE PARAM-ADD(OPTIONAL) TOOL-REPLACE TOOL-DESC_CHANGE + PARAM-ADD(OPTIONAL) TOOL-DESC_CHANGE + PARAM-DESC_CHANGE PARAM-STRICT/RELEX PARAM-STRICT/RELEX + PARAM-DESC_CHANGED PARAM-ADD(REQUIRED) PARAM-REMOVE(OPTIONAL) PARAM-ADD(OPTIONAL) + PARAM-DESC_CHANGE PARAM-PROMOTE/DEMOTE PARAM-REMOVE(REQUIRED) PARAM-REMOVE(OPTIONAL) + PARAM-DESC_CHANGE TOOL-DESC_CHANGE + PARAM-ADD(OPTIONAL) TOOL-REMOVE TOOL-ADD + TOOL-DESC_CHANGE
Update to the tool’s natural language description. Introduction of a new tool. Update to a specific parameter’s description. Addition of a new optional parameter. Replacement of an existing tool with a new implementation. Tool description update combined with optional parameter addition. Concurrent updates to both tool and parameter descriptions. Modification of parameter validation constraints (strictness or relaxation). Constraint modification accompanied by description update. Addition of a new mandatory parameter. Removal of an existing optional parameter. Optional parameter addition with other parameters description update. Change in parameter status between optional and required. Removal of a previously mandatory parameter. Optional parameter removal with remaining parameters description updates. Tool description update combined with optional parameter addition. Deprecation and removal of an existing tool. New tool addition accompanied by other tool’s description updates.
17
F
Details of Benchmark
F.1
Statistics of MCPEvolBench
Figures 9, 10 and 11 present the functional classification of MCP servers, the number of tool calls required to complete each task, and the number of tools included in the context of each task, respectively.
Figure 10: Distribution of tool number for task completion. F.2
Figure 11: Distribution of involved tool number.
AST-based Code Anchoring
It is impractical to directly inject the entire codebase of an MCP server into the model’s context window, as it introduces excessive redundant information. Therefore, we localize the registration and implementation snippets corresponding to each tool. In practice, we observe that the collected tool implementations in MCP Servers predominantly fall into four categories: Register Handler (Listing 1, 29.77%), Switch Handler (Listing 2, 26.97%), Cross-File Handler (Listing 3, 16.03%), and If-Block Handler (Listing 4, 3.56%). The percentages indicate the proportion of each category. The remaining cases are identified through human annotation. server . tool ( ’get - station - code - of - citys ’ , ’ Retrieve the station_code for a city in sea - rail intermodal transport using its Chinese name ’, { citys : z . string () . describe ( ’ The city to query , e . g . , " Beijing ". For multiple cities , separate with "|" , e . g . , " Beijing | Shanghai ". ’) , } , async ({ citys }) = > { let result = {}; for ( const city of citys . split ( ’| ’) ) { if (!( city in CITY_CODES ) ) { result [ city ] = { error : ’ City not found . ’ }; } else { result [ city ] = CITY_CODES [ city ]; } } return { content : [{ type : ’ text ’ , text : JSON . stringify ( result ) }] , }; }) ;
Listing 1: Code Example of Register Handler export function handleToolCall ( request ) { const { name , arguments : args = {} } = request . params ; switch ( name ) { case ’ find_component ’: return handleFindComponent ( args ) ; case ’ get_component_props ’: return h an dl e Ge tC o mp on en t Pr op s ( args ) ; case ’ get_usage_examples ’: return ha nd leG etU sa geE xa mpl es ( args ) ;
18
case ’ search_docs ’: return handleSearchDocs ( args ) ; case ’ list_components ’: return handleListComponents ( args ) ; case ’ get_ires_tokens ’: return handleGetIresTokens ( args ) ; case ’ g et _ ir es _c o mp on en t _i nf o ’: return h a n d l e G e t I r e s C o m p o n e n t I n f o ( args ) ; case ’ analyze_component ’: return h a n d l e A n a l y z e C o m p o n e n t M e n t i o n s ( args ) ; case ’ get_ design _guid elines ’: return h a n d l e G e t D e si g n G u i d e l i n es ( args ) ; default : throw new Error ( ‘ Unknown tool : $ { name } ‘) ; } }
Listing 2: Code Example of Switch Handler // cross two files : A . ts and B . ts // A . ts : const memoryBankTools = { list_projects : { name : " list_projects " , description : " List all projects in the memory bank " , inputSchema : { type : " object " , properties : {} , required : [] } } }; // B . ts : async handleCommand ( command ) { try { if ( command . operation === " list_projects " ) { const contents = await this . lis tDirec toryC ontent s ( this . rootPath ) ; return { success : true , content : JSON . stringify ( contents ) , }; } } catch ( error ) { // Optional : Add error handling here console . error ( " Error in handleCommand : " , error ) ; return { success : false , content : ‘ Failed to list projects : $ { error . message } ‘ , }; } }
Listing 3: Code Example of Cross-File Handler server . setRequestHandler ( CallToolRequestSchema , async ( request ) = > { if ( request . params . name === " analyze_file " ) { const rawPath = String ( request . params . arguments ?. path ) ; const targetPath = path . resolve ( rawPath ) ; try { await fs . access ( targetPath ) ; } catch ( _e ) { return { content : [
19
{ type : ’ text ’ , text : ‘ File not found at path : $ { targetPath }. Please provide a valid absolute or relative path from the project root . ‘ } ] }; } return { content : [ { type : ’ text ’ , text : ‘ Successfully accessed file at : $ { targetPath } ‘ } ] }; } throw new Error ( ‘ Unknown tool : $ { request . params . name } ‘) ; }) ;
Listing 4: Code Example of If-Block Handler Table 11: Overview of 11 mutation operators for MCP server evolution Level
Name
Description
TOOL
Operator 1: Tool Addition
TOOL
Operator 2: Tool Replacement
TOOL
Operator 3: Tool Removal
TOOL
Operator 4: Tool Integration
Adds a new tool to extend the server’s functionality. Existing tools are kept unchanged to reduce regression risk and preserve backward compatibility. Replaces an existing tool with an updated version to improve capability or design. The new tool should cover the original tool’s main use cases, while other tools remain unaffected. Removes an obsolete tool and transfers its essential behavior into another existing tool. The aim is to reduce the number of tools while retaining key functionality. Adds a new tool and refines related tool descriptions to improve overall consistency. The goal is clearer tool roles, less overlap, and easier discovery for users.
PARAM
Operator 5: Flexible Expansion
PARAM
Operator 6: Constraint Mutation
PARAM
Operator 7: Parameter Pruning
PARAM
Operator 8: Interface Refactoring
DESC
Operator 9: Tool Description Update
DESC
Operator 10 Parameter Description Update
DESC
Operator 11: Joint Description Update
F.3
Extends a tool interface by adding a small number of optional parameters. These parameters increase flexibility while keeping existing calls working as before. Changes parameter constraints, required/optional status, or data types to make the interface contract more accurate. The aim is better alignment between the specification and actual usage. Simplifies the interface by removing redundant or low-value parameters. It reduces maintenance cost and should acknowledge compatibility risks when required inputs are removed. Updates the tool description together with parameter additions/removals. Its purpose is to keep documentation consistent with the current interface. Edits the tool description to better reflect what the tool does. It improves clarity and accuracy without changing behavior. Edits parameter descriptions to make their meaning and intended usage clearer. It improves documentation while keeping the parameter set unchanged. Improves both tool and parameter descriptions to present a consistent and accurate specification. It reduces ambiguity without changing behavior.
Definition of Mutation Operators
The definitions of the mutation operators are presented in Table 11, and their corresponding prompts are detailed in Appendix H.1.
20
G
Supplementary Case Study
In this section, we first present case study illustrating three mutation operators across distinct levels used for MCP server evolution. For brevity, we omit parameter types and their constraints. Subsequently, we provide two examples where original workflows fail due to tool evolution. G.1
Case Study of Mutation Operators
Original MCP Server
Evolved MCP Server
Description: Provides time retrieval and timezone conversion capabilities using IANA timezone names. Tools: • get_current_time: Retrieves the current date and time. - timezone: Timezone (e.g., Asia/Shanghai, UTC). Defaults to system timezone. - format: Output format. Options: ’iso’ (default), ’locale’, ’timestamp’. • get_time_info: Retrieves detailed time components (year, month, day, hour, minute, second, etc.). - timezone: Timezone. Defaults to system timezone.
Description: Provides time retrieval and timezone conversion capabilities using IANA timezone names. Tools: • get_current_time: ... (unchanged) • get_time_info: ... (unchanged) • convert_timezone (new): Converts a given time from one timezone to another. - time: Input time (ISO 8601, timestamp, or locale string). Defaults to current time. - from_timezone: Source timezone. Defaults to system timezone. - to_timezone: Target timezone (e.g., America/New_York). - format: Output format. Options: ’iso’ (default), ’locale’, ’timestamp’.
Figure 12: Case Study: Application of the Tool Addition operator to the MCP server @guanxiong/mcp-server-time, introducing the convert_timezone tool. Original MCP Server
Evolved MCP Server
Description: A Model Context Protocol (MCP) server providing weather data capabilities. Tools: • get-alerts: Retrieves weather alerts for a specific US state. - state: Two-letter state code (e.g., CA, NY). Length must be 2. • get-forecast: Gets the weather forecast for a specific location. - latitude: Latitude of the location (−90 to 90). - longitude: Longitude of the location (−180 to 180). • get-current-conditions: Fetches current weather conditions from the nearest observation station. - latitude: Latitude of the location (−90 to 90). - longitude: Longitude of the location (−180 to 180).
Description: A Model Context Protocol (MCP) server providing weather data capabilities. Tools: • get-alerts (unchanged): ... • get-forecast (new): Gets the weather forecast for a specific location. - latitude: Latitude of the location (−90 to 90). - longitude: Longitude of the location (−180 to 180). - unit (new): Optional temperature unit for forecast output. Supports ‘C‘ or ‘F‘, defaults to ‘C‘. - days (new): Optional number of forecast days to include in the response summary, defaults to 1. • get-current-conditions (unchanged): ...
Figure 13: Case Study: Application of the Flexible Expansion operator to the MCP server mcp-weather-demo, introducing the unit and days parameters in existing get-forecast tool.
21
Original MCP Server
Evolved MCP Server
Description: A TypeScript-based MCP server for processing and manipulating Microsoft Word documents (.docx). Tools: • create_document: null - filePath: null - title: null - author: null • open_document: null - filePath: null • add_table: Inserts a table into a Word document at the specified file path with the given dimensions, optional headers, and optional cell data. - filePath: The path to the Word document where the table will be added. - rows: The number of rows for the table. - cols: The number of columns for the table. - headers: An optional array of strings to use as column headers for the table. - data: An optional two-dimensional array of strings representing the cell data for each row in the table.. ......
Description: A TypeScript-based MCP server for processing and manipulating Microsoft Word documents (.docx). Tools: • create_document (unchanged): ... • open_document (changed): Opens an existing document for editing. - filePath: Path to the .docx file. • add_table (unchanged): ... ......
Figure 14: Case Study: Application of the Joint Description Update operator to the MCP server @puchunjie/doc-tools-mcp, introducing the new description for open_document tool. G.2
Case Study of Workflow Failure
Task: I’m planning a business trip from Hangzhou to Beijing next Friday and need to make sure there are high-speed train tickets available. Additionally, help me set up a task file in the markdown directory to track my trip itinerary and to-do list related to this business trip. Original Tool: get-tickets: {“fromStation”: the name or the station_code, ... };. (old parameters) Evolved Tool: get-tickets: {“fromStation”: the name or the station_code, ... “minPrice”: ..., “maxPrice”: ...}. (new parameters) Success with Original MCP Servers Action 1: get-current-date(Friday) Observation 1: Time: 2026-04-17 Action 2: get-tickets(fromStation=Hangzhou, toStation= Beijing, date=2026-04-17, ...) Observation 2: Available Trains: • G814 (06:50–13:07): HGH → VNP ... • G38 (07:47–12:22): HGH → VNP ... • G816 (08:49–14:46): HGH → VNP ... • ...... Action 3: ...... (Task Fulfillment: 10)
Fail with Evolved MCP Servers Action 1: get-current-date(Friday) Observation 1: Time: 2026-04-17 Action 2: get-tickets(fromStation=Hangzhou, toStation= Beijing, date=2026-04-17, minPrice=0 , maxPrice=0 ...) Observation 2: No Available Trains Find. ... (Repeated erroneous tool calls) Action 3: ...... (Task Fulfillment: 4)
Figure 15: Case Study from GPT-5.4’s Trajectory: Reasoning error from parameter evolution. Two new parameters minPrice and maxPrice filter the tickets price in the get-tickets tool. The agent failed to utilize the new parameters, setting them to zero and resulting in no available tickets. Areas highlighted in yellow show changes compared to the original workflow.
22
Task: I want to create a simple public webpage that shows the latest 5 papers from the "Embodied AI" research area. Can you fetch the most recent papers list from a relevant research website and then publish that list as a clean, readable HTML page that anyone can access via a URL?. Original Tool: fetch_url: Fetch a URL, support HTML, text, and image. (old description) Evolved Tool: fetch_url: Fetch a URL with configurable request body support for HTML, text, and image responses. (new description) Success with Original MCP Servers
Fail with Evolved MCP Servers
Action 1: fetch_url({"url":"https://arxiv.org/search/? query=embodied+ai&searchtype=all &source=header&order=-announced_date_first"}) Observation 1: Search results: • Title: Infrastructure First: Enabling Embodied AI for Science in the Global South, Authors: [Shaoshan Liu, ...], Year: 2026 • Title: Multimodal Foundation Models for Embodied Reasoning in Physical Environments, Authors: [Chen Wang, ...], Year: 2026 • Title: Learning Dexterous Manipulation Through Embodied Simulation and Transfer, Authors: [Michael Zhang, ...], Year: 2026 • Title: Embodied Navigation with Semantic Memory and Hierarchical Planning, Authors: [Priya Sharma, ...], Year: 2026 • Title: Social Embodied AI: Understanding Human-Robot Interaction in Collaborative Tasks, Authors: [David Kim, ...], Year: 2026 • ... ... Action 2: deploy_html(value=<html text>..., ...) Observation 2: Web available in https://mcp. edgeone.site/share/skmbynWu586JpO8qi_P5l Action 4: ... (Task Fulfillment: 10)
Action 1: fetch_url({"url":"https://arxiv.org/search/? query=embodied+ai &searchtype=all &abstracts=show &order=-announced_date_first &size=5"} ) Observation 1: Failed to retrieve web page content. ... (Repeated erroneous tool calls) Action 2: I am unable to obtain the latest papers, so I can only provide some previous papers. Paper results: • Title: Voyager: An Open-Ended Embodied Agent with Large Language Models, Authors: [Guanzhi Wang, ...], Year: 2023 • Title: Multimodal Foundation Models for Embodied Reasoning in Physical Environments, Authors: [Chen Wang, ...], Year: 2023 • Title: RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control, Authors: [Anthony Brohan, ...], Year: 2023 • Title: PaLM-E: An Embodied Multimodal Language Model, Authors: [Danny Driess, ...], Year: 2023 • ... Action 3: deploy_html(value=<html text>..., ...) Observation 3: Web available in https://mcp. edgeone.site/share/fwR7eoSsBO5vE2veGFK-I Action 4:... (Task Fulfillment: 3)
Figure 16: Case Study from Claude-Sonnet-4-6’s Trajectory: Planning error from tool description evolution. A change in the fetch_url tool description led the agent to use invalid parameters (abstracts=show and size=5). This caused the retrieval of latest papers to fail, forcing the agent to publish a webpage containing outdated literature from 2023. Areas highlighted in yellow show changes compared to the original workflow.
H
Prompts for MCPEvol-Bench
In this section, we detail all prompts utilized by LLMs in our framework. Placeholders enclosed in curly braces (e.g., {server_name}, {task}) are replaced with actual text at runtime. We first introduce the prompts for operator selection and 11 mutation operators employed in LLM-Driven MCP Server Evolution, which guide the LLM to perform precise code mutations. Subsequently, we present the prompts used for the LLM-assisted process during benchmark construction. Finally, we present the prompts used for agent execution and evaluation. H.1
Prompts for MCP Server Evolution Mutation
The following outlines the prompts for selecting mutation operators, defining mutation operators, and generating test cases. The mutation operator prompts consist of three components: code modification examples, the operator’s function, and file modification instruction. Prompt for Mutation Operator Selection # Server name: {server_name} server_description: {server_description}
23
# Registered tools (schemas summary) {server_tools} # Task You must choose ONE mutation strategy for benchmarking MCP server robustness. ## Categories and operator_id ### TOOL (code change via diff_hunks; Functionality Changes) - "0" Standard_Add: add a new tool; no specific existing tool target. - "1" Standard_Replace: replace one existing tool; needs target_tool_name. - "2" Standard_Optimization: add a new tool and tune related descriptions; no single target tool. - "3" Standard_Delete: remove a tool and migrate behavior; needs target_tool_name. ### PARAM (code change; Capability Refinement) - "0" Param_Extend: add optional parameters. - "1" Param_Constrain: tighten/loosen constraints, required, types. - "2" Param_SYNC: add/remove parameters and sync tool description. - "3" Param_Pruning: remove parameters. ### DESC (documentation only; Semantic Adjustment) - "0" rewrite tool description only. - "1" rewrite parameter descriptions only (needs parameters in schema). - "2" rewrite both tool and parameter descriptions (needs parameters). ## Rules - Pick operator_id that best matches the server’s purpose and the tools’ schemas. - Make maximum functional/contextual changes within the scope of the original MCP server’s functionality. - If necessary, consider Standard_Delete or Param_Pruning to remove some redundant content. - target_tool_name: required for PARAM and DESC (must be one of the listed tools present in this server). - For TOOL with operator_id "1" or "3", set target_tool_name to the tool to replace or delete. - For TOOL "0" or "2", set target_tool_name to null. - If a tool has no input parameters, do NOT choose PARAM operators that require parameter structure beyond "0", or choose DESC "0" only. Output ONLY valid JSON (no markdown fences): {"mutation_category":"TOOL"|"PARAM"|"DESC","operator_id":"0"|"1"|"2"|"3", "target_tool_name":string|null,"selection_rationale":"why this operator fits this server/tools (1-3 sentences)"}
Prompt for Code Modification Examples "diff_hunks": [ { "old_code": "function fetch(url) {\n return axios.get(url);\n}", "new_code": "function fetch(url, timeout = 30) {\n return axios.get(url, {timeout});\n}", "file_name": "[server_name_or_path]/index.js" }, { "old_code": "", "new_code": "function setTimeout(seconds) {\n globalTimeout = seconds;\n}",
24
"anchor_code": "import axios from axios;\n\n// HTTP utilities", "file_name": "[server_name_or_path]/index.js" }, { "old_code": "function setTimeout(seconds) {\n globalTimeout = seconds;\n}", "new_code": "", "file_name": "[server_name_or_path]/server.js" } ] Note: - "anchor_code": Required only when "old_code" is empty. Specify the exact line(s) of code immediately before the insertion point. - "old_code" / "anchor_code": Must **match the original file content exactly** to enable reliable text search and replacement. - "file_name": Refers to the file that changes the code. MUST include the **MCP Server Name** to uniquely identify the target file.
Prompt for Tool Addition Operator ## Objective Add a new tool (self-designed) to an existing MCP Server configuration to enhance functionality without breaking existing features. ## Requirements 1. Preserve Core Functionality: Retain all existing tools defined in the current Server description. Do not remove, rename, or alter the logic/schema of any existing tool. 2. Integration: Append the new tool to the existing tool list. Ensure the new tool name does not conflict with existing names. 3. New Tool Design: Design the new tool’s capabilities to enhance or complement the MCP Server’s functionality and improve usability. ## Output Format { "integration_summary": "Brief overview of the new tool’s purpose and integration strategy", "new_tool_config": { "name": "tool_name", "description": "purpose of the tool", "input_schema": { ...} }, {Code_Modification_Examples} }
Prompt for Tool Replacement Operator ## Objective Replace an existing tool in an MCP Server configuration with a new version (self-designed) while maintaining server integrity. ## Requirements 1. Complete Replacement: Remove the definition of the old tool entirely and insert the new tool in its place. 2. Preserve Other Tools: All tools NOT being replaced must remain exactly unchanged. Their schemas, descriptions, and logic must be preserved 100%. 3. Update References: If the server description contains summaries, examples, or documentation referencing the old tool name, update them to
25
reflect the new tool’s name and capabilities. 4. Functional Equivalence: The new tool must be able to replace the old tool’s functionality either: - Directly: The new tool alone can perform all critical functions of the old tool, OR - In Combination: The new tool, when used with other existing tools, can achieve the same outcomes as the deleted old tool. ## Output Format { "replacement_summary": "brief overview of what changed and why", "old_tool_name": "name of the tool being replaced", "new_tool_config": { "name": "new_tool_name", "description": "purpose of the tool", "input_schema": { ...} }, {Code_Modification_Examples} }
Prompt for Tool Removal Operator ## Objective Remove an obsolete tool from an MCP Server configuration and refactor an existing tool to absorb its functionality. Do NOT add a new tool; instead, extend the capabilities of a surviving tool to handle the deleted tool’s use cases. ## Requirements 1. Complete Deletion: Remove the definition of Target Tool entirely from the tools list. 2. Function Migration: Modify ONE existing tool’s implementation logic to absorb the deleted tool’s core functionality. 3. Update Documentation: Update the modified tool’s description and/or parameter descriptions to clearly indicate it now handles the deleted tool’s use cases. 4. Preserve Other Tools: All other tools (except the one being modified for migration) must remain completely untouched. ## Output Format { "deletion_summary": "brief overview of what was removed and why", "deleted_tool_name": "name of the tool being removed", "migration_target_tool": "name of the existing tool that absorbs the functionality", {Code_Modification_Examples} }
Prompt for Tool Integration Operator ## Objective Add a new tool (self-designed) to an MCP Server configuration and autonomously optimize related existing tool descriptions for better coherence. ## Requirements 1. Add New Tool: full definition.
Successfully add the new tool to the tools list with its
26
2. Autonomous Optimization: Analyze the new tool’s capabilities and proactively update descriptions of related existing tools to improve discoverability, reduce redundancy, or clarify relationships. Only modify description/text fields; DO NOT change name, inputSchema, or outputSchema. 3. Preserve Other Tools: All tools whose descriptions are not updated must remain 100% unchanged. ## Output Format { "addition_summary": "brief overview of the new tool and its purpose", "new_tool_name": "name of the tool being added", "auto_optimizations": [ { "tool_name": "name of optimized tool", "reason": "why this tool’s description was updated", "changes": "summary of description changes" }, ... ], {Code_Modification_Examples} }
Prompt for Flexible Expansion Operator ## Objective Simulate a tool evolution process by modifying the implementation code to introduce new optional parameters. ## Requirements 1. Extend Functionality: Add optional parameters to enhance flexibility. 2. Ensure Compatibility: Existing tool description and its functions remain unchanged (backward compatible) and continue to work. 3. Update Implementation: Modify the tool’s logic to handle the new parameters gracefully (e.g., default values). 4. Maintain Quality: Keep code clean and consistent with existing styles. ## Output Format { "evolution_summary": "Brief overview of the extension, logic updates and code changes", "parameter_additions": [ { "name": "param_name", "type": "string|int|bool|...", "required": false, "description": "purpose of the parameter" }, ... (no more than three) ], {Code_Modification_Examples} }
Prompt for Constraint Mutation Operator ## Objective Simulate a tool evolution process by modifying the implementation code to adjust parameter constraints, required status, or data types.
27
## Requirements 1. Modify Constraints: Tighten or loosen validation rules (e.g., min/max values, regex patterns, length limits). 2. Update Requirements: Change parameter status between optional and required if logically justified. 3. Correct Types: Adjust data types to better match actual usage or improve precision. 4. Enforce Logic: Update implementation to validate and handle the new constraints properly. ## Output Format { "evolution_summary": "Brief overview of the constraint evolution, logic updates and validation changes", "parameter_changes": [ { "name": "param_name", "change_type": "constraint|required|type", "old_value": "original_value_or_type", "new_value": "new_value_or_type" }, ...(no more than three) ], {Code_Modification_Examples} }
Prompt for Parameter Pruning Operator ## Objective Simulate a tool evolution process by modifying the implementation code to remove redundant or low-value parameters. ## Requirements 1. Identify Candidates: Select parameters for removal based on redundancy, or simplification goals. 2. Remove Parameters: Update the schema to remove optional or required parameters. 3. Clean Logic: Remove all code logic related to the deleted parameters (no dead code). 4. Assess Breaking Changes: Explicitly flag if removing a required parameter breaks existing clients. ## Output Format { "evolution_summary": "Brief overview of the pruning action and logic updates", "parameter_removals": [ { "name": "param_name", "type": "string|int|bool|...", "required": true|false }, ...(no more than three) ], {Code_Modification_Examples} }
28
Prompt for Interface Refactoring Operator ## Objective Simulate a tool evolution process where the tool description and parameter structure (add/remove) change simultaneously, ensuring documentation reflects the new interface shape. ## Requirements 1. Mandatory Tool Description Update: The tool-level description must change to reflect the new capability or reduced scope. 2. Parameter Structural Changes: Include only additions or removals of parameters (no constraint/type modifications). 3. Explicit Required Status: Clearly indicate whether each changed parameter is required or optional. 4. Description Sync: Provide the description for added parameters (new) or removed parameters (old). ## Output Format { "evolution_summary": "Brief overview of the co-evolution", "tool_description_change": { "old_description": "original tool description", "new_description": "updated tool description", "reason": "why the tool description changed" }, "parameter_changes": [ { "name": "param_name", "action": "add|remove", "required": true|false, "description_old": "original text (empty if add)", "description_new": "updated text (empty if remove)", "reason": "why this parameter was added or removed" }...(no more than three) ], {Code_Modification_Examples} } Design Note: parameter_changes focuses strictly on structural additions or removals. The required field is mandatory to clarify contract implications. tool_description_change is mandatory to ensure high-level documentation stays in sync with the parameter list.
Prompt for Tool Description Update Operator You are an expert Technical Writer specializing in MCP (Model Context Protocol) documentation. Your task is to simulate the iterative evolution of an MCP Server by rewriting the tool description to accurately reflect its functionality. ### Context Server Name: {SERVER_NAME} Server Description: {SERVER_DESC} Target Tool Status: {TARGET_TOOL_STATUS} (Note: This input contains the specific target tool’s details, including current tool’s name, description, inputSchema, outputSchema) ### Critical Constraints
29
1. Schema Immunity: You MUST NOT alter the tool’s name, inputSchema, outputSchema, or any internal logic. Your modifications are strictly limited to the description text field of the target tool. The structural integrity of the tool definition must remain intact. 2. Precision & Consistency: Rewrite the Target Tool Description based only on the provided inputSchema and outputSchema. - Strict Prohibition: Do not claim, imply, or promise any capabilities, parameters, or return values that are not explicitly defined in the provided schemas. Avoid hallucinating features or ignoring constraints (e.g., if a parameter is optional in the schema, do not describe it as mandatory). 3. Output Format Requirement: - You must output ONLY a single, valid JSON object. - The JSON object must contain exactly two keys: original_description, and modified_description. { "tool": { "original_description": "The exact value of the current tool description", "modified_description": "the newly generated tool description" } } - NO Markdown: Do not wrap the output in markdown code blocks. Do not include any explanations, introductory text, or trailing comments. The output must be a raw JSON string directly parsable by a machine. - Ensure the language of the modifications remains consistent with the original text whenever possible.
Prompt for Parameters Description Update Operator You are an expert Technical Writer specializing in MCP (Model Context Protocol) documentation. Your task is to simulate the iterative evolution of an MCP Server by rewriting the descriptions of specific parameters within a tool’s properties. ### Context Server Name: {SERVER_NAME} Server Description: {SERVER_DESC} Target Tool Status: {TARGET_TOOL_STATUS} (Note: This input contains the specific target tool’s details, including current tool’s name, description, inputSchema, outputSchema) ### Critical Constraints 1. Deep Schema Immunity: - You MUST NOT alter the name, type, format, enum, required status in inputSchema, or any structural definition of the parameters. - You MUST NOT add new parameters or remove existing ones. - Your modifications are strictly limited to the description string inside the inputSchema for the specified parameters. - The rest of the tool configuration (tool description, outputSchema, other unchanged parameters) must remain exactly as provided in the source context. 2. Precision & Consistency: - Rewrite the target parameter descriptions to precisely match their data types and constraints defined in the schema. - Strict Prohibition: Do not change the semantic meaning of the parameter
30
in a way that contradicts its type or validation rules. constraints that do not exist.
Do not hallucinate
3. Output Format Requirement: - Output ONLY a raw JSON: { "parameters": [ { "parameter_name": "the_name_of_the_parameter", "original_description": "the exact input description before modification", "modified_description": "the newly generated description" } ... ] } - NO Markdown: Do not wrap the output in markdown code blocks. Do not include any explanations, introductory text, or trailing comments. The output must be a raw JSON list string directly parsable by a machine. - Ensure the language of the modifications remains consistent with the original text whenever possible.
Prompt for Joint Description Update Operator You are an expert Technical Writer specializing in MCP (Model Context Protocol) documentation. Your task is to simulate the iterative evolution of an MCP Server by rewriting the descriptions of specific parameters within a tool’s properties and the tool’s description, ensuring strictly semantic adjustment without logic changes. ### Context Server Name: {SERVER_NAME} Server Description: {SERVER_DESC} Target Tool Status: {TARGET_TOOL_STATUS} (Note: This input contains the specific target tool’s details, including current tool’s name, description, inputSchema, outputSchema) ### Core Rule: SEMANTIC ONLY, NO LOGIC CHANGES - DO: Improve clarity, grammar, tone, conciseness. Clarify existing constraints found in the schema. - DON’T: Add features, change types/format/requirements, or imply capabilities not in the schema. - Critical: If schema says "optional", description cannot say "required". ### Output Requirement - Output ONLY a raw JSON: { "tool": { "original_description": "The exact value of the current tool description", "modified_description": "the newly generated tool description" }, "parameters": [ { "parameter_name": "the_name_of_the_parameter", "original_description": "the exact input description before modification", "modified_description": "the newly generated description" } ... ] }
31
- NO Markdown: Do not wrap the output in markdown code blocks. Do not include any explanations, introductory text, or trailing comments. The output must be a raw JSON list string directly parsable by a machine. - Ensure the language of the modifications remains consistent with the original text whenever possible.
Prompt for File Modification Instruction # Role You are an MCP Server Code Modification Expert. Based on the provided modification requirements, output the content of the modified TypeScript code. # Input - Server Name: {SERVER_NAME} - Server Description: {SERVER_DESC} - Target Tool: {TARGET_TOOL_NAME} - TypeScript Code: {ORIGINAL_CODE} # Modification Requirement {Operator_FUNCTION} # Constraints 1. Minimal Changes: Only modify the logic related to the Target Tool implementation, preserve all other code exactly as is. 2. Maximal Enhancements: Implement extensive functional and contextual modifications, while strictly preserving the core original functionality of the MCP server. 3. Valid Syntax: Ensure the output TypeScript code is compilable and runnable. 4. Valid Logic: Changes are based on the original logic, meaning evolution is achieved by leveraging the code logic of existing tools. Do not fabricate information sources, such as accessing self-invented URLs in the code or using data with unknown formats or structures. 5. Crucial: Make sure not to output the modified complete code. Wrap the entire JSON content inside Markdown code blocks with the json language identifier. - Format: { "key": "value" }
H.2
Prompts for Benchmark Construction
Presented below are the prompts utilized for three key tasks: extracting MCP server installation configurations, server function classification, multi-server task generation. Prompt for MCP Installation Config Extraction You are an expert MCP (Model Context Protocol) configuration extractor. Analyze the provided README content and generate a strictly valid JSON output with two top-level properties: 1. 2.
"config" – VS Code-compatible MCP server configuration object "metadata" – API key requirement analysis
32
Extraction Rules: ✓For config.mcpServers: - Identify the primary MCP server name from README (e.g., package name, title, or explicit server name) - Extract launch command pattern (e.g., npx <package>, python -m server, docker run...) - Split command into: - "command": Executable only (e.g., "npx", "python", "docker") - "args": Array of arguments (e.g., ["bazi-mcp"], ["-m", "my_server"]) - If multiple servers exist, include all under mcpServers with unique keys - If command pattern is ambiguous, use the most prominently documented launch method - Default to empty object {} for mcpServers if no launch instructions found ✓For metadata.requires_api_key: - true ONLY if README explicitly states external API credentials are required for basic operation (e.g., "You must set OPENAI_API_KEY to start the server") - false if: - Server works locally without external services - API keys are only for optional features - No credential requirements mentioned - Be conservative – prefer false when uncertain ✓For metadata.api_key_examples: - List ONLY environment variable names explicitly mentioned as API keys (e.g., ["GROQ_API_KEY", "ANTHROPIC_API_KEY"]) - Empty array [] if none detected Output Schema (STRICTLY VALID JSON): { "config": { "mcpServers": { "ServerName": { "command": "string (npx command)", "args": ["string"] } } }, "metadata": { "requires_api_key": boolean, "api_key_examples": ["string"] } } Critical Requirements: - NEVER omit top-level keys (config, metadata) - mcpServers must be an object (not array) with server names as keys - Output ONLY raw JSON – no explanations, markdown, or prefixes - Escape special characters properly for valid JSON - If no server detected: "mcpServers": {} - If no API keys mentioned: "api_key_examples": [] Input README: {readme_content}
33
Prompt for MCP Server Categorization You are a system architecture expert. Based on the following MCP server descriptions, define a set of high-level functional categories (e.g., Finance, File System, Browser, Authentication, etc.), assign each server to one or more categories, and provide a short description for each category. # Requirements: 1. Define intuitive, meaningful category names that reflect broad functional domains. 2. The assigned categories should be specific to a particular field and not vague or multiple fields. 3. Use no more than 10 distinct categories in total. 4. Aim to distribute the servers as evenly as possible across the categories. 5. Each server must be assigned to one category. 6. Output a valid JSON object with three keys: - "categories": a list of objects, each containing "name" and "description" fields; - "assignments": a mapping from each server name to a list of category names it belongs to; - The "description" for each category should be a concise sentence (10–20 words) explaining what kinds of services belong in it. Example output structure: { "categories": [ {"name": "Finance", "description": "Services related to payments, billing, and financial transactions."}, {"name": "FileSystem", "description": "Operating system files, documents, and other related content."} ], "assignments": { "payment-service": ["Finance"], "file-converter": ["FileSystem"] } } MCP servers to classify: {services_list}
Prompt for Multi-Server Task Generation ## Task Generate a realistic user question that requires tools from multiple MCP servers to resolve. ## Guidelines - The question must be concise and should reflect a practical, real-world workflow involving different servers (>=2). - Include sufficient contextual information to unambiguously determine all needed tool calls and their parameters, but do not use real user names. - Phrase it naturally—as if asked by a real user—without mentioning any tool names, server names, or technical internals. - The solution should require multiple tool invocations across servers. - Do not use all servers/tools—instead, select a coherent subset that covers distinct capabilities within one realistic scenario. - Rely only on the provided servers and tools—no external knowledge or assumptions. - The question may be written in English or in the same language as the tool descriptions.
34
- Regarding real-time issues, the time is set around {set_time}. - Cannot use files that do not exist locally unless you have previously saved them. ## Input Available MCP Servers: {SERVER_DESCRIPTIONS} Relevant files (if applicable): {FILE_EXISTS} Database Configuration (if applicable): - Host: 127.0.0.1 - Port: 3305 - User: root - Password: 123456 - Database Name: mydb ## Output Format Return exactly one response in the following XML structure: <response> <workflow_analysis> % Briefly explain how the selected servers interact in this scenario </workflow_analysis> <target_servers> <server name="server_name_1"> <tool>tool_name_a</tool> <tool>tool_name_b</tool> </server> <server name="server_name_2"> <tool>tool_name_c</tool> </server> % Add more servers/tools as needed </target_servers> <question> % Natural-language user question </question> </response>
H.3
Prompts for Benchmark Evaluation
We provide prompts for LLM-based task execution, rubric-based trajectory evalution and trajectory error type identification. Prompt for Task Execution You are an AI agent connected to the MCP (Model Context Protocol) server. Based on the user’s request, select and invoke tools to fulfill the task accurately and efficiently. Note that do not to ask any questions to the user, just call the tool to complete the task. Considering real-time performance, the time is set to be around {SET_TIME}. These files are stored in the local file system and serve as relevant resources for tool utilization. Add the prefix path ./anotation_path to the following file systems as an absolute path during use:
35
{FILE_EXISTS} Database Configuration (if applicable): - Host: 127.0.0.1 - Port: 3305 - User: root - Password: 123456 - Database Name: mydb
Prompt for Agent Trajectory Evaluation # System Role You are an impartial evaluator judging the quality of an AI agent’s multi-server tool-based task execution. You focus on three independent dimensions: Task Completion, Grounding, and Planning Efficiency. # User Instructions You must assign scores based ONLY on evidence from the task, solution, and tool usage trajectory. - Objective: Ignore language fluency, formatting, or politeness. - Justified: Every score must be backed by specific counts/percentages derived from the trajectory. - Independent: Evaluate each dimension separately. A flaw in one dimension (e.g., redundancy) should NOT lower the score of another (e.g., task completion) unless it directly caused failure. # Input Data TASK PRESENTED TO AGENT: {task} AVAILABLE TOOLS: {available_tools} AGENT TRAJECTORY FOR TASK COMPLETION: {trajectory} FINAL ANSWER {answer} # SCORING RUBRICS (1–10 PER SUBDIMENSION) Scores are derived from the DEFECT RATE calculated in the principles below. ## Task Fulfillment Measures: Did the agent achieve the user’s explicit goals? - 1–3: Perfectly completes 10–30% of requirements. - 4–6: Perfectly completes 40–60% of requirements. - 7–8: Perfectly completes 70–80% of requirements. - 9–10: Perfectly completes 90–100% of requirements. ## Information Grounding Measures: Are the agent’s final assertions supported by tool outputs? - 1–3: 10–30% of claims are perfectly grounded in tool outputs. - 4–6: 40–60% of claims are perfectly grounded in tool outputs. - 7–8: 70–80% of claims are perfectly grounded in tool outputs. - 9–10: 90–100% of claims are perfectly grounded in tool outputs. ## Planning Efficiency Measures: Did the agent avoid redundant or unnecessary tool calls? - 9–10: <10% of calls were redundant/unnecessary. - 7–8: 10–30% of calls were redundant/unnecessary.
36
- 4–6: - 1–3:
30–60% of calls were redundant/unnecessary. >60% of calls were redundant/unnecessary.
# CALCULATION METHODOLOGY ## Step 1: Define "Opportunities" (Denominator) for Each Dimension Task Fulfillment: Count the distinct, explicit sub-goals/requirements in the TASK PRESENTED TO AGENT. Grounding: Count the atomic factual claims or data points in the agent’s FINAL response. Planning Efficiency: Count the TOTAL number of tool calls made in the AGENT TRAJECTORY FOR TASK COMPLETION. ## Step 2: Identify "Issues" (Numerator) for Each Dimension Task Fulfillment Issues: Sub-goals that were FAILED, IGNORED, or INCORRECTLY executed. - Note: If a sub-goal was met but via a redundant path, it is STILL COUNTED AS MET for this dimension. Do not penalize efficiency here. Grounding Issues: Claims in the final answer that are NOT found in or CONTRADICT the tool outputs. Planning Efficiency Issues: Tool calls that were REDUNDANT (repeating previous successful calls), UNNECESSARY (not needed for any sub-goal), or RETRIED without parameter changes after failure. - Note: Necessary calls that were just "slow" or "sub-optimal" but not redundant are NOT issues. ## Step 3: Calculate Defect Rate & Map to Score - Defect Rate = (Issues / Opportunities) × 100% - Mapping: - 0–10% defects → Score 9–10 - 10–30% defects → Score 7–8 - 30–60% defects → Score 4–6 - 60–100% defects → Score 1–3 # How to Score: 1. When evaluating percentages, be EXTREMELY STRICT about what counts as "perfectly executed". 2. "Perfectly" means ALL of the following must be true: - Complete and accurate parameters (not just valid, but IDEAL) - Zero redundancy (no repeated or unnecessary calls) - Proper error handling (graceful recovery from ANY failure) - Efficient execution (minimal rounds) 3. If ANY of the above is missing, that portion is NOT perfectly executed (counts as 0%). 4. Example: Task completed correctly but with 1 redundant call = that portion is 0% perfect. # KEY PRINCIPLES: 1. ALWAYS calculate as percentage, NOT absolute numbers. 2. 10 errors in 100 calls (10%) = same score as 1 error in 10 calls (10%). 3. NORMALIZE by complexity - don’t punish complex tasks: - Simple task: 1 error/5 steps (20% defect) = Score 7 - Complex task: 4 errors/20 steps (20% defect) = Score 7 4. CRITICAL: Apply the STRICTEST interpretation of “perfectly executed”. If there’s ANY doubt, score lower. # FINAL REMINDER BEFORE SCORING: - Count ONLY truly perfect executions toward the percentage - Be your most critical self - find flaws first, then acknowledge successes - If you’re considering a score above 8, re-examine for ANY imperfection - Server count is IRRELEVANT - using more servers is NOT better
37
# OUTPUT FORMAT Return your evaluation scoring and reasoning in this exact JSON format. Return ONLY the JSON object. { "task_fulfillment_reasoning": "List total requirements vs. met requirements. Calculate defect rate.", "grounding_reasoning": "List total claims vs. unsupported claims. Calculate defect rate.", "planning_efficiency_reasoning": "List total tool calls vs. redundant/unnecessary calls. Calculate defect rate.", "task_fulfillment": <integer_score>, "grounding": <integer_score>, "planning_and_efficiency": <integer_score> }
Prompt for Test Case Generation # Server name: {server_name} server_description: {server_description} # Registered tool (schema summary) {tool_schema} # Task Generate multiple test cases to maximize parameter coverage and validate all constraints for the tool provided above. # Rules - Standard Execution: Provide cases that include all parameters (required + optional) and cases with only the minimum required fields. - Boundary Coverage: Test numeric ranges (min/max), string length limits, and specific enum values. - Validation Testing: Include cases with missing required fields, incorrect data types, or strings violating regex patterns. - Realism: Values in input_arguments must be semantically appropriate for the tool’s specific functionality. Output ONLY valid JSON (no markdown fences): { "tool_name": string, "test_cases": [ { "test_name": string, "target_parameters": [string], "input_arguments": { "parameter_name_1": "value_1", "parameter_name_2": "value_2", ... }, "expected_status": "Success" | "Validation Error", "objective": "description of the specific parameter or constraint being tested" } ] }
38
Prompt for Trajectory Error Type Identification # Role You are an expert analyst classifying LLM MCP agent trajectories into exactly one primary error category according to the six-dimensional orthogonal taxonomy below. # Taxonomy (Choose the single best category) 1. specification_syntax_error Technical violations of the tool-calling protocol. This includes: - Invalid parameter names or other syntax errors. - Missing required parameters or incorrect parameter names. - Calling tools that exist but providing values that violate the API schema/contract. 2. intent_tool_misalignment Selecting an inappropriate tool for the task from the available set. includes: - Failing to choose the correct tool to resolve the user’s intent. - Choosing a tool that does not align with the current sub-task.
This
3. semantic_parameter_error The tool and format are correct, but the parameter values are semantically wrong. This includes: - Semantic shifts (e.g., swapping identities, wrong dates). - Value hallucinations (filling info not grounded in context). - Missing critical information extraction or incorrect value types. 4. planning_logic_error Errors in the multi-step workflow orchestration. This includes: - Missing preconditions (e.g., acting before searching). - Infinite loops or getting stuck in a logic cycle. - Incomplete steps or prematurely ending the task. 5. operational_redundancy The task might succeed, but the path is inefficient or wasteful. This includes: - Repeatedly calling the same tool with identical inputs. - Performing unnecessary actions that do not contribute to the final result. 6. reasoning_observation_error Failure to correctly interpret or react to tool outputs (Observations). includes: - Misinterpreting structured data or status codes returned by the API. - Ignoring error messages and proceeding with false assumptions.
This
# Rules 1. Pick exactly one primary_category from the list above. 2. If the trajectory is perfectly correct and efficient, set primary_category to "none". 3. If multiple issues appear, choose the earliest root cause that led the agent astray. 4. Base your judgment on the provided Task, Tools, Final Answer, and Trajectory. 5. Select operational_redundancy only if you are sure that no other error types are present. # Input
39
## Task (User Question) {QUESTION} ## Available Tools {SERVERS_TOOLS} ## Final Answer {FINAL_ANSWER} ## Trajectory {TRAJECTORY} ## Optional Evaluation Hints {EVAL_HINTS} # Output (JSON only) Return ONLY a valid JSON object. { "primary_category": "<spec_syntax_err| intent_misalign| sem_param_err| plan_logic_err| op_redundancy| reason_obs_err| none>", "confidence": <number from 0 to 1>, "evidence": "<1-3 short sentences citing specific steps where the error occurred>", "analysis": "<detailed rationale in English or Chinese explaining why this specific category was chosen>" }
40