AgentSearchBench
AgentSearchBench: A Benchmark for AI Agent Search in the Wild Bin Wu∗,† Arastun Mammadli∗ , Xiaoyu Zhang, Emine Yilmaz Centre for Artificial Intelligence, University College London
arXiv:2604.22436v1 [cs.AI] 24 Apr 2026
Abstract The rapid growth of AI agent ecosystems is transforming how complex tasks are delegated and executed, creating a new challenge of identifying suitable agents for a given task. Unlike traditional tools, agent capabilities are often compositional and execution-dependent, making them difficult to assess from textual descriptions alone. However, existing research and benchmarks typically assume well-specified functionalities, controlled candidate pools, or only executable task queries, leaving realistic agent search scenarios insufficiently studied. We introduce AgentSearchBench, a largescale benchmark for agent search in the wild, built from nearly 10,000 real-world agents across multiple providers. The benchmark formalizes agent search as retrieval and reranking problems under both executable task queries and high-level task descriptions, and evaluates relevance using execution-grounded performance signals. Experiments reveal a consistent gap between semantic similarity and actual agent performance, exposing the limitations of description-based retrieval and reranking methods. We further show that lightweight behavioral signals, including executionaware probing, can substantially improve ranking quality, highlighting the importance of incorporating execution signals into agent discovery. Our code is available at https://github.com/Bingo-W/AgentSearchBench.
1
Introduction
The rapid emergence of AI agentic systems is reshaping how humans accomplish complex tasks by enabling execution to be delegated to autonomous agents across a wide range of domains (Fang et al., 2025a; Gao et al., 2026). Modern agents can reason, plan, and interact with external tools and services to complete multi-step objectives (Huang et al., 2024; Ferrag et al., 2025; Qin et al., 2025). This progress has led to a rapidly expanding ecosystem of agentic components, ranging from general-purpose assistants to highly specialized taskoriented modules. As humans increasingly rely on agents developed by diverse third-party providers, a fundamental challenge arises: how can suitable agents be reliably identified and selected for a given task? Addressing this challenge is critical not only for end users seeking effective task completion, but also for developers and orchestration systems aiming to compose scalable and robust agentic workflows (Fourney et al., 2024; Hu et al., 2025). However, identifying suitable agents is inherently challenging. Compared to traditional tools whose functionality is typically scoped to specific operations (Shi et al., 2025), agent capabilities are often more compositional and execution-dependent, making them difficult to assess without observing task outcomes. As a result, textual descriptions provide only a partial signal of real competence (Qu et al., 2025; Wu et al., 2025a; Fang et al., 2025b): agents with similar descriptions may perform differently in practice, while semantically dissimilar agents can achieve comparable results. This semantic–performance misalignment is further amplified in large and open agent ecosystems, where overlapping functionalities and non-uniform description formats make capability comparison difficult (Yuan et al., 2025b). Consequently, agent search is fundamentally more complex than conventional tool retrieval or model selection. ∗ Equal Contribution † Corresponding author: [email protected]
1
AgentSearchBench
ToolBench (Qin et al., 2024) ToolRet (Shi et al., 2025) TREC 2025 (Kanoulas et al., 2025) AgentSquare (Shang et al., 2025) OKC Bench (Yuan et al., 2025a) AgentSearchBench
Agent/LLM/Tool
#Candidates
Realistic
Task Type
Tool Tool LLM Agent Agent Agent
16,464 43,215 1,131 16 127 9,759
✓ ✓ ✓ ✗ ✗ ✓
Exec. Exec. Exec. Exec. Exec. Exec. / Non-exec.
Table 1: Comparison of AgentSearchBench with tool and agent retrieval benchmarks. “Realistic” indicates whether agents/tools are sourced from real-world platforms. “Task Type” indicates support for executable (Exec.) or non-executable (Non-exec.) task specifications. Despite growing interest in agentic systems, existing research and benchmarks have not yet provided a realistic setting for studying agent search (in Table 1). Prior work on tool retrieval and related benchmarks primarily assumes that functionality can be inferred from structured descriptions or well-specified interfaces (Qin et al., 2024; Shi et al., 2025), which does not capture the compositional and execution-dependent nature of agent capabilities. Meanwhile, recent studies on automated agentic system design typically evaluate methods in small-scale or controlled environments where candidate agents are clearly differentiated (Shang et al., 2025; Yuan et al., 2025a). Such assumptions differ substantially from open ecosystems, where many agents exhibit overlapping capabilities and must be selected under uncertainty. Furthermore, existing tool and agent selection benchmarks largely focus on executable task queries with predefined inputs and outputs, whereas real-world agent discovery often begins from high-level task descriptions that are not directly executable. As a result, performance-grounded agent search in realistic open ecosystems remains insufficiently studied. To bridge these gaps, we introduce AgentSearchBench, a large-scale benchmark for agent search built from nearly 10,000 real-world agents. AgentSearchBench captures the diversity of open agent ecosystems by including agents from different providers with varying description styles, capability granularity, and functional overlap. Built on this resource, we formalize agent search as retrieval and reranking problems under both executable task queries and high-level task descriptions. Crucially, agent relevance is defined using execution-grounded performance signals rather than textual similarity. We further develop a scalable evaluation pipeline that generates task instances and converts execution outcomes into fine-grained relevance annotations for both retrieval and ranking assessment. Through extensive benchmarking experiments, we reveal a consistent gap between semantic similarity and actual task performance, providing empirical evidence for the executiondependent nature of agent capabilities. Retrieval and reranking methods that rely primarily on matching task descriptions with agent documentation often fail to surface highperforming agents, particularly when search begins from high-level task descriptions where capability requirements are implicit. To better understand this limitation, we further study execution-aware probing, which augments description-based ranking with lightweight behavioral signals obtained from agent execution. Results show that even limited probing can substantially improve ranking quality, highlighting the importance of incorporating execution signals into realistic agent discovery pipelines. Our contributions can be summarized as follows: (1) We formulate agent search as a new retrieval and ranking problem under execution-dependent capability uncertainty. (2) We construct AgentSearchBench, a large-scale benchmark with nearly 10,000 real-world agents, supporting both executable task queries and high-level task descriptions under an executiongrounded evaluation framework. (3) We provide extensive empirical analysis revealing a substantial semantic–performance gap and demonstrate the effectiveness of lightweight behavioral probing for improving agent ranking.
2
Related Work
Agentic Systems and Orchestration. Recent advances in agentic systems have enabled autonomous agents to solve complex tasks through reasoning, planning, and tool interaction 2
AgentSearchBench
(Huang et al., 2024; Ferrag et al., 2025; Qin et al., 2025). As agent ecosystems rapidly expand, a key challenge is how to select suitable agents for a given task. Existing work focuses on agent design and orchestration, including multi-agent collaboration (Fourney et al., 2024; Hu et al., 2025) and workflow composition frameworks (Yue et al., 2025; Shang et al., 2025; Yuan et al., 2025a), typically assuming a predefined and limited set of candidates. However, in open ecosystems with many overlapping agents (Yuan et al., 2025b; Wu et al., 2025a; Zhang et al., 2025a), agent selection must be performed under significant capability uncertainty, motivating agent search as a distinct problem. Tool Retrieval and Selection. Existing works on tool retrieval and selection (Qin et al., 2024; Shi et al., 2025) aim to identify suitable tools for task execution. These methods typically retrieve tools based on textual descriptions or structured schemas (Tang et al., 2026; Lu et al., 2025; Qu et al., 2024), and are primarily evaluated on executable task queries with predefined inputs and outputs (Qin et al., 2024; Qu et al., 2024). While effective in such settings, these assumptions do not capture real-world agent search, which often begins from high-level and non-executable task descriptions. Moreover, agent capabilities are compositional, inconsistently documented, and execution-dependent, making textual similarity insufficient for assessing suitability. Recent work explores improving tool representations via schema unification or execution signals (Yuan et al., 2025b; Qu et al., 2025; Wu et al., 2025a), but mainly focuses on tool usage rather than discovery over large candidate pools. In contrast, we formulate agent search as a performance-grounded problem that supports both executable queries and high-level task descriptions. Information Retrieval and Learning-to-Rank. Information retrieval and learning-torank provide the foundation for modeling agent search (Robertson & Zaragoza, 2009; BehnamGhader et al., 2024). Existing methods estimate relevance using textual similarity or annotated labels (Craswell et al., 2021; 2025). However, they assume relevance is static and observable without interaction, which does not hold for agents. In agent search, relevance is inherently execution-dependent, requiring evaluation through task performance. We therefore extend retrieval and ranking to incorporate execution-grounded relevance signals.
3
Problem Formulation
3.1
Agent Search Problem
Agent search aims to retrieve and rank suitable agents from a large candidate repository given a user task. Let T denote the task specification, and let C = { a1 , a2 , . . . , an } denote the candidate agent repository. Each agent is represented by descriptive documentation and, when available, an executable interface. An agent search system defines a scoring function f ( a, T ) that estimates the relevance of an agent a ∈ C to task T , and produces a ranked list
Oranked = argsorta∈C f ( a, T ).
(1)
Agent search involves two objectives: (1) retrieving top-k agents capable of solving the task, and (2) ranking them according to task performance quality. Unlike traditional information retrieval, where relevance is determined by static content matching, agent search requires assessing functional capability through task execution. 3.2
Task Query and Task Description
Agent search operates under different levels of task specification. We consider two types of inputs: executable task queries and high-level task descriptions. Task Query. A task query Tq is a concrete and executable instruction that can be directly evaluated by running an agent. Task queries include both single-agent tasks and multi-agent tasks composed of multiple executable subtasks (Qin et al., 2024; Shi et al., 2025). Given a task query, the goal is to identify agents that can successfully complete the task and rank them based on their execution performance. Task Description. In many realistic scenarios, users provide high-level goals that are not directly executable. We denote such inputs as task descriptions Td . To evaluate agent 3
AgentSearchBench
Design a 2,500 sqft floor plan with 4 bedrooms … space for functionality and comfort …
3. Generating Relevance Labels
1. Task Query
Hybrid Retriever
1 to 5 Scoring
top k Sample Agent(s)
Generate Task
Add Context
Filter Closed Book Tasks
Given Task Generate Task Description
Select Top Tasks
Modernize a legacy codebase to current enterprise standards
Build a Rubric
Coherent Subset
Likert Scale
Executable Task Query
Select Candidates
Pool of Candidate Tasks
LLM Judge
Relevance Labels
Multi-Platform Executor
2. Task Description
Figure 1: Task and Relevance Label Generation Pipeline of AgentSearchBench. capability under these settings, each task description is associated with a set of executable task queries, Q(Td ) = {Tq1 , . . . , Tqm }, which instantiate the high-level goal under different concrete scenarios. Agent relevance is then determined based on performance across Q(Td ), enabling evaluation of consistent capability rather than success on a single task instance. 3.3
Task-Performance-Based Relevance
Relevance in agent search is defined based on task execution performance. For an executable task query Tq , we define the relevance of an agent a using a task completion score y( a, Tq ) = E( a, Tq ), where E evaluates the quality of the agent’s response (e.g., via LLM-as-a-judge). For a task description Td , relevance cannot be evaluated directly. Instead, we aggregate performance over its associated queries: y( a, Td ) =
1 ∑ y(a, Tq ). |Q(Td )| T ∈Q(T ) q
(2)
d
This formulation measures agent capability as consistent performance across multiple task instances, rather than relying on textual similarity or single-task outcomes. In practice, observed execution performance may not always align with documented capabilities. When an agent successfully solves a task that is not supported by its documentation, we treat the success as potentially less reliable than that of an agent whose documented functionality is consistent with its observed performance. Accordingly, we incorporate documentation–performance alignment as an auxiliary signal when constructing ranking labels, so that relevance reflects both task success and the reliability of that success.
4
AgentSearchBench: A Benchmark for Agent Search
We construct AgentSearchBench from a large-scale collection of real-world agents and generate tasks through a hierarchical process. We first create executable task queries, then derive high-level task descriptions grounded in query-level evidence. Relevance is obtained via execution-based evaluation and converted into retrieval and ranking labels. The overall pipeline is shown in Figure 1. 4.1
Large-Scale Realistic Agent Collection
We build a repository of nearly 10k agents collected from public platforms, including GPT store1 , Google Cloud Marketplace2 , and AgentAI Platform3 . By sourcing agents from realworld ecosystems rather than synthesizing them, AgentBase captures practical challenges 1 https://chatgpt.com/gpts 2 https://cloud.google.com/marketplace 3 https://agent.ai/
4
AgentSearchBench
such as capability overlap and inconsistent documentation, enabling realistic evaluation of agent search. 4.2
Task Query Construction
We synthesize executable task queries from agent documentation following documentgrounded task generation (Qin et al., 2024). To reduce evaluation cost, we retrieve a candidate set of top-K agents using a hybrid scoring function that combines BM25 (lexical) (Robertson & Zaragoza, 2009), BGE (semantic) (Xiao et al., 2024), and ToolRet (tool-aware) retrieval (Shi et al., 2025): s( a, Tq ) = α slexical + β ssemantic + γ stool .
(3)
Retrieved agents are executed on each query and evaluated using a 5-point LLM-as-judge (Gu et al., 2024). We filter out degenerate queries where no agent succeeds, or all agents succeed. Multi-agent queries are constructed by composing executable subtasks from capabilityaligned clusters. We retain a composed query only if it semantically entails all subtasks via natural language inference (NLI) (Wu et al., 2025b): (i )
Entail(Tqmulti , Tq ) = 1, ∀i. 4.3
(4)
Task Description Construction
We construct task descriptions by abstracting high-level objectives from clusters of semantically related queries. Given query clusters {C1 , . . . , C M }, we remove outliers and generate a description Td for each cluster using LLMs. To associate executable queries with each description, we apply a rubric-based judge (Sharma et al., 2026) that evaluates the relevance of each candidate query Tq to Td from mul tiple aspects. Formally, let r(Td , Tq ) = r1 (Td , Tq ), . . . , r D (Td , Tq ) denote the aspect-wise relevance scores. For each aspect d, we select the top-2 queries according to rd , and construct S the associated query set as Q(Td ) = dD=1 Top2Tq rd (Td , Tq ), resulting in 10 queries when D = 5. To ensure reliable evaluation, we re-evaluate high-performing agents on missing subtasks and filter inconsistent task descriptions. 4.4
Relevance Annotation
Relevance is derived from execution-based performance using a 5-point LLM-as-judge. For retrieval, we convert scores into binary labels: rel( a, Tq ) = 1(y( a, Tq ) ≥ 4).
(5)
For multi-agent queries and task descriptions, we define graded relevance based on subtask completion: r ( a, T ) =
1 ∑ rel(a, Tq ). |S(T )| T ∈S(T )
(6)
q
To account for documentation–performance misalignment, agents that successfully complete tasks without corresponding documented capability are assigned discounted relevance scores (e.g., 0.5). These signals are used to construct golden rankings for evaluation.
5
Benchmark Statistic
We summarize the scale of AgentSearchBench in Figure 2. The benchmark contains nearly 9,760 agents collected from multiple open platforms, among which 7,867 provide executable 5
AgentSearchBench
Data Analytics Customer Support Visual Content Professional Advisory General Purpose Prompting Content Production Software Engineering Research and Analysis Financial Services Healthcare Services Education Services Lifestyle Services Creative Writing Social Platforms Career Services
Statistics # Total Agents # Total Task - # Single-Agent Task Query - # Multi-Agent Task Query - # Task Description Avg. query per description Avg. evaluated agents per query # Total executions
9,759 3,211 2,452 500 259 10 20 66,740
(a) Overview statistics.
11.3% 10.1% 9.9% 9.6% 9.2% 9.2% 9.2% 8.9%
3.4% 3.3% 3.3% 2.9% 2.8% 2.6% 2.3% 2.1%
(b) Agent diversity.
(c) Task diversity.
Figure 2: Benchmark statistics and semantic diversity of AgentSearchBench.
10 5 10
20
30
40
# Relevant Agents Task Description
50
20
40
60
80
100
# Relevant Agents
120
140
Single-Agent Multi-Agent
15 10 5 0
60
% Queries
% Tasks
15.0 12.5 10.0 7.5 5.0 2.5 0.0
0
80
% Agents
15
0
20
% Queries
% Tasks
20
Multi-Agent Task Query
Task Query
Single-Agent Multi-Agent
30 25 20 15 10 5 0
60 40 20
0.0
0.2
0.4
0.6
Normalized Entropy Task Description
0.8
0
1.0
1
2
3
4
5
6
7
8
9
1
2
3
4
5
6
7
8
9
# Subtasks Covered Task Description
40
% Agents
Task Query
25
30 20 10
0.0
(a) Relevant agents per query.
0.2
0.4
0.6
Normalized Entropy
0.8
(b) Score entropy.
1.0
0
# Subtasks Covered
(c) Subtask coverage.
Figure 3: Difficulty of Task Query and Task Description of AgentSearchBench. interfaces. We construct 2,952 executable task queries and 259 task descriptions, each associated with an average of 10 queries. Each query is evaluated on the top-20 retrieved agents, resulting in 66,740 execution runs. These statistics highlight the scale and executioncentric design of AgentSearchBench. Diversity and Difficulty. As shown in Figure 2(b–c), both agents and tasks exhibit broad and long-tailed semantic distributions, reflecting diverse capability coverage. Figure 3(a) shows that many queries have multiple relevant agents, making retrieval alone insufficient. Meanwhile, the entropy distribution in Figure 3(b) indicates substantial performance variance across agents, motivating fine-grained reranking. Finally, Figure 3(c) shows that agents typically cover only a subset of subtasks, highlighting partial and overlapping capabilities. Overall, AgentSearchBench provides a realistic and challenging setting for performance-grounded agent search.
6
Agent Search Evaluation
6.1
Experimental Setup
We evaluate agent search under both executable task queries and high-level task descriptions. For retrieval, methods search over the full agent repository using binary relevance labels derived from execution outcomes. For reranking, each method is given the top-20 agents with highest execution-based relevance and is evaluated against the golden ranking induced by aggregated subtask completion performance. For retrieval, we report Precision, Recall, NDCG, and Completeness (Qu et al., 2024; Shi et al., 2025). For reranking, we report NDCG and Completeness using graded relevance labels. We compare representative methods from four retrieval families, including sparse, dense, tool-aware, and decoder-only embedding models (Robertson & Zaragoza, 2009; Formal et al., 2021; Santhanam et al., 2022; Izacard et al., 2022; Ni et al., 2022; Wang et al., 2021; Xiao et al., 2024; Shi et al., 2025; Lu et al., 2025; Qu et al., 2024), and four reranking families, including cross-encoders, tool-specific rankers, decoder-only rerankers, and LLM-based 6
AgentSearchBench
Model
Type @5
NDCG @10 @20
@5
Precision @10 @20
@5
Recall @10 @20
Completeness @5 @10 @20
SPLADE v2 BM25 ColBERT v2 Contriever MS-Marco GTR-T5 Base MiniLM-L6 v2 BGE-Large v1.5 COLT ToolLens COLT ToolBench Tool-Embed ToolRet E5-Mistral 7B Qwen-Embedding 8B
Sparse Sparse Dense Dense Dense Dense Dense Tool Tool Tool Tool Decoder-only Decoder-only
4.09 32.41 27.71 20.75 23.20 29.02 31.78 9.35 16.25 34.02 37.52 19.57 25.25
3.49 26.73 23.07 17.43 19.50 25.28 28.24 8.12 13.84 28.40 31.73 16.31 22.59
Task Query 3.48 4.09 3.19 22.68 32.41 24.56 20.47 27.71 21.14 16.72 20.75 16.11 17.55 23.20 18.05 22.46 29.02 23.57 26.14 31.78 26.48 7.91 9.35 7.59 12.28 16.25 12.81 25.67 34.02 26.37 28.87 37.52 29.36 15.29 19.57 14.99 21.43 25.25 21.32
2.29 14.07 13.44 11.12 11.51 14.97 17.64 5.47 8.00 17.23 19.36 9.77 14.83
0.35 2.62 2.12 1.72 1.89 2.29 2.48 0.81 1.30 2.50 2.85 1.60 1.94
1.35 9.33 7.94 6.34 6.87 8.63 10.07 2.92 4.82 9.77 10.97 6.05 7.81
3.72 21.36 19.68 17.07 16.92 21.33 25.49 7.85 12.13 24.46 27.80 15.62 21.14
2.90 20.81 16.52 15.04 16.00 18.23 20.18 6.95 11.74 20.22 21.91 15.27 16.15
9.21 38.10 33.89 30.73 32.09 36.52 38.61 18.07 25.39 38.14 42.18 30.71 33.17
18.93 51.38 43.95 45.11 44.63 51.52 52.52 30.61 38.66 53.91 57.53 44.49 48.33
SPLADE v2 BM25 ColBERT v2 Contriever MS-Marco GTR-T5 Base MiniLM-L6 v2 BGE-Large v1.5 COLT ToolLens COLT ToolBench Tool-Embed ToolRet E5-Mistral 7B Qwen-Embedding 8B
Sparse Sparse Dense Dense Dense Dense Dense Tool Tool Tool Tool Decoder-only Decoder-only
12.02 16.35 15.38 15.38 15.87 20.67 23.08 11.06 12.50 21.15 21.15 15.87 20.67
8.24 13.42 13.19 15.03 15.03 17.69 19.26 10.11 12.64 20.08 19.87 14.78 18.84
Task Description 6.93 12.02 7.50 10.78 16.35 12.50 12.80 15.38 12.31 14.22 15.38 14.81 12.96 15.87 14.52 15.84 20.67 17.02 16.07 23.08 18.08 8.60 11.06 9.90 11.57 12.50 12.60 17.19 21.15 19.81 17.21 21.15 19.13 11.56 15.87 14.71 16.51 20.67 18.08
6.11 9.25 12.19 13.37 11.92 14.57 14.40 7.88 10.84 15.94 15.77 10.22 15.29
0.35 0.46 0.27 0.31 0.34 0.59 0.42 0.18 0.28 0.42 0.37 0.33 0.46
0.83 1.28 1.14 1.49 1.56 1.84 1.94 1.05 1.49 2.01 2.02 1.74 1.87
2.49 3.69 5.12 5.70 4.93 6.24 6.12 3.47 4.53 6.41 6.69 4.58 6.15
0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.48 0.48 0.00 0.48 0.96 0.00 0.48 0.48 0.96 0.48 0.00
0.48 0.48 1.92 1.92 1.92 1.44 3.37 0.96 1.92 1.92 3.37 1.92 2.40
Table 2: Retrieval results on Task Query Tq (Single-Agent and Multi-Agent Task Queries) and Task Descriptions Td . We highlight the best performance in each type of model. Model
Random Shuffle MiniLM-L12 v2 MXBAI Reranker Large BGE Reranker v2 Tool-Rank 4B Tool-Rank 8B MonoT5 Base MS-Marco Qwen Reranker 0.6B Qwen Reranker 4B RankGPT Qwen-3 32B RankGPT LLaMA-3.3 70B RankGPT GPT-5.2
Type
/ Cross-Encoder Cross-Encoder Cross-Encoder Tool Tool Decoder-only Decoder-only Decoder-only LLM-based LLM-based LLM-based
@1
NDCG @5 @20
51.43 48.27 52.09 63.09 63.84 66.67 57.84 63.34 64.67 61.00 61.09 63.59
50.98 48.06 53.42 59.84 64.45 64.36 57.37 63.72 64.53 61.05 59.92 64.57
Completeness @1 @5
Task Query 74.61 28.50 73.53 28.50 76.32 30.00 79.59 30.50 81.78 28.00 81.96 30.50 78.69 27.00 81.08 30.50 81.97 29.50 80.18 29.00 79.80 29.50 81.88 30.50
61.00 52.00 58.00 59.00 59.50 60.00 60.00 58.50 62.50 58.50 53.50 56.50
@1
NDCG @5 @20
Completeness @1 @5
53.00 48.00 52.84 54.31 52.24 53.92 52.31 53.96 58.00 53.95 52.60 66.00
Task Description 48.27 76.60 0.00 57.49 81.58 0.00 61.33 82.22 0.00 60.55 81.97 0.00 61.12 82.32 0.00 61.97 82.76 0.00 59.10 81.33 0.00 60.47 82.04 0.00 60.58 82.84 0.00 60.78 82.36 0.00 61.24 82.34 0.00 64.66 84.69 0.00
0.00 0.00 0.48 0.96 0.48 0.96 0.96 0.48 0.00 0.48 0.96 0.00
Table 3: Reranking results on Task Query Tq (Single-Agent and Multi-Agent Task Queries) and Task Descriptions Td . We highlight the best performance in each type of model. rankers (Wang et al., 2021; Chen et al., 2024; Li et al., 2025; Lu et al., 2025; Nogueira et al., 2020; Zhang et al., 2025b; Sun et al., 2023; Ma et al., 2024). All methods retrieve or rerank a fixed top-K candidate set under the same evaluation protocol. 6.2
Benchmarking Analysis
Table 2 shows retrieval performance under execution-based relevance. On task queries, toolaware retrievers outperform dense and sparse baselines, while on task descriptions, dense retrievers become more competitive, with BGE achieving the strongest overall performance. 7
AgentSearchBench
2.5
5.0
7.5
10.0
12.5
# Included Ranks
15.0
17.5
20.0
(a) Golden ranking accumulated agent scores on 2452 Single-Agent Task Queries.
101
100
Oracle (Golden Rank) Dense Sparse Decoder-Only Tool-Specific 2.5
5.0
7.5
10.0
12.5
# Included Ranks
15.0
17.5
20.0
(b) Golden ranking accumulated agent scores on 500 Multi-Agent Task Queries.
Accumulated Golden Score
Oracle (Golden Rank) Dense Sparse Decoder-Only Tool-Specific
100
Accumulated Golden Score
Accumulated Golden Score
101
101
Oracle (Golden Rank) Dense Sparse Decoder-Only Tool-Specific
100
2.5
5.0
7.5
10.0
12.5
# Included Ranks
15.0
17.5
20.0
(c) Golden ranking accumulated agent scores on 259 Task Descriptions.
Figure 4: The Gap between Surface-matching and Execution.
However, performance drops substantially when moving from executable queries to highlevel task descriptions, and completeness remains low across all methods, highlighting the difficulty of retrieving agents that can fully satisfy abstract requirements. Overall, these results indicate that while retrieval can capture coarse relevance, it struggles to identify agents with comprehensive task-solving capability, especially under high-level task specifications without explicit executable demands. Table 3 shows reranking results on execution-grounded candidate pools. On task queries, different model families achieve broadly similar performance, suggesting that surface-level signals are often sufficient for relatively concrete tasks. On task descriptions, however, decoder-only and LLM-based rerankers perform more strongly, possibly because their stronger generative capacity helps infer latent or implicit requirements behind high-level task descriptions. Nevertheless, completeness remains limited, showing that improved ordering does not fully resolve the challenge of identifying agents that can completely satisfy complex requirements. To further examine this limitation, Figure 4 compares accumulated golden performance under model rankings with the oracle ranking. All methods remain substantially below the oracle, with gains distributed gradually rather than concentrated at top ranks. This indicates that many high-performing agents are ranked too low, reflecting a misalignment between agent documentation and actual execution performance. This misalignment leads to a persistent semantic–performance gap, where documentation-based matching fails to accurately capture execution effectiveness.
6.3
Benchmark Validation
We validate the realism of our benchmark by comparing retrieval trends on synthetic queries with those on external realistic benchmarks. As shown in Figure 5a, representative sparse, dense, and tool-aware retrievers exhibit consistent relative trends across settings: dense and tool-aware methods remain substantially stronger than sparse retrieval, while absolute performance on realistic queries is notably lower. This suggests that our synthetic benchmark preserves the relative performance ordering of different methods while providing a controlled evaluation setting. In contrast, realistic queries are inherently more difficult, as they do not guarantee the existence of highly relevant agents in the candidate pool, resulting in lower absolute performance. We further validate the reliability of LLM-based relevance annotation by comparing it with human judgments. Following the relevance label in our AgentSearchBench, we focus on binary relevance signals by grouping scores of 4–5 as positive and 1–3 as negative, which are the primary signals used in our benchmark. We conduct a human evaluation on 500 execution instances with three AI PhD-level annotators and observe high agreement between LLM-based and human judgments, with a Cohen’s kappa of κ = 0.93 and accuracy 96.67%. These results indicate that LLM-as-judge provides reliable supervision for large-scale evaluation, supporting its use for constructing execution-grounded relevance labels. 8
Synthetic
HLE
Finance
0.285
0.30 NDCG@5
0.25
NDCG@5 (Full-document Indexing)
Synthetic vs Realistic Benchmarks (HLE and Finance) 0.35
0.251
0.236
0.194
0.191 0.166
0.20
0.147
0.15 0.10 0.047 0.022
0.05 0.00
BM25
BGE
0.675
y=x
Full-doc indexing wins
0.650 0.625
MiniLM-L12 v2 BGE Reranker v2 MXBAI Reranker Large Tool-Rank 4B Tool-Rank 8B MonoT5 Base MS-Marco Qwen Reranker 0.6B Qwen Reranker 4B RankGPT GPT-5.2 RankGPT LLaMA-3.3 70B RankGPT Qwen-3 32B
0.600 0.575 0.550 0.525 0.500 0.50
0.55
0.60
0.65
NDCG@5 (Full-document Indexing)
AgentSearchBench
NDCG@5 (Description-only Indexing)
ToolRet
y=x
0.70 Full-doc indexing wins 0.68 0.66
MiniLM-L12 v2 BGE Reranker v2 MXBAI Reranker Large Tool-Rank 4B Tool-Rank 8B MonoT5 Base MS-Marco Qwen Reranker 0.6B Qwen Reranker 4B RankGPT GPT-5.2 RankGPT LLaMA-3.3 70B RankGPT Qwen-3 32B
0.64 0.62 0.60 0.58 0.56 0.54
0.550 0.575 0.600 0.625 0.650 0.675 0.700
NDCG@5 (Description-only Indexing)
(a) NDCG@5: Comparison between (b) NDCG@5: description-only (c) NDCG@5: description-only Realistic and Synthetic Single-agent vs full-document indexing on vs full-document indexing on Task Queries. 2,952 Task Queries. 259 Task Descriptions.
Figure 5: Comparison between indexing and query realism. 60
Low variance
Medium variance
Win Rate (%)
55
53.1 50.0
50
48.5 45.5
45
42.4
40
39.4
25
39.4 36.4
35 30
High variance 53.1
34.4
33.3 30.3
bge
qwen-4B
tool-8B
Model
NDCG@5
NDCG@10
BGE Reranker v2 w/ Probing Tool-Rank 8B w/ Probing Qwen Reranker 4B w/ Probing RankGPT GPT-5.2 w/ Probing
57.93 58.16 ↑+0.40% 60.82 61.71 ↑+1.46% 60.96 61.91 ↑+1.56% 61.25 59.60 ↓-2.69%
80.76 80.82 ↑+0.07% 82.09 82.24 ↑+0.18% 82.12 82.38 ↑+0.32% 82.40 82.94 ↑+0.66%
rankgpt-gpt
(a) NDCG@5: win rate v.s. probe score variance
(b) Execution-Aware Probing enhancements on 100 Task Descriptions Td .
Figure 6: Execution-Aware Probing for Reranking on Task Description. 6.4
Execution-Aware Probing
We next study whether lightweight behavioral signals can improve agent ranking. First, we compare description-only indexing with full-document indexing that additionally includes usage examples. As shown in Figure 5b and Figure 5c, most methods improve under full-document indexing. These observations suggest that usage examples, often provided and verified by developers through execution, offer behavioral evidence beyond static descriptions. We then investigate explicit probing, where LLMs generate probing queries, candidate agents are executed on these queries, and the resulting responses are used as additional ranking signals. Figure 6a shows that probing is most effective when the probing responses exhibit medium or high variance across agents, whereas low-variance probes provide limited discrimination. Using these execution-derived signals, most rerankers achieve consistent improvements as shown in Figure 6b, demonstrating that lightweight behavioral probing can complement description-based ranking and better capture execution-level capability.
7
Conclusion
We introduce AgentSearchBench, a large-scale benchmark for agent search in open ecosystems, covering nearly 10,000 real-world agents and supporting both executable task queries and high-level task descriptions. By grounding relevance in execution outcomes, our benchmark reveals a substantial semantic–performance gap: methods based on textual similarity often fail to identify the best-performing agents. While existing retrieval and reranking approaches provide useful coarse signals, they remain limited in capturing executiondependent capability, especially for abstract and multi-step tasks. We show that incorporating lightweight behavioral signals, such as richer indexing and execution-aware probing, can improve ranking quality. These results highlight the importance of execution-aware 9
AgentSearchBench
methods and establish AgentSearchBench as a practical testbed for performance-grounded agent search.
References Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. Llm2vec: Large language models are secretly powerful text encoders. CoRR, abs/2404.05961, 2024. doi: 10.48550/ARXIV.2404.05961. URL https://doi.org/10.48550/arXiv.2404.05961. Antoine Bigeard, Langston Nashold, Rayan Krishnan, and Shirley Wu. Finance agent benchmark: Benchmarking llms on real-world financial research tasks. CoRR, abs/2508.00828, 2025. doi: 10.48550/ARXIV.2508.00828. URL https://doi.org/10.48550/arXiv.2508. 00828. Center for AI Safety, Scale AI, and HLE Contributors Consortium. A benchmark of expertlevel academic questions to assess AI capabilities. Nature, 649:1139–1146, 2026. doi: 10.1038/s41586-025-09962-4. URL https://arxiv.org/abs/2501.14249. Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. M3embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, volume ACL 2024 of Findings of ACL, pp. 2318–2335. Association for Computational Linguistics, 2024. doi: 10.18653/V1/2024.FINDINGS-ACL. 137. URL https://doi.org/10.18653/v1/2024.findings-acl.137. Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Jimmy Lin. Ms marco: Benchmarking ranking models in the large-data regime. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval, pp. 1566–1576, 2021. Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, Jimmy Lin, Ellen M Voorhees, and Ian Soboroff. Overview of the trec 2022 deep learning track. arXiv preprint arXiv:2507.10865, 2025. Jinyuan Fang, Yanwen Peng, Xi Zhang, Yingxu Wang, Xinhao Yi, Guibin Zhang, Yi Xu, Bin Wu, Siwei Liu, Zihao Li, Zhaochun Ren, Nikos Aletras, Xi Wang, Han Zhou, and Zaiqiao Meng. A comprehensive survey of self-evolving AI agents: A new paradigm bridging foundation models and lifelong agentic systems. CoRR, abs/2508.07407, 2025a. doi: 10.48550/ARXIV.2508.07407. URL https://doi.org/10.48550/arXiv.2508.07407. Wei Fang, Yang Zhang, Kaizhi Qian, James Glass, and Yada Zhu. Play2prompt: Zero-shot tool instruction optimization for llm agents via tool play. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 26274–26290, 2025b. Mohamed Amine Ferrag, Norbert Tihanyi, and Mérouane Debbah. From LLM reasoning to autonomous AI agents: A comprehensive review. CoRR, abs/2504.19678, 2025. doi: 10.48550/ARXIV.2504.19678. URL https://doi.org/10.48550/arXiv.2504.19678. Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. SPLADE v2: Sparse lexical and expansion model for information retrieval. CoRR, abs/2109.10086, 2021. URL https://arxiv.org/abs/2109.10086. Adam Fourney, Gagan Bansal, Hussein Mozannar, Cheng Tan, Eduardo Salinas, Erkang Zhu, Friederike Niedtner, Grace Proebsting, Griffin Bassman, Jack Gerrits, Jacob Alber, Peter Chang, Ricky Loynd, Robert West, Victor Dibia, Ahmed Awadallah, Ece Kamar, Rafah Hosn, and Saleema Amershi. Magentic-one: A generalist multi-agent system for solving complex tasks. CoRR, abs/2411.04468, 2024. doi: 10.48550/ARXIV.2411.04468. URL https://doi.org/10.48550/arXiv.2411.04468. 10
AgentSearchBench
Huan-ang Gao, Jiayi Geng, Wenyue Hua, Mengkang Hu, Xinzhe Juan, Hongzhang Liu, Shilong Liu, Jiahao Qiu, Xuan Qi, Qihan Ren, Yiran Wu, Hongru Wang, Han Xiao, Yuhang Zhou, Shaokun Zhang, Jiayi Zhang, Jinyu Xiang, Yixiong Fang, Qiwen Zhao, Dongrui Liu, Cheng Qian, Zhenhailong Wang, Minda Hu, Huazheng Wang, Qingyun Wu, Heng Ji, and Mengdi Wang. A survey of self-evolving agents: What, when, how, and where to evolve on the path to artificial super intelligence. Trans. Mach. Learn. Res., 2026, 2026. URL https://openreview.net/forum?id=CTr3bovS5F. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge. The Innovation, 2024. Mengkang Hu, Yuhang Zhou, Wendong Fan, Yuzhou Nie, Ziyu Ye, Bowei Xia, Tao Sun, Zhaoxuan Jin, Yingru Li, Zeyu Zhang, Yifeng Wang, Qianshuo Ye, Bernard Ghanem, Ping Luo, and Guohao Li. OWL: Optimized workforce learning for general multi-agent assistance in real-world task automation. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=MBJ46gd1CT. Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. Understanding the planning of LLM agents: A survey. CoRR, abs/2402.02716, 2024. doi: 10.48550/ARXIV.2402.02716. URL https: //doi.org/10.48550/arXiv.2402.02716. Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. Trans. Mach. Learn. Res., 2022, 2022. URL https://openreview.net/ forum?id=jKN1pXi7b0. Evangelos Kanoulas, Panagiotis Eustratiadis, Mark Sanderson, Jamie Callan, Yongkang Li, Jingfen Qiao, and Vaishali Pal. TREC 2025 Million LLMs Track. https://trec-mllm. github.io/, 2025. Xianming Li, Aamir Shakir, Rui Huang, Julius Lipp, and Jing Li. Prorank: Prompt warmup via reinforcement learning for small language models reranking. arXiv preprint arXiv:2506.03487, 2025. Xuan Lu, Haohang Huang, Rui Meng, Yaohui Jin, Wenjun Zeng, and Xiaoyu Shen. Tools are under-documented: Simple document expansion boosts tool retrieval. CoRR, abs/2510.22670, 2025. doi: 10.48550/ARXIV.2510.22670. URL https://doi.org/10. 48550/arXiv.2510.22670. Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. Fine-tuning llama for multi-stage text retrieval. In Grace Hui Yang, Hongning Wang, Sam Han, Claudia Hauff, Guido Zuccon, and Yi Zhang (eds.), Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, Washington DC, USA, July 14-18, 2024, pp. 2421–2425. ACM, 2024. doi: 10.1145/3626772.3657951. URL https://doi.org/10.1145/3626772.3657951. Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego, Ji Ma, Vincent Y. Zhao, Yi Luan, Keith B. Hall, Ming-Wei Chang, and Yinfei Yang. Large dual encoders are generalizable retrievers. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 9844–9855. Association for Computational Linguistics, 2022. doi: 10.18653/V1/2022.EMNLP-MAIN.669. URL https://doi.org/10.18653/v1/2022.emnlp-main.669. Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. Document ranking with a pretrained sequence-to-sequence model. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pp. 708–718. Association for Computational Linguistics, 2020. doi: 10.18653/V1/2020.FINDINGS-EMNLP.63. URL https://doi.org/10.18653/v1/2020.findings-emnlp.63. 11
AgentSearchBench
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to master 16000+ real-world apis. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreview.net/forum?id=dHng2O0Jjr. Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Xuanhe Zhou, Yufei Huang, Chaojun Xiao, Chi Han, Yi R. Fung, Yusheng Su, Huadong Wang, Cheng Qian, Runchu Tian, Kunlun Zhu, Shihao Liang, Xingyu Shen, Bokai Xu, Zhen Zhang, Yining Ye, Bowen Li, Ziwei Tang, Jing Yi, Yuzhang Zhu, Zhenning Dai, Lan Yan, Xin Cong, Yaxi Lu, Weilin Zhao, Yuxiang Huang, Junxi Yan, Xu Han, Xian Sun, Dahai Li, Jason Phang, Cheng Yang, Tongshuang Wu, Heng Ji, Guoliang Li, Zhiyuan Liu, and Maosong Sun. Tool learning with foundation models. ACM Comput. Surv., 57(4): 101:1–101:40, 2025. doi: 10.1145/3704435. URL https://doi.org/10.1145/3704435. Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-Rong Wen. Towards completeness-oriented tool retrieval for large language models. In Edoardo Serra and Francesca Spezzano (eds.), Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, CIKM 2024, Boise, ID, USA, October 21-25, 2024, pp. 1930–1940. ACM, 2024. doi: 10.1145/3627673.3679847. URL https://doi.org/10.1145/3627673.3679847. Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-Rong Wen. From exploration to mastery: Enabling llms to master tools via selfdriven interactions. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL https://openreview. net/forum?id=QKBu1BOAwd. Stephen E. Robertson and Hugo Zaragoza. The probabilistic relevance framework: BM25 and beyond. Found. Trends Inf. Retr., 3(4):333–389, 2009. doi: 10.1561/1500000019. URL https://doi.org/10.1561/1500000019. Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. Colbertv2: Effective and efficient retrieval via lightweight late interaction. In Marine Carpuat, Marie-Catherine de Marneffe, and Iván Vladimir Meza Ruı́z (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pp. 3715–3734. Association for Computational Linguistics, 2022. doi: 10.18653/V1/ 2022.NAACL-MAIN.272. URL https://doi.org/10.18653/v1/2022.naacl-main.272. Aamir Shakir, Darius Koenig, Julius Lipp, and Sean Lee. Boost your search with the crispy mixedbread rerank models, 2024. URL https://www.mixedbread.com/blog/ mxbai-rerank-v1. Yu Shang, Yu Li, Keyu Zhao, Likai Ma, Jiahe Liu, Fengli Xu, and Yong Li. Agentsquare: Automatic LLM agent search in modular design space. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL https://openreview.net/forum?id=mPdmDYIQ7f. Manasi Sharma, Chen Bo Calvin Zhang, Chaithanya Bandi, Ankit Aich, Huy Nghiem, Tahseen Rabbani, Ye Htet, Brian Jang, Aishwarya Balwani, Sumana Basu, Denis Peskoff, Clinton Wang, Marcos Ayestaran, Sean M. Hendryx, Brad Kenstler, and Bing Liu. Researchrubrics: A benchmark of prompts and rubrics for deep research agents. In The Fourteenth International Conference on Learning Representations, 2026. URL https: //openreview.net/pdf?id=ErnvfmSX0P. Zhengliang Shi, Yuhan Wang, Lingyong Yan, Pengjie Ren, Shuaiqiang Wang, Dawei Yin, and Zhaochun Ren. Retrieval models aren’t tool-savvy: Benchmarking tool retrieval for large language models. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 - August 1, 2025, volume ACL 2025 of Findings of 12
AgentSearchBench
ACL, pp. 24497–24524. Association for Computational Linguistics, 2025. URL https: //aclanthology.org/2025.findings-acl.1258/. Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. Is chatgpt good at search? investigating large language models as re-ranking agents. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 14918–14937. Association for Computational Linguistics, 2023. doi: 10.18653/V1/2023.EMNLP-MAIN.923. URL https://doi.org/10.18653/v1/2023. emnlp-main.923. Yichen Tang, Weihang Su, Yiqun Liu, and Qingyao Ai. Multi-field tool retrieval. CoRR, abs/2602.05366, 2026. doi: 10.48550/ARXIV.2602.05366. URL https://doi.org/10. 48550/arXiv.2602.05366. Qwen Team. Qwen3-tts technical report. CoRR, abs/2601.15621, 2026. doi: 10.48550/ARXIV. 2601.15621. URL https://doi.org/10.48550/arXiv.2601.15621. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 1116, 2024, pp. 11897–11916. Association for Computational Linguistics, 2024. doi: 10.18653/ V1/2024.ACL-LONG.642. URL https://doi.org/10.18653/v1/2024.acl-long.642. Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. Minilmv2: Multihead self-attention relation distillation for compressing pretrained transformers. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACL-IJCNLP 2021 of Findings of ACL, pp. 2140–2151. Association for Computational Linguistics, 2021. doi: 10.18653/V1/2021.FINDINGS-ACL.188. URL https://doi.org/ 10.18653/v1/2021.findings-acl.188. Bin Wu, Edgar Meij, and Emine Yilmaz. A joint optimization framework for enhancing efficiency of tool utilization in llm agents. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 22361–22373, 2025a. Yilin Wu, Han Yuan, Li Zhang, and Zheng Ma. Natural language inference as a judge: Detecting factuality and causality issues in language model self-reasoning for financial analysis. In Proceedings of The 10th Workshop on Financial Technology and Natural Language Processing, pp. 210–220, 2025b. Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. C-pack: Packed resources for general chinese embeddings. In Grace Hui Yang, Hongning Wang, Sam Han, Claudia Hauff, Guido Zuccon, and Yi Zhang (eds.), Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, Washington DC, USA, July 14-18, 2024, pp. 641–649. ACM, 2024. doi: 10.1145/3626772.3657878. URL https://doi.org/10.1145/3626772.3657878. Michelle Yuan, Khushbu Pahwa, Shuaichen Chang, Mustafa Devrim Kaba, MONICA SUNKARA, Jiarong Jiang, Xiaofei Ma, and Yi Zhang. Automated composition of agents: A knapsack approach for agentic component selection. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025a. URL https://openreview.net/forum?id= 1LPPMAUlaT. Siyu Yuan, Kaitao Song, Jiangjie Chen, Xu Tan, Yongliang Shen, Kan Ren, Dongsheng Li, and Deqing Yang. Easytool: Enhancing llm-based agents with concise tool instruction. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 951–972, 2025b. 13
AgentSearchBench
Yanwei Yue, Guibin Zhang, Boyang Liu, Guancheng Wan, Kun Wang, Dawei Cheng, and Yiyan Qi. MasRouter: Learning to route LLMs for multi-agent systems. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15549–15572, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.757. URL https: //aclanthology.org/2025.acl-long.757/. Haozhen Zhang, Tao Feng, and Jiaxuan You. Router-r1: Teaching LLMs multi-round routing and aggregation via reinforcement learning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025a. URL https://openreview.net/forum?id= DWf4vroKWJ. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176, 2025b. Yuanhang Zheng, Peng Li, Wei Liu, Yang Liu, Jian Luan, and Bin Wang. Toolrerank: Adaptive and hierarchy-aware reranking for tool retrieval. In Nicoletta Calzolari, MinYen Kan, Véronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC/COLING 2024, 20-25 May, 2024, Torino, Italy, pp. 16263– 16273. ELRA and ICCL, 2024. URL https://aclanthology.org/2024.lrec-main.1413.
14
AgentSearchBench
A
More Details about AgentSearchBench
A.1
Schema Design
A key challenge in constructing AgentBase lies in the heterogeneous representation of agents across platforms, where capability descriptions, usage instructions, and accessibility conditions are presented in inconsistent formats. To enable systematic indexing and fair comparison in agent search, we introduce a unified schema that standardizes agent information into four semantic groups: (1) Agent metadata provides stable identity and provenance signals, supporting deduplication and version tracking; (2) Capability description captures the functional semantics of agents through textual descriptions, category tags, and modality indicators; (3) Usage guidance characterizes how agents are invoked and interacted with in practice, including quick-start instructions and example interactions; (4) Availability and constraints record practical deployment conditions such as pricing, accessibility, base models, and update timestamps, reflecting real-world feasibility considerations. This structured representation enables scalable agent collection, consistent retrieval over heterogeneous sources, and reproducible evaluation of agent search systems. We show our schema design in Table 4 and one example in Table 5. A.2
Example of Task Query and Task Description
We show the example of single-and multi-agent task query, and task description in Table 6. A.3
Implementation Details of Benchmark Construction
We use GPT-5.2 as the backbone for all generation steps with temperature τ = 1. For task description generation, we begin with a candidate pool of 100 task queries, which are filtered down to 10 via rubric-based scoring across 5 criteria, each criterion associated with approximately 2 subtasks. For multi-agent task query we pick 2–4 subtasks (mean 2.91), sampled to be semantically related yet non-redundant. Concretely, given an anchor task query, we retrieve the top-K most similar candidates and skip the highest-ranked kskip to avoid near-duplicate subtasks. We retrieve a candidate set of top-K agents using a hybrid scoring function combining BM25 (Robertson & Zaragoza, 2009), BGE (Xiao et al., 2024), and ToolRet (Shi et al., 2025) as lexical, semantic, and tool-aware retrievers respectively: s( a, Tq ) = α slex + β ssem + γ stool
(7)
where α + β + γ = 1. Each component score is min-max normalised before aggregation, and the top-K agents are selected by the fused score.
B
More Details about Experimental Setup
B.1
Completeness Computation
For a given task T associated with M ≥ 1 subtasks {s1 , . . . , s M }, each with ground-truth relevant set Rsm , Completeness is defined as: COMP@K (T ) = I[∀ m ∈ {1, . . . , M} : πK (T ) ∩ Rsm ̸= ∅] ,
(8)
where πK (T ) denotes the top-K retrieved tools. A task is complete at K if the retrieved set contains at least one relevant tool per subtask. For single-agent task query; when M = 1, Completeness reduces to standard hit rate. B.2
The Baselines
We introduce more details of each type of model used in retrieval and reranking benchmark. 15
AgentSearchBench
Schema Group
Representative Fields
Agent Metadata Capability Description Usage Guidance Availability & Constraints
Versioned agent ID, name, official URL, platform source Functional description, category tags, supported modalities Quick-start instructions, example input–output interactions Pricing/accessibility, base model, update time, auxiliary metadata
Table 4: Unified schema for representing agents collected across platforms.
Field
Value
ID Source Name Description
agt:openaiagents:[email protected] openaiagents JAVA Code Guide A JAVA development assistant focusing on coding standards and quality. developer, browser, dalle, python GPT-5.2 multi-modal Explain JAVA exception handling standards. How can I improve this MySQL query? Review my JAVA code snippet. What are the best practices for JAVA unit testing? free https://chatgpt.com/g/g-EYiFThMtQ 26 Jan 2026 gizmoId: g-EYiFThMtQ; gpt rating: 4.0
Tools Model Input mode Quick start
Access URL Indexed Misc
JAVA Code Guide is an AI-powered bot that leverages advanced GPT technology to assist JAVA developers in writing clean, efficient, and high-quality code. With its extensive knowledge of JAVA best practices . . .
Table 5: An example agent from AgentBase.
Retrieval We evaluate four families of retrieval models: (1) Sparse, based on termweighting and learned sparse representations Robertson & Zaragoza (2009); Formal et al. (2021); (2) Dense, using bi-encoder architectures trained for semantic similarity Santhanam et al. (2022); Izacard et al. (2022); Ni et al. (2022); Wang et al. (2021); Xiao et al. (2024); (3) Tool-specific, retrievers explicitly trained on tool corpora Qu et al. (2024); Lu et al. (2025); Zheng et al. (2024); Shi et al. (2025); (4) Decoder-only, leveraging autoregressive LMs for retrieval Wang et al. (2024); Team (2026).
Reranking We evaluate four families of reranking models: (1) Cross-encoder, scoring query– document pairs jointly Wang et al. (2021); Shakir et al. (2024); Chen et al. (2024); (2) Toolspecific, rerankers trained on tool corpora Zheng et al. (2024); (3) Decoder-only, reranking via autoregressive scoring Nogueira et al. (2020); Zhang et al. (2025b); (4) LLM-based, prompting large language models to produce relevance judgements Sun et al. (2023). 16
AgentSearchBench
Type
Value
Task Description
Monitor, summarize, and compare news and web sources.
Single-Agent Task Query
Create a mind map (hierarchical bullet nodes) summarizing the Nature article at https://www.nature.com/articles/d41586-023-03988-2, capturing its main sections, subsections, and key takeaways. Assume I am a graduate student writing a quick briefing and need a clear, structured outline. I’m trying to reset my health and routine, so first ask me 5 questions about my schedule, priorities, and constraints, then build a 2-week productivity plan with a daily time-block schedule, a prioritized task list, and a simple tracker template, plus a 7-day Mediterranean meal plan at 1,800 kcal/day with recipes, step-by-step cooking tips, drink pairings, and a consolidated grocery list, and a 7-day English study plan with daily grammar drills, a vocabulary list, short speaking prompts, answer keys, and a simple progress tracker.
Multi-Agent Task Query
Table 6: An example Single-Agent Task Query, Multi-Agent Task Query, and a Task Description Td from AgentSearchBench.
B.3
The Prompts
Prompt: Execution-Aware Probing Listwise Ranker
—SYSTEM— You are an intelligent assistant that scores the quality of AI agent responses to a given probing query. I will provide you with [num] agent responses, each indicated by a number identifier [i]. Score them based on their response quality to the provided probing query. ## Scoring Guidelines (1–5) • 5 (Excellent): Fully satisfies the user’s request with correct, sufficient, and concise information. • 4 (Good): Mostly satisfies the request but has minor omissions or minor unnecessary detail. • 3 (Fair): Partially satisfies the request but is incomplete, indirect, or inefficient. • 2 (Poor): Minimally relevant or largely unhelpful. • 1 (Very Poor): Incorrect, irrelevant, or fails the task entirely. —USER— Probing Query: [probe] [for i in range(num)] [[i+1]] [responses[i]] [endfor] Score the [num] responses above based on their quality. The output format should be [score[i], score[i+1], . . . ], e.g., [1, 5, 3, 5, . . . ] for each response. Only respond with the ordered scoring results.
17
AgentSearchBench
C
More Results
C.1
Performance on Each Realistic Queries
We show the results on the queries from two realistic benchmark in Table 10, i.e., Humanity’s Last Exam (HLE) Center for AI Safety et al. (2026), and Finance Agent Benchmark Bigeard et al. (2025). C.2
More Results on Indexing
Model
Type
BM25 SPLADE v2 ColBERT v2 Contriever MS-Marco GTR-T5 Base MiniLM-L6 v2 BGE-Large v1.5 ToolRet COLT ToolBench COLT ToolLens E5-Mistral 7B Qwen-Embedding 8B
Sparse Sparse Dense Dense Dense Dense Dense Tool Tool Tool Decoder-only Decoder-only
@10
NDCG @20 @50
0.116 0.062 0.143 0.170 0.145 0.152 0.186 0.139 0.138 0.088 0.025 0.174
0.106 0.058 0.139 0.163 0.133 0.151 0.171 0.134 0.130 0.082 0.025 0.164
@10
Precision @20 @50
Task Description 0.088 0.109 0.095 0.054 0.057 0.053 0.136 0.136 0.131 0.153 0.167 0.155 0.124 0.138 0.121 0.139 0.155 0.148 0.160 0.182 0.160 0.124 0.142 0.131 0.125 0.137 0.123 0.077 0.088 0.077 0.026 0.027 0.026 0.153 0.164 0.152
0.068 0.044 0.112 0.123 0.099 0.114 0.129 0.103 0.102 0.063 0.023 0.124
@10
Recall @20
@50
Completeness @10 @20 @50
0.022 0.011 0.028 0.034 0.028 0.035 0.039 0.029 0.029 0.018 0.005 0.032
0.039 0.021 0.053 0.064 0.050 0.067 0.066 0.054 0.051 0.031 0.009 0.063
0.065 0.041 0.115 0.130 0.098 0.119 0.134 0.103 0.105 0.062 0.023 0.127
0.000 0.000 0.010 0.005 0.005 0.010 0.000 0.000 0.010 0.000 0.000 0.005
0.000 0.000 0.019 0.029 0.014 0.029 0.024 0.019 0.024 0.010 0.000 0.019
0.014 0.010 0.058 0.077 0.048 0.053 0.062 0.043 0.053 0.024 0.005 0.062
Table 7: Retrieval results on Task Description Td with full indexing.
Model
Type
BM25 SPLADE v2 ColBERT v2 Contriever MS-Marco GTR-T5 Base MiniLM-L6 v2 BGE-Large v1.5 ToolRet COLT ToolBench COLT ToolLens E5-Mistral 7B Qwen-Embedding 8B
Sparse Sparse Dense Dense Dense Dense Dense Tool Tool Tool Decoder-only Decoder-only
@1
NDCG @5 @20
0.332 0.096 0.317 0.323 0.296 0.327 0.371 0.322 0.304 0.234 0.079 0.297
Single-Agent Task Query 0.297 0.309 0.332 0.269 0.081 0.091 0.096 0.072 0.275 0.295 0.317 0.248 0.273 0.293 0.323 0.245 0.243 0.253 0.296 0.216 0.272 0.284 0.327 0.244 0.330 0.362 0.371 0.299 0.283 0.312 0.322 0.259 0.241 0.251 0.304 0.211 0.191 0.200 0.234 0.170 0.063 0.068 0.079 0.057 0.238 0.259 0.297 0.212
@1
Precision @5 @20 0.158 0.048 0.154 0.154 0.129 0.147 0.192 0.167 0.128 0.105 0.036 0.136
@1
Recall @5
@20
@1
Hit @5
@20
0.038 0.011 0.036 0.037 0.034 0.035 0.042 0.034 0.036 0.026 0.008 0.034
0.157 0.044 0.141 0.140 0.124 0.136 0.173 0.144 0.121 0.096 0.031 0.118
0.366 0.114 0.356 0.351 0.296 0.335 0.438 0.382 0.293 0.233 0.082 0.312
0.332 0.096 0.317 0.323 0.296 0.327 0.371 0.322 0.304 0.234 0.079 0.297
0.669 0.256 0.629 0.633 0.588 0.638 0.692 0.656 0.581 0.506 0.215 0.571
0.853 0.484 0.830 0.835 0.792 0.820 0.878 0.863 0.788 0.712 0.393 0.788
Table 8: Retrieval results on Single-Agent Task Query Tq with full indexing.
Model
Type
BM25 SPLADE v2 ColBERT v2 Contriever MS-Marco GTR-T5 Base MiniLM-L6 v2 BGE-Large v1.5 ToolRet COLT ToolBench COLT ToolLens E5-Mistral 7B Qwen-Embedding 8B
Sparse Sparse Dense Dense Dense Dense Dense Tool Tool Tool Decoder-only Decoder-only
@1
NDCG @5 @20
@1
@1
Recall @5
@20
@1
Hit @5
@20
Completeness @1 @5 @20
0.318 0.102 0.326 0.254 0.282 0.304 0.378 0.300 0.230 0.194 0.012 0.228
0.257 0.080 0.251 0.225 0.225 0.256 0.310 0.269 0.191 0.166 0.018 0.201
Multi-Agent Task Query 0.318 0.238 0.146 0.015 0.102 0.074 0.053 0.005 0.326 0.232 0.147 0.015 0.254 0.217 0.156 0.012 0.282 0.211 0.152 0.012 0.304 0.241 0.169 0.014 0.378 0.292 0.207 0.016 0.300 0.258 0.190 0.013 0.230 0.180 0.130 0.010 0.194 0.157 0.118 0.008 0.012 0.020 0.017 0.000 0.228 0.193 0.149 0.011
0.054 0.017 0.050 0.047 0.045 0.052 0.063 0.057 0.039 0.035 0.004 0.042
0.129 0.048 0.129 0.137 0.128 0.147 0.180 0.164 0.116 0.102 0.015 0.133
0.318 0.102 0.326 0.254 0.282 0.304 0.378 0.300 0.230 0.194 0.012 0.228
0.654 0.298 0.618 0.610 0.570 0.652 0.714 0.682 0.536 0.502 0.092 0.566
0.886 0.594 0.822 0.860 0.836 0.882 0.926 0.914 0.818 0.772 0.262 0.842
0.012 0.002 0.004 0.014 0.010 0.008 0.010 0.006 0.010 0.006 0.000 0.010
0.189 0.066 0.187 0.185 0.182 0.205 0.250 0.224 0.157 0.140 0.018 0.175
Precision @5 @20
0.074 0.012 0.034 0.070 0.048 0.060 0.076 0.094 0.054 0.040 0.000 0.046
Table 9: Retrieval results on Multi-Agent Task Query Tq with full indexing. 18
0.244 0.050 0.078 0.194 0.168 0.214 0.232 0.270 0.154 0.144 0.010 0.178
AgentSearchBench
Model
Type @1
NDCG @5 @20
Precision @5 @20
@1
@1
Recall @5
@20
BM25 SPLADE v2 ColBERT v2 BGE-large v1.5 E5 Mistral 7B ToolRet Tool-Embed
Sparse Sparse Dense Dense Decoder-only Tool Tool
HLE: Humanity’s Last Exam 0.042 0.022 0.028 0.042 0.017 0.021 0.013 0.025 0.021 0.008 0.104 0.066 0.071 0.104 0.050 0.208 0.191 0.194 0.208 0.150 0.042 0.057 0.070 0.042 0.058 0.250 0.194 0.225 0.250 0.150 0.271 0.181 0.174 0.271 0.142
0.011 0.012 0.026 0.076 0.032 0.089 0.060
0.006 0.003 0.022 0.020 0.007 0.033 0.043
0.013 0.014 0.039 0.163 0.040 0.137 0.116
0.028 0.036 0.080 0.258 0.098 0.301 0.181
BM25 SPLADE v2 ColBERT v2 BGE-large v1.5 E5 Mistral 7B ToolRet Tool-Embed
Sparse Sparse Dense Dense Decoder-only Tool Tool
Finance Agent Benchmark 0.061 0.047 0.049 0.061 0.000 0.050 0.067 0.000 0.030 0.043 0.044 0.030 0.182 0.166 0.233 0.182 0.121 0.125 0.138 0.121 0.242 0.147 0.198 0.242 0.212 0.116 0.117 0.212
0.020 0.024 0.017 0.065 0.039 0.068 0.030
0.009 0.000 0.003 0.046 0.035 0.058 0.048
0.024 0.046 0.033 0.157 0.106 0.087 0.078
0.053 0.120 0.048 0.365 0.156 0.266 0.111
0.042 0.048 0.036 0.115 0.085 0.103 0.079
Table 10: Retrieval results on 200 Real Task Query Tq .
Model
Type
MiniLM-L12 v2 BGE Reranker v2 MXBAI Reranker Large Tool-Rank 4B Tool-Rank 8B MonoT5 Base MS-Marco Qwen Reranker 0.6B Qwen Reranker 4B (payload trunc.) RankGPT GPT-5.2 RankGPT LLaMA-3.3 70B RankGPT Qwen-3 32B Random Shuffle
Cross-Encoder Cross-Encoder Cross-Encoder Tool Tool Decoder-only Decoder-only Decoder-only LLM LLM LLM Proxy
NDCG @5 @20
Completeness @1 @5 @20
MRR
ρ
τ
@1 0.580 0.530 0.480 0.580 0.440 0.480 0.580 0.580 0.480 0.380 0.580 0.410
Task Description 0.686 0.842 0.659 0.831 0.555 0.796 0.652 0.840 0.642 0.810 0.639 0.806 0.642 0.828 0.669 0.835 0.641 0.833 0.604 0.813 0.608 0.820 0.497 0.771
0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
0.218 0.185 0.215 0.108 0.134 0.164 0.157 0.126 0.133 0.125 0.142 0.105
0.280 0.283 0.185 0.268 0.184 0.115 0.205 0.279 0.403 0.264 0.260 0.095
0.229 0.223 0.149 0.209 0.138 0.083 0.158 0.216 0.322 0.210 0.202 0.073
0.397 0.400 0.415 0.417 0.372 0.380 0.409 0.411 0.424 0.421 0.393 0.350
0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
0.200 0.200 0.200 0.200 0.200 0.200 0.200 0.200 0.200 0.200 0.200 0.200
Table 11: Reranking results on Task Description Td with golden labels and full indexing. Model
MiniLM-L12 v2 BGE Reranker v2 MXBAI Reranker Large Tool-Rank 4B Tool-Rank 8B MonoT5 Base MS-Marco Qwen Reranker 0.6B Qwen Reranker 4B RankGPT GPT-5.2 RankGPT LLaMA-3.3 70B RankGPT Qwen-3 32B Random Shuffle
Type
Cross-Encoder Cross-Encoder Cross-Encoder Tool Tool Decoder-only Decoder-only Decoder-only LLM LLM LLM Proxy
@1
NDCG @5 @20
Completeness @1 @5 @20
0.535 0.650 0.480 0.655 0.700 0.550 0.650 0.660 0.650 0.580 0.605 0.460
Task Query 0.530 0.765 0.300 0.630 0.810 0.305 0.515 0.730 0.290 0.665 0.830 0.305 0.655 0.825 0.325 0.575 0.785 0.280 0.645 0.820 0.310 0.655 0.825 0.305 0.670 0.825 0.300 0.610 0.795 0.260 0.620 0.790 0.295 0.500 0.740 0.280
0.545 0.600 0.585 0.620 0.605 0.610 0.595 0.610 0.610 0.515 0.550 0.665
0.850 0.850 0.850 0.850 0.850 0.850 0.850 0.850 0.850 0.850 0.850 0.850
MRR
ρ
τ
0.360 0.340 0.300 0.380 0.330 0.340 0.360 0.360 0.365 0.355 0.360 0.205
0.040 0.115 0.110 0.260 0.250 0.120 0.220 0.250 0.245 0.165 0.165 0.030
0.030 0.110 0.090 0.215 0.210 0.100 0.180 0.210 0.205 0.135 0.140 0.025
0.450 0.470 0.440 0.490 0.480 0.450 0.475 0.485 0.485 0.470 0.460 0.370
Table 12: Reranking results on Task Query Tq (Single-Agent and Multi-Agent Task Queries) with golden labels and full indexing.
19