ScholarQuest: A Taxonomy-Guided Benchmark for Agentic Academic Paper Search in Open Literature Environments Tingyue Pan, Mingyue Cheng∗, Daoyu Wang, Yitong Zhou, Jie Ouyang, Qi Liu, Enhong Chen State Key Lab of Cognitive Intelligence, University of Science and Technology of China {pty12345,wdy030428,yitong.zhou,ouyang_jie}@mail.ustc.edu.cn {mycheng,qiliuql,cheneh}@ustc.edu.cn Abstract
(a) From Similarity Search to Agentic Search Agentic Search
arXiv:2606.20235v1 [cs.IR] 18 Jun 2026
Similarity-based Search
Academic paper search is a core step in scientific research, and LLM-based search agents are emerging as a promising paradigm for iterative, intent-driven literature exploration. However, existing benchmarks are insufficient for systematically evaluating agentic academic search under realistic open literature environments. We propose ScholarQuest, a large-scale, taxonomyguided benchmark for agentic academic paper search. ScholarQuest is constructed from over 1,000 computer science topics and four representative research intents, including methodoriented, setting-anchored, comparison-based, and scope-controlled queries. It further provides scalable answer construction and a shared retrieval backend ScholarBase for reproducible evaluation. Benchmarking results show that agentic methods outperform single-shot retrieval baselines, yet the best-performing agent only achieves 0.314 Recall@100 and 0.355 Recall@All, indicating substantial room for improvement. In addition, analyses of search efficiency, intent-level robustness, and failure cases further highlight the benchmark’s ability to provide multi-dimensional evaluation signals for academic paper search agents. Our code and data are publicly available1 .
1
Introduction
Academic paper search underpins effective knowledge discovery in scientific research (Timmins and McCabe, 2005; Marchionini, 2006). Traditional approaches mainly rely on lexical or semantic matching, offering efficient access to large-scale literature collections (Shi et al., 2025). However, when facing fine-grained and conditional queries, they remain limited by the single-shot ranked-list paradigm (Gusenbauer and Haddaway, 2020). This motivates the rise of agentic paper search, where LLM-based agents can autonomously decide when ∗ 1
Corresponding author. https://github.com/pty12345/ScholarQuest
Ranking
1st 2nd 3rd
1st 2nd 3rd
Papers
Papers
Fails to capture complex intent Rarely use citation relationships
Search
Expand
More relevant and richer coverage Closer to real research workflow
(b) We Need Better Evaluation for Agentic Search Existing Benchmark Query side • Human bias • Limited coverage Answer side • Hard to scale high-quality answer • Lack of standard evaluation
ScholarQuest Principles Taxonomy-guided queries Diverse domains and intents Standard evaluation
Figure 1: Motivation of ScholarQuest. Agentic paper search moves beyond similarity-based retrieval through iterative exploration. This calls for better benchmarks with controlled queries and standardized evaluation.
and how to search, expand, and refine candidate papers based on accumulated evidence (He et al., 2025; Pan et al., 2026). Despite this progress, the systematic evaluation of agentic academic paper search remains underdeveloped. Existing benchmarks are mainly limited in three aspects. On the query side, they often rely on manually constructed or paper-derived queries, which may introduce annotator bias and provide limited coverage of diverse research intents (He et al., 2025; Shi et al., 2025). On the answer side, constructing high-quality relevant paper sets is costly and difficult to scale, since relevant papers are often scattered across different terminologies and subfields (He et al., 2025). Moreover, existing benchmarks do not provide a publicly available and standardized evaluation environment, making toolcalling results difficult to reproduce across different systems. As shown in Table 1, these limitations make existing benchmarks insufficient for evaluating agentic paper search under broad topics, diverse query types, and reproducible environments. To address these challenges, we propose Schol-
Table 1: Comparison with existing academic paper search benchmarks. ScholarQuest differs in broader topic coverage, taxonomy-guided queries, controlled intents, and automatically built answers. Benchmark
Topic Scope
Query Source
Construction
Query Intent Design
Answer Source
RealScholar (He et al., 2025) AutoScholar (He et al., 2025) SPARBench (Shi et al., 2025)
CV / NLP Seed Papers AI / NLP
Human Answer-derived Human
Query-first Answer-first Query-first
Uncontrolled Uncontrolled Uncontrolled
Human-labeled Related Work Human-labeled
1,000+ CS topics
Taxonomy-guided
Query-first
4 intent types
Automatically-built
ScholarQuest
arQuest, a large-scale, taxonomy-guided evaluation benchmark for agentic academic paper search. ScholarQuest covers over 1,000 computer science topics and includes four common types of research queries: method-oriented, setting-anchored, comparison-based, and scope-controlled queries. These query types reflect common scientific search needs, such as finding papers using a specific method, studying a problem under a given setting, comparing technical claims, or restricting the search scope. This taxonomy-guided design gives ScholarQuest broad topic coverage and controlled query diversity beyond ad-hoc query collection. ScholarQuest also provides scalable answer construction and a standardized literature environment. To build high-quality answer sets, we develop an automated pipeline that combines initial retrieval, citation expansion, multi-stage relevance filtering, and quality verification. This pipeline aims to improve both coverage and precision while reducing the cost of large-scale manual annotation. To support reproducible evaluation, we further build a million-scale testbed ScholarBase based on the arXiv database. Beyond storing paper metadata and citation relations, ScholarBase provides unified retrieval, inspection, and citation-expansion interfaces, enabling different systems to be evaluated under the same controlled literature environment. Built on ScholarBase, ScholarQuest evaluates academic search systems beyond final retrieval quality, covering Recall@k, search efficiency, tooluse behavior, and robustness across research intents and answer-set sizes. Experiments show that agentic methods outperform single-shot retrievers, but still face limitations in efficiency, constraintsensitive search, and robustness under different query conditions. Failure analysis further shows that many errors stem from off-target exploration rather than insufficient search effort. These findings highlight the need to evaluate not only what papers are retrieved, but also how the search process is conducted. We release the benchmark and our retrieval backend ScholarBase to support re-
producible research on academic paper search. Our contributions are summarized as follows: • We propose ScholarQuest, a large-scale, taxonomy-guided benchmark for agentic academic paper search, covering over 1,000 CS topics, four research intents, and a millionscale paper retrieval backend ScholarBase. • We build a scalable construction pipeline that integrates taxonomy-guided query generation, multi-source retrieval, citation expansion, relevance filtering, and quality verification. • We benchmark representative search systems and identify key limitations of current agents in search efficiency, scope control, and robustness across query conditions.
2
Related Work
2.1
Academic Paper Search Benchmarks
Academic paper search is essential for both human researchers and deep-search agents that rely on trustworthy scholarly evidence (Wang et al., 2025; Cheng et al., 2026; Jin et al., 2025). Recent benchmarks have begun to evaluate LLM-based systems for academic paper search. PaSa introduces AutoScholarQuery, a large synthetic dataset constructed from top-tier AI conference publications, and RealScholarQuery, a small set of realworld academic queries for realistic evaluation (He et al., 2025). AutoScholarQuery provides useful scale, but its construction is tied to source papers and their Related Work section, which may inherit topic and citation biases. RealScholarQuery improves realism, but its limited size makes it difficult to systematically cover fine-grained CS topics and diverse query intents. SPARBench further improves annotation quality through expert-screened queries and multi-source relevance validation, but it remains small-scale with broad domain coverage rather than a systematic CS topic hierarchy (Shi et al., 2025). In contrast, our benchmark offers taxonomy-grounded topic coverage, controllable query distributions, and broader gold answer pools.
2.2
LLM-based Paper Search Agents
LLM-based methods have recently been explored for academic paper search. Agentic workflowbased approaches, such as PaSa, formulate paper search as a multi-step process that searches, inspects, and selects relevant papers (He et al., 2025). SPAR further integrates query understanding, multi-source retrieval, citation-based exploration, and reranking for training-free academic retrieval (Shi et al., 2025). Building on this line of work, PaperScout improves tool-use autonomy by enabling the agent to decide when and how to invoke search and expansion actions according to the evolving search state (Pan et al., 2026). These studies show the promise of agentic literature exploration beyond conventional single-shot retrieval. ScholarQuest complements them by providing a systematic benchmark to evaluate such agents across retrieval quality, search efficiency, multi-turn decision ability, and robustness under diverse research intents.
3
ScholarQuest
In this section, we first establish a clear task formulation, then describe the benchmark construction pipeline, followed by the four research intents used to evaluate different retrieval capabilities. 3.1
Task Formulation
ScholarQuest evaluates academic paper search as a multi-turn retrieval task. Given a research query q and a literature database D, a search system iteratively analyzes the current paper pool Pt and issues retrieval or exploration requests to D. The returned papers are merged into Pt+1 , allowing the system to accumulate candidate papers and update search state over multiple turns. The system finally returns a ranked paper list P̂, which is evaluated against the ground-truth relevant paper set P ∗ . 3.2
Benchmark Construction
3.2.1 Query Generation We first collect over 1,600 topics from the ACM Computing Classification System (CCS), which provides a poly-hierarchical organization of computing topics (Rous, 2012). To focus the benchmark on computer science paper search, we utilize Qwen3-Max (Yang et al., 2025) to map each ACM CCS topic to one or more arXiv subject categories according to the arXiv category taxonomy2 . We 2
https://arxiv.org/category_taxonomy
Comparisonbased 15.3% Scopecontrolled 28.5%
Methodoriented 27.2% Settinganchored 29.0%
Figure 2: Dataset distribution of ScholarQuest. Side bars show query counts across arXiv CS categories, while the center pie shows query intent distribution.
retain topics assigned to arXiv CS categories and discard topics outside the computer science scope. This process results in over 1,000 CS topic seeds, which serve as the input for query generation. Given each topic seed, we further utilize Qwen3Max to generate research queries under four intent types: method-oriented, setting-anchored, comparison-based, and scope-controlled queries. This design allows us to systematically cover diverse academic search intents rather than relying on queries derived from individual papers. We then apply deduplication and quality control to remove queries that are ambiguous, overly broad, overly narrow, duplicated, or difficult to judge from paper titles and abstracts. After filtering, we obtain 1,111 high-quality queries for ScholarQuest. 3.2.2
Answer Discovery
Multi-source Candidate Retrieval. In the retrieval stage, we collect an initial candidate set that serves as the starting point for subsequent expansion. For each query, we use both the original query and multiple LLM-generated rewrites, including method-focused, task-focused, broadertopic, narrower-topic, and terminology-based formulations. These queries are issued to multiple complementary retrieval sources, including Google Search API3 , arXiv API4 , and Semantic Scholar API5 . We retain only papers that can be matched to the arXiv database, and merge duplicated results 3
https://serper.dev https://info.arxiv.org 5 https://www.semanticscholar.org 4
Method Oriented cs.CV
Setting Anchored
Video Summarization
Constraint: RL
cs.CY
Which papers explore the use of deep reinforcement learning for video summarization? arXiv: 2505.23268 Title: Unsupervised Transcriptassisted Video Summarization... Abstract: We propose a multimodal arXiv: 1807.03089 pipeline ... To generate a more Title: Video Summarisation by Classifica- condensed ... The pipeline is trained tion with Deep Reinforcement Learning within an RL framework.
Which papers evaluate privacy policies in the healthcare domain? 1 arXiv: 2306.11557 arXiv: 2405.17971
arXiv: 1801.00054
Title: Deep Reinforcement Learning for Unsupervised Video Summarization…
cs.SE
Software Maintenance
Title: A Comparative Audit of Privacy Policies from Healthcare Organizations..
Constraint: Static Analysis
arXiv: 2304.12015 Title: ITER: Iterative Neural Repair for Multi-Location Patches Abstract:Automated program repair has achieved promising results ... we propose an iterative program repair paradigm called ITER ...
Title:A Qualitative Framework for mHealth Privacy Practices
arXiv: 2410.14607 arXiv: 2511.11347 Title: Privacy Challenges...in RAG- Title:Evaluating Privacy Measures Enhanced LLMs for Healthcare Chatbots... in Healthcare Apps ...
cs.LG
I am looking for papers on software maintenance tools that exclude static analysis tools from their scope. arXiv: 2008.01676 Title: Anchor: Locating Android Framework-specific Crashing Faults Abstract: Android framework-specific app crashes are hard to debug ... we propose Anchor, a two-phase suspicious bug location suggestion tool ...
Constraint: Healthcare
Privacy Policies
Transfer Learning
Constraint: Random Initialization
Which papers report that transfer learning outperforms random initialization in model performance? arXiv: 1909.10955 Title: Efficiently Reusing Old Models z Across Languages via Transfer Learning Abstract: It is especially true in transfer learning for ... We achieve better translation quality and ... than when training from random initialization ...
Scope Controlled
arXiv: 1811.01137 Title: Transfer Learning in Neural Machine Translation ... Abstract: ... When compared with training an NMT model from scratch, our approach allows us to reach higher performance ...
Comparison Based
Figure 3: Cases of the four research-intent types in ScholarQuest. Each quadrant presents a representative specifictype query with its domains, topic seed, constraints and example gold papers from the answer set.
by arXiv identifier. Formally, the retrieval-based candidate pool is Cqret =
[
[
Retrieves (r),
(1)
r∈R(q) s∈S
where R(q) contains the original query and its rewrites, S denotes the retrieval sources. Citation-Graph Expansion. Since direct retrieval may miss relevant papers with different terminology, we further expand the retrieved candidates through citation relations. Starting from high-confidence papers in Cqret , we collect their references and citing papers when available, and selectively perform second-hop expansion from highconfidence neighbors while pruning noisy branches. The final candidate pool is defined as Cq = Cqret ∪ Cqcite ,
(2)
where Cqcite denotes papers obtained from citationgraph expansion. This step complements direct retrieval by recovering papers that are connected through scholarly relations but may be missed by lexical or semantic matching. Multi-stage Relevance Adjudication. We derive the final answer set from the expanded candidate pool through multi-stage relevance adjudication. First, we remove clear mismatches using retrieval scores, metadata signals, and a small relevance model, while keeping the filtering stage
recall-oriented. The remaining candidates are evaluated by multiple LLM-based relevance judges using the query, paper title, and abstract. Each judge assigns a score from 0 to 2, denoting mismatch, partial match, and strict match, respectively. After aggregating the scores into Rel(q, p), we select papers above threshold τ = 2: Aq = {p ∈ Cq | Rel(q, p) ≥ τ }.
(3)
Borderline and high-disagreement cases are further adjudicated to reduce labeling errors. 3.2.3 Human Audit To assess label quality without requiring exhaustive manual annotation, we conduct a targeted human audit over 450 query-paper pairs stratified by automatic relevance score. Human annotators judge relevance based on the query, paper title, abstract, and metadata. The audit results show that final highconfidence positives have 86.0% strict-match precision and 98.7% relaxed precision, where relaxed precision counts both strict and partial matches. The audit also helps localize residual false-negative risk to borderline candidates, motivating the additional adjudication of high-disagreement cases. Detailed statistics are provided in Appendix A. 3.3
Key Features of ScholarQuest
ScholarQuest covers four representative research intents in academic paper search. As shown in Figure 3, each type reflects a common search scenario and targets a distinct retrieval capability.
Table 2: Recall-oriented results on ScholarQuest. We report Recall@25, Recall@100, and Recall@All across four query types and the overall query set. The bold indicates the best result, while underline indicates the second-best. Method
Method Oriented
Setting Anchored
Scope Controlled
Comparison Based
Overall
R@25 R@100 R@All R@25 R@100 R@All R@25 R@100 R@All R@25 R@100 R@All R@25 R@100 R@All Dense Retrieval Hybrid Retrieval
0.155 0.153
0.300 0.303
0.403 0.110 0.344 0.115
0.219 0.226
0.306 0.029 0.257 0.038
0.087 0.091
0.159 0.136 0.111 0.133
0.234 0.250
0.326 0.104 0.276 0.107
0.208 0.214
0.290 0.244
Google Search Google Scholar Semantic Scholar DeepXiv (Qian et al., 2026)
0.106 0.084 0.064 0.091
0.146 0.111 0.085 0.214
0.167 0.143 0.101 0.341
0.059 0.073 0.103 0.068
0.094 0.102 0.116 0.181
0.117 0.120 0.125 0.304
0.003 0.004 0.001 0.007
0.006 0.010 0.002 0.023
0.013 0.010 0.002 0.064
0.122 0.107 0.089 0.039
0.135 0.125 0.103 0.118
0.143 0.130 0.103 0.286
0.068 0.071 0.057 0.054
0.094 0.100 0.084 0.138
0.112 0.113 0.099 0.288
PaSa (He et al., 2025) 0.245 SPAR (Shi et al., 2025) 0.242 PaperScout (Pan et al., 2026) 0.275
0.345 0.333 0.408
0.366 0.230 0.351 0.223 0.451 0.246
0.296 0.282 0.341
0.312 0.103 0.293 0.102 0.366 0.091
0.193 0.188 0.182
0.242 0.256 0.222 0.246 0.239 0.275
0.294 0.280 0.327
0.324 0.201 0.301 0.197 0.358 0.214
0.281 0.270 0.314
0.310 0.291 0.355
Method-oriented Queries. This dimension focuses on finding papers that use, extend, or analyze a specific method. It tests whether systems can identify methodological relevance beyond surfacelevel keyword overlap. Setting-anchored Queries. This category targets papers under a specific task, dataset, or experimental setting. It evaluates whether systems can satisfy fine-grained contextual constraints rather than retrieve broadly related papers. Scope-controlled Queries. This feature specifies explicit boundaries for the desired paper set, such as a target domain, task, or model family. It evaluates whether systems can control retrieval breadth and avoid loosely relevant results. Comparison-based Queries. This aspect focuses on papers that compare methods, assumptions, results, or technical claims. It tests whether systems can capture relational search intents involving multiple research objects.
4
Experiment
In this section, we evaluate the overall retrieval performance and analyze agentic search behavior in terms of efficiency and robustness. Finally, we diagnose shared failure patterns across search agents. 4.1
Experimental Setup
Dataset. We evaluate all methods on ScholarQuest, which contains 1,111 high-quality queries constructed from over 1,000 computer science topic seeds. As shown in Figure 2, the dataset spans a broad range of arXiv CS categories, covering major areas such as cs.LG, cs.CV, cs.AI, and cs.CL, while also preserving a long-tail distribution over diverse subfields. The queries are divided into four intent types: method-oriented, setting-anchored, scope-controlled, and comparison-based, with rel-
(a) Avg. Toolcall
(b) Avg. Search
(c) Avg. Expand
(d) Avg. Rounds
(e) Avg. Candidates
(f) R@100 / 100 Cand.
PaSa
PaperScout
SPAR
Figure 4: Efficiency and process statistics of paper search agents, including number of tool calls, rounds, observed candidates, and R@100/100 Cand., which normalizes Recall@100 by every 100 observed candidates.
atively balanced proportions of 27.2%, 29.0%, 28.5%, and 15.3%, respectively. Environment. We build ScholarBase, an opensource paper-search environment constructed from the S2 PaperData snapshot (Lo et al., 2020), which retains arXiv papers with abstracts, and stores paper metadata and citation relations. It supports BM25 (Harman, 1995) sparse retrieval with SQLite FTS56 , dense retrieval with Qdrant over BGEM3 (Chen et al., 2024) title–abstract embeddings, and RRF-based hybrid retrieval (Cormack et al., 2009). The environment further provides RESTful APIs for semantic search, title matching, metadata lookup, and citation/reference traversal, enabling all methods to search, inspect, and expand papers under the same controlled literature backend. Baselines. We compare three groups of baselines. The first group includes standard retrieval baselines implemented in our ScholarBase, including dense retrieval and RRF-based hybrid search. The second group includes widely used academic search systems, including Google Search7 , Google 6 7
https://sqlite.org https://serper.dev
Figure 5: Recall@All density distributions of agentic paper search methods. Panels (a)–(c) show query-type distributions within each method, while panel (d) compares the three methods over all queries. The x-axis is Recall@All and the y-axis is probability density.
Scholar8 , Semantic Scholar API9 and DeepXiv (Qian et al., 2026). The third group includes opensource agentic paper search systems, including PaSa (He et al., 2025), SPAR (Shi et al., 2025), and PaperScout (Pan et al., 2026). Metrics. Considering that paper search emphasizes comprehensive answer coverage, we primarily report Recall@25, Recall@100, and Recall@All under different retrieval budgets. To analyze the search process, we further report the number of interaction rounds, search calls, expansion calls, observed candidates, and Recall@100 per 100 observed candidates. We also examine robustness across research intents and answer-set sizes. 4.2
Main Results
Table 2 reports Recall@25, Recall@100, and Recall@All across four research intents and the overall query set. Traditional retrieval methods show clear limitations on complex paper search queries, especially on compositional constraints, as shown by the low R@100 of Google Search (0.006) and Google Scholar (0.010) on scope-controlled queries. Dense and Hybrid retrieval improve over sparse retrieval, achieving 0.208 and 0.214 overall R@100. However, they remain single-shot retrievers and cannot iteratively refine queries, expand promising papers, or integrate intermediate evidence. In contrast, agentic methods with multi-round decision-making achieve higher overall R@100 scores, ranging from 0.270 to 0.314. 8 9
https://scholar.google.com https://www.semanticscholar.org
Figure 6: Recall trajectories under increasing tool-call budgets. The upper and lower panels show Recall@All and Recall@50, respectively, with shaded regions indicating query-level variation.
The results also show clear differences among agentic search methods. PaperScout achieves 0.408 R@100 on method-oriented, outperforming PaSa and SPAR by 0.063 and 0.075, respectively. It also leads on setting-anchored and comparison-based, with R@100 scores of 0.341 and 0.327, higher than PaSa (0.296, 0.294) and SPAR (0.282, 0.280). One possible explanation is that more adaptive multiturn search can help agents better navigate scattered literature evidence. We conduct further analysis in the next section to examine tool-use patterns, search efficiency, and exploration behavior. 4.3
Analysis of Agentic Search Behavior
Beyond overall retrieval performance, we further analyze agentic search behavior from the perspectives of search efficiency, research-intent robustness, and answer-set-size robustness. Search Efficiency. Retrieval quality alone is insufficient for paper search, where agents should avoid exhaustive candidate expansion. As shown in Figure 4, the three agents exhibit distinct tooluse patterns. PaSa uses the most tool calls (60.1), mainly expansions (55.1), and observes the largest candidate set (744), but its recall efficiency is relatively low at 0.051 Recall@100 per 100 candidates. SPAR is also expansion-oriented, with 39.1 expansion calls out of 47.1 tool calls, observing 515 candidates and achieving 0.064 recall efficiency. In contrast, PaperScout uses fewer expansions (19.0) while making more search calls (15.3) across more interaction rounds (9.2), leading to fewer observed candidates (408) and the highest recall efficiency
Table 3: Average Recall@All across gold answer-set size buckets. #Query denotes the number of queries. Answer Size 5–25 26–50 51–100 101–150 151–200
Recall@All
#Query 457 218 225 130 81
PaSa
SPAR
PaperScout
0.352 0.319 0.267 0.265 0.235
0.338 0.307 0.242 0.239 0.205
0.405 0.371 0.304 0.281 0.278
(0.120). This efficiency advantage may stem from its autonomous tool-use policy, which allows PaperScout to decide when and how to search or expand based on the evolving search state. Figure 6 further shows recall trajectories under different tool-call budgets. PaSa improves rapidly at the early stage but soon saturates, while SPAR shows steadier but more limited gains. PaperScout starts more gradually, but continues to improve with additional interactions and eventually reaches higher recall under a smaller or comparable toolcall budget. This suggests that adaptive tool-use can improve the marginal utility of each tool call. Robustness Across Research Intents. Agentic methods generally outperform traditional retrieval methods across metrics. However, their performance varies substantially across research intents, revealing important limitations of current paper search agents. As shown in Table 2, all three agents perform relatively well on method-oriented, settinganchored, and comparison-based queries, where the search intent is mainly guided by positive semantic cues such as methods, tasks, settings, or comparative claims. In contrast, scope-controlled is consistently the most challenging category: PaSa, SPAR, and PaperScout achieve only 0.193, 0.188, and 0.182 R@100, respectively, which are clearly lower than their performance on other intents. Figure 5 further provides a distributional view of this limitation. For all three agents, the recall distributions of scope-controlled queries are concentrated in the low-recall region, indicating that this category is not only worse on average but also more prone to query-level failures. Unlike positiveintent queries, scope-control queries require agents to retrieve papers that match the main topic while preserving exclusion constraints or fine-grained scope boundaries during multi-round exploration. Although PaperScout shows a more right-shifted overall recall distribution, its scope-control curve
Table 4: Common zero-recall cases by query type. #Failures denotes the number of queries where all three agents retrieve no gold papers; Avg. Candidates report the average accessed candidate papers. Query Type
Avg. Candidates
#Failures PaSa
SPAR PaperScout
Comparison-based Scope-controlled Setting-anchored Method-oriented
7 6 5 2
926.6 871.2 713.6 1306.0
589.9 618.0 551.8 831.0
405.7 315.3 472.8 532.0
Overall
20
894.6
612.9
408.0
remains heavily skewed toward low recall. These results suggest that adaptive tool use improves robustness, but constraint-sensitive paper search remains a shared limitation of agentic methods. Robustness Across Answer-set Sizes. We further evaluate robustness by grouping queries according to the number of ground-truth answers. As shown in Table 3, all agents exhibit lower average recall as the answer set becomes larger, suggesting that broad-answer queries remain challenging under limited search budgets. For instance, PaSa drops from 0.352 in the 5–25 bucket to 0.235 in the 150–200 bucket, while SPAR drops from 0.338 to 0.205. PaperScout achieves the highest average recall in every bucket, with scores of 0.405, 0.371, 0.304, 0.281, and 0.278 from small to large answer sets. Its advantage persists even in the largest bucket, where it outperforms PaSa and SPAR by 0.043 and 0.073 average recall, respectively. These results indicate that PaperScout better balances focused exploration and broad coverage across varying answer-set sizes. More detailed results are shown in Table 7. 4.4
Failure Analysis
We analyze common zero-recall cases, where PaSa, SPAR, and PaperScout all fail to retrieve any gold paper. As shown in Table 4, such failures appear across all four query types. Although these cases are rare, covering only 20 out of 1,111 complete agentic queries (1.80%), they are highly diagnostic: all three agents access a substantial number of candidate papers on these queries, yet none of the retrieved papers match the gold answers. Figure 7 presents representative examples for each query type, showing that failures arise from different forms of intent mismatch rather than a uniform retrieval error. For claim-comparison queries, agents retrieve related papers but miss
Failure Analysis
Search Query Claim Comparison BQ_000367: Which papers report that autonomous robots outperform human teleoperation in specific tasks? Scope Control BQ_000472: I am looking for papers on remote replication that exclude local replication from their scope. Setting Anchor BQ_001550: Which papers evaluate image search techniques on the CIFAR-10 dataset? Method Capability BQ_002645: Which papers explore the use of reinforcement learning for collaborative content creation?
...
Agents retrieve topic-related robotics papers, but fail to locate papers containing explicit evidence for the superiority claim. The difficulty lies in evidence-level comparison, not topical relevance.
...
Agents treat the excluded concept as a positive retrieval keyword, instead of performing set-level filtering. The query should be solved as “topic A minus topic B,” but agents search for both A and B.
...
Agents recognize the dataset anchor but drift toward popular CIFAR-10 benchmark papers or adjacent vision tasks, rather than papers evaluating the target method under the specified dataset.
...
Agents capture method keywords, but drift away from the intended method–application combination. They retrieve RL-for-content or generation-related papers, but miss specific capability of collaborative content creation.
Agentic Search Success / Gold 0/9 Agentic Search Success / Gold 0/8 Agentic Search Success / Gold 0 / 30 Agentic Search Success / Gold 0/8
Figure 7: Representative common zero-recall cases across query types. Each case shows a query where all three agents retrieve no gold papers, together with the corresponding gold-answer size and the main failure cause.
evidence for the specific comparison. In the autonomous-robot case, they fail to find papers explicitly showing autonomous robots outperform human teleoperation, yielding 0 out of 9 gold papers. For scope-control queries, agents struggle with negative constraints: in the remote replication case, the target is essentially “remote replication minus local replication,” but agents treat the excluded concept as a positive keyword, yielding 0 out of 8 gold papers. For setting-anchor queries, agents recognize anchors such as CIFAR-10 or Pascal VOC, but drift toward popular benchmark papers instead of papers evaluating the target method under the specified setting. For method-capability queries, agents match the method keyword but fail to preserve compositional intent, such as reinforcement learning for collaborative content creation. Beyond query-specific failure patterns, Table 4 further shows that common zero-recall failures are not due to a lack of search effort, but to off-target exploration. PaSa and SPAR access hundreds of candidate papers on average across all failed query types, while PaperScout also explores a non-trivial candidate pool through iterative search. The core problem is therefore not whether agents search, but whether they can reach the correct region of the literature space. When the initial search direction is biased toward a semantically plausible but incorrect neighborhood, further retrieval or citation expansion tends to enlarge that neighborhood rather than recover the missing gold papers. This highlights a key challenge in paper
search: agents should not only search broadly, but also target the right evidence, constraints, and method–setting combinations.
5
Conclusion
We propose ScholarQuest, a taxonomy-guided benchmark for agentic academic paper search in open literature environments. Unlike existing benchmarks that rely on limited human queries or paper-derived query construction, ScholarQuest provides broad CS topic coverage, four representative research intents, scalable answer construction, and a shared ScholarBase backend for reproducible evaluation. Experiments show that agentic search methods outperform single-shot retrieval systems, with the best agentic method improving overall R@100 from 0.214 to 0.314 over the strongest non-agentic baseline, yielding a relative gain of 46.7%. Meanwhile, our analysis reveals limitations in search efficiency, query-type robustness, and constraint handling, with key failures caused by off-target exploration rather than insufficient effort. ScholarQuest offers a transparent and diagnostic testbed for identifying these limitations and tracking future progress in agentic paper search. Future work may develop agents with stronger intent preservation, constraint-aware filtering, and evidence-level reasoning throughout multi-round exploration. We invite the community to use ScholarQuest to study, compare, and build more reliable academic paper search agents.
6
Limitations
ScholarQuest is designed as a controlled benchmark for agentic paper search, but it still has several boundaries. First, ScholarQuest focuses on computer science topics and uses an arXiv-grounded literature environment, so it does not cover the full diversity of scholarly communication across disciplines, venues, and publication formats. Second, relevance judgments are based on paper titles, abstracts, and metadata rather than full-text evidence, which makes the benchmark scalable but may miss fine-grained claims that only appear in the body of a paper. Finally, although our answer construction pipeline combines multi-source retrieval, citation expansion, LLM-based relevance adjudication, and human audit, automatic construction may still miss some relevant papers in open literature environments. These limitations motivate future extensions toward broader literature sources, full-text-aware relevance assessment, and stronger answer-set validation.
References Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv preprint arXiv:2402.03216, 4(5). Mingyue Cheng, Daoyu Wang, Qi Liu, Shuo Yu, Xiaoyu Tao, Yuqian Wang, Chengzhong Chu, Yu Duan, Mingkang Long, and Enhong Chen. 2026. Mind2report: A cognitive deep research agent for expert-level commercial report synthesis. arXiv preprint arXiv:2601.04879. Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin, 70(4):213. Gordon V Cormack, Charles LA Clarke, and Stefan Buettcher. 2009. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, pages 758–759. Michael Gusenbauer and Neal R Haddaway. 2020. Which academic search systems are suitable for systematic reviews or meta-analyses? evaluating retrieval qualities of google scholar, pubmed, and 26 other resources. Research synthesis methods, 11(2):181–217. Donna K Harman. 1995. Overview of the third text retrieval conference (TREC-3), volume 225. DIANE Publishing.
Yichen He, Guanhua Huang, Peiyuan Feng, Yuan Lin, Yuchen Zhang, Hang Li, et al. 2025. Pasa: An llm agent for comprehensive academic paper search. arXiv preprint arXiv:2501.10120. Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. 2025. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516. Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. 2020. S2ORC: The semantic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4969–4983, Online. Association for Computational Linguistics. Gary Marchionini. 2006. Exploratory search: from finding to understanding. Communications of the ACM, 49(4):41–46. Tingyue Pan, Jie Ouyang, Mingyue Cheng, Qingchuan Li, Zirui Liu, Daoyu Wang, Mingfan Pan, Shuo Yu, and Qi Liu. 2026. Paperscout: An autonomous agent for academic paper search with process-aware sequence-level policy optimization. arXiv preprint arXiv:2601.10029. Karl Pearson. 1896. Vii. mathematical contributions to the theory of evolution.—iii. regression, heredity, and panmixia. Philosophical Transactions of the Royal Society of London. Series A, containing papers of a mathematical or physical character, (187):253–318. Hongjin Qian, Ziyi Xia, Ze Liu, Jianlyu Chen, Kun Luo, Minghao Qin, Chaofan Li, Lei Xiong, Junwei Lan, Sen Wang, et al. 2026. Deepxiv-sdk: An agentic data interface for scientific literature. arXiv preprint arXiv:2603.00084. Bernard Rous. 2012. Major update to acm’s computing classification system. Communications of the ACM, 55(11):12–12. Xiaofeng Shi, Yuduo Li, Qian Kou, Longbin Yu, Jinxin Xie, and Hua Zhou. 2025. Spar: Scholar paper retrieval with llm-based agents for enhanced academic search. arXiv preprint arXiv:2507.15245. Charles Spearman. 1961. The proof and measurement of association between two things. Fiona Timmins and Catherine McCabe. 2005. How to conduct an effective literature search. Nursing standard, 20(11):41–47. Daoyu Wang, Mingyue Cheng, Shuo Yu, Zirui Liu, Ze Guo, Xin Li, and Qi Liu. 2025. Paperarena: An evaluation benchmark for tool-augmented agentic reasoning on scientific literature. arXiv preprint arXiv:2510.10909. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
A
Implementation Details
We provide reproducibility details for ScholarQuest here. We organize the details into benchmark construction and dataset evaluation. A.1
Details of Benchmark Construction
Query Construction. We start from 1,682 ACM CCS topics and use Qwen3-Max to map each topic to one or more arXiv subject categories. Only topics assigned to arXiv CS categories in Table 8 are retained, resulting in 1,638 CS topic seeds. For each seed, Qwen3-Max generates exactly four queries, one for each intent type: method-oriented, setting-anchored, comparison-based, and scopecontrolled. After deduplication and quality filtering, we keep 1,111 high-quality queries. Answer Construction. For each query, we generate 10 rewritten search queries and retrieve the top 10 results for each rewrite from complementary retrieval sources, including Google Search, arXiv, and Semantic Scholar. Candidate papers are matched to arXiv records, normalized by arXiv ID, and deduplicated before scoring. We use a recalloriented prefilter to remove clear mismatches, followed by LLM-based relevance adjudication using the query, paper title, abstract and metadata. Papers judged as strict matches are treated as primary answer candidates. For citation expansion, each high-confidence seed paper retrieves up to 30 citing papers and all available references. We further perform second-hop expansion from high-confidence firsthop neighbors while pruning noisy branches. Newly discovered papers are normalized, deduplicated, and scored with the same relevance-filtering pipeline. The expansion stops when no eligible high-confidence seed remains, the answer set reaches 500 papers, or the number of scored unique papers reaches 10,000 for a query. Human Evaluation. To validate the final-stage LLM relevance scoring, we sample 150 querypaper pairs from each score group, resulting in 450 pairs in total, and ask three Ph.D. experts to re-score them based on the user query, paper title, abstract, and metadata. Human annotators follow the same three-level rubric as the LLM judges: 0 = mismatch, 1 = partial match, and 2 = strict match. For each pair, we aggregate the three expert annotations by majority vote and use this aggregated label as the human score in the following agreement anal-
Table 5: Default configuration for ScholarQuest benchmark construction. Configuration
Value
Raw ACM CCS topics 1,682 Retained CS topic seeds 1,638 Final queries 1,111 Queries per topic seed 4 Query rewrites per query 10 Search top-k per rewrite 10 Retrieval sources Google Search, arXiv, Semantic Scholar Query-level concurrency 4 Citation expansion depth Up to 2 hops Citation limit per seed paper 30 Reference limit per seed paper All available references First-stage filtering Recall-oriented prefilter Relevance scoring rubric 0–2 Answer relevance threshold 2.0 Maximum answer count 500 Maximum scored papers per query 10,000 Relevance judges Multiple LLM-based judges
ysis. As shown in Table 6, LLM scores align well with human annotations, with Pearson correlation of 0.867 (Pearson, 1896), Spearman correlation of 0.867 (Spearman, 1961), and quadratic weighted Cohen’s κ of 0.866 (Cohen, 1968). The score-2 group used for final answer selection reaches 86.0% strict-match precision (129/150) and 98.7% relaxed precision (148/150 with human score ≥ 1). In contrast, only 1/150 score-0 candidates and 12/150 score-1 candidates are judged as strict matches by humans, suggesting that residual false-negative risk is concentrated in borderline cases. Overall, these results indicate that the automated construction pipeline can reliably separate strictmatch answers from mismatches, with residual errors mainly concentrated in borderline cases. We further present four representative disagreement cases in Table 9, illustrating typical failure modes of LLM-based relevance scoring, such as overreliance on topical overlap and insufficient sensitivity to fine-grained query constraints. A.2
Details of Dataset Evaluation
We compare nine baselines from three groups: standard retrieval methods, external academic search systems, and agentic search methods. • Dense Retrieval. Dense retriever using BGEM3 title–abstract embeddings. • Hybrid Retrieval. Hybrid retriever combining BM25 and dense retrieval with RRF. • Google Search. A general web search baseline for paper discovery. • Google Scholar. A widely used academic search engine baseline.
LLM Score
Human Score
Total
Hit Rate
0
1
2
0 1 2
133 21 2
16 117 19
1 12 129
150 150 150
88.7% 78.0% 86.0%
Total
156
152
142
450
84.2%
Table 6: Confusion matrix between LLM-based relevance scores and human annotations. Hit Rate denotes the exact agreement rate within each LLM score group.
• Semantic Scholar. An academic search API with paper metadata and citation information. • DeepXiv. An external scientific literature search interface. • PaSa. LLM-based agent for multi-step academic paper search. • SPAR. Training-free workflow with retrieval, citation exploration, and reranking. • PaperScout. Autonomous paper search agent that adaptively decides when and how to search or expand. Retrieval Baselines. We evaluate two ScholarBase retrieval baselines: dense retrieval and RRFbased hybrid retrieval. Dense retrieval ranks papers with BGE-M3 title–abstract embeddings. Hybrid retrieval combines sparse and dense rankings through reciprocal rank fusion. Both methods take the original query as input and return ranked papers from ScholarBase. Academic Search Systems. We evaluate four external academic search systems: Google Search, Google Scholar, and search APIs of Semantic Scholar and DeepXiv. For each query, each system returns up to 300 papers. Returned papers are matched to ScholarQuest answers through arXiv identifiers, titles, and available metadata, ensuring that all methods are evaluated against the same arXiv-grounded answer sets. Agentic Search Methods. We evaluate three agentic paper search systems: PaSa, SPAR, and PaperScout. To ensure a fair comparison, we run their released inference code and open checkpoints without modifying their model weights or decision logic. All tool interfaces are provided by ScholarBase, including paper search, metadata lookup, and citation/reference expansion, so that different
agents interact with the same literature backend. During evaluation, all retrieved or expanded papers are normalized by arXiv ID before metric computation. We also record process statistics, including interaction rounds, search calls, expansion calls, observed candidates, and Recall@100 per 100 observed candidates.
B
Tool-call Evidence for Agentic Search
We further inspect the tool-use trajectory of BQ_002897, the method-capability case shown in Appendix D. The query asks for papers that explore deep reinforcement learning for video summarization. This case is useful because the gold answers form a compact but nontrivial neighborhood: some answers are directly reachable through targeted keyword search, while others are only recovered through citation/reference expansion. Table 10 lists every tool call that directly increases cumulative recall. PaperScout reaches all 12 gold answers after 20 tool calls. Its first four search calls recover 9/12 answers, and two later expansion calls recover the remaining three answers. In contrast, SPAR and PaSa both stop at 8/12. The missing answers are also systematic: both SPAR and PaSa miss 2109.01309 and 1807.09418; SPAR additionally misses 2007.14552 and 2002.03740, while PaSa additionally misses 2105.06441 and 2007.14552. Table 11 gives a gold-answer-level view of the same query. It shows that PaperScout’s advantage does not come from more direct hits alone: it uses early successful searches as anchors and then follows expansion paths from relevant seeds. For example, expanding 1801.00054 recovers 2002.03740 and 2105.06441, while expanding 1805.02792 recovers 1807.09418. This explains why only a small number of tool calls change recall: marginal gold discoveries are sparse, and many additional calls inspect already relevant neighborhoods or provide post-saturation confirmation rather than adding new gold answers.
C
Annotator Compensation
All human annotators are compensated for their work. The compensation rate is set to no less than ten times the local minimum hourly wage for each participant, reflecting both the specialized expertise required and the time-intensive nature of benchmark refinement and evaluation.
Table 7: Average Recall@100 of agentic paper search methods across gold answer-set size buckets and query types. PaperScout is strongest on method and setting queries, while scope-control remains difficult and recall decreases as answer sets grow. Answer Size
Method-oriented
Setting-anchored
Scope-controlled
Comparison-based
Overall
PaSa SPAR PaperScout PaSa SPAR PaperScout PaSa SPAR PaperScout PaSa SPAR PaperScout PaSa SPAR PaperScout 5–25 26–50 51–100 101–150 151–200
0.401 0.346 0.320 0.257 0.238
0.390 0.340 0.299 0.249 0.237
0.512 0.413 0.344 0.262 0.243
0.342 0.312 0.241 0.242 0.175
0.337 0.296 0.216 0.213 0.159
0.389 0.370 0.285 0.262 0.195
0.268 0.214 0.147 0.168 0.144
0.252 0.219 0.149 0.166 0.130
0.252 0.190 0.152 0.143 0.142
0.311 0.235 0.182 0.183 0.165
0.300 0.230 0.139 0.127 0.110
0.342 0.294 0.213 0.173 0.177
0.338 0.295 0.229 0.214 0.176
0.327 0.288 0.215 0.202 0.164
0.387 0.337 0.251 0.209 0.181
Table 8: Full names of arXiv computer science subject categories. These categories are used to map ACM CCS topics into the CS topic space for ScholarQuest construction. Category
Full Name
Category
Full Name
cs.AI cs.CC cs.CG cs.CR cs.CY cs.DC cs.DM cs.ET cs.GL cs.GT cs.IR cs.LG cs.MA cs.MS cs.NE cs.OH cs.PF cs.RO cs.SD cs.SI
Artificial Intelligence Computational Complexity Computational Geometry Cryptography and Security Computers and Society Distributed, Parallel, and Cluster Computing Discrete Mathematics Emerging Technologies General Literature Computer Science and Game Theory Information Retrieval Machine Learning Multiagent Systems Mathematical Software Neural and Evolutionary Computing Other Computer Science Performance Robotics Sound Social and Information Networks
cs.AR cs.CE cs.CL cs.CV cs.DB cs.DL cs.DS cs.FL cs.GR cs.HC cs.IT cs.LO cs.MM cs.NA cs.NI cs.OS cs.PL cs.SC cs.SE cs.SY
Hardware Architecture Computational Engineering, Finance, and Science Computation and Language Computer Vision and Pattern Recognition Databases Digital Libraries Data Structures and Algorithms Formal Languages and Automata Theory Graphics Human-Computer Interaction Information Theory Logic in Computer Science Multimedia Numerical Analysis Networking and Internet Architecture Operating Systems Programming Languages Symbolic Computation Software Engineering Systems and Control
Table 9: Representative disagreement cases between human judgments and LLM-generated labels. Human labels are obtained through manual review, while LLM labels correspond to the original automatic relevance labels. Scores 0, 1, and 2 denote mismatch, partial match, and strict match, respectively. Query and Label
Paper Title and Abstract Evidence
LLM Label Bias
Query: Which papers evaluate network domains in the context of data center networks? Human: 0 (mismatch) LLM: 1 (partial match)
Title: Efficient Coflow Scheduling in Hybrid-Switched Data Center Networks Evidence: The paper studies coflow scheduling in hybrid-switched data center networks, focusing on scheduling, communication time, hybrid links, and performance guarantees. However, it does not explicitly evaluate network domains.
Keyword over-crediting. The LLM is attracted by the strong topic match “data center networks,” but overlooks the missing constraint of “network domains,” leading to an overestimated relevance label.
Query: Which papers report that machine learning-based information extraction outperforms rule-based systems? Human: 1 (partial match) LLM: 0 (mismatch)
Title: PAM: Understanding Product Images in Cross Product Category Attribute Extraction Evidence: The paper studies product attribute extraction with a transformer-based model using product text, OCR tokens, and visual objects. It is related to machine learning-based information extraction, but does not directly report outperforming rule-based systems.
Over-strict claim matching. The LLM treats the comparison with rule-based systems as mandatory, while human reviewers still regard the paper as a partial match due to its clear tasklevel match.
Query: Which papers study image search using deep learning techniques? Human: 1 (partial match) LLM: 2 (strict match)
Title: Visual Discovery at Pinterest Evidence: The paper presents Pinterest’s visual discovery engine for visual search and recommendation products. It mentions object detection and improved engagement, but does not explicitly state “deep learning” or specify neural architectures.
Implicit-knowledge overuse. The LLM likely infers deep learning from visual search and object detection, upgrading the paper to a strict match without explicit textual evidence for the method constraint.
Query: Which papers claim that modern imaging techniques outperform traditional methods? Human: 2 (strict match) LLM: 1 (partial match)
Title: Twin-beam sub-shot-noise raster-scanning microscope Evidence: The paper proposes a quantum imaging microscope and reports improved precision over a shot-noise-limited classical version while preserving resolution and optical power.
Terminology mismatch. The LLM underestimates relevance because the paper expresses “traditional methods” as a domain-specific “classical version,” whereas human reviewers recognize the comparison as sufficient evidence.
Table 10: Evidence chain for BQ_002897. Each row is a tool call that directly increases cumulative recall. PaperScout first locates most gold answers through targeted search, then uses expansion to recover the remaining answers. Method
Call
Tool
Query or Expansion Seed
PaperScout PaperScout PaperScout
1 2 3
search search search
PaperScout
4
search
PaperScout PaperScout
6 20
expand expand
deep reinforcement learning AND video summarization 2001.05864 1/12 reinforcement learning AND video summarization 2007.14552, 2106.10528, 2109.01309 4/12 deep reinforcement learning AND action selection AND video 1801.00054, 1807.03089, 2005.09531, 8/12 summarization 2407.04258 9/12 reinforcement learning AND keyframe selection AND video 2505.23268 summarization 1801.00054 2002.03740, 2105.06441 11/12 1805.02792 1807.09418 12/12
Newly Found Gold Answers
SPAR
1
search
SPAR SPAR SPAR SPAR
2 3 7 14
expand expand expand search
SPAR
32
expand
PaSa
1
search
PaSa PaSa PaSa
8 10 50
expand expand expand
Literature review of application domains in deep reinforcement 2407.04258 learning for video summarization 2405.08890 1801.00054, 2001.05864 2101.06072 1807.03089, 2005.09531 2407.04258 2505.23268 State-of-the-art in deep reinforcement learning for video sum- 2105.06441 marization 2412.08357 2106.10528
Recall
1/12 3/12 5/12 6/12 7/12 8/12
Survey papers on video summarization using deep reinforce- 1801.00054, 1807.03089, 2005.09531, 5/12 ment learning 2407.04258, 2505.23268 2101.06072 2001.05864 6/12 2410.04449 2106.10528 7/12 2105.04066 2002.03740 8/12
Table 11: Gold-answer provenance for BQ_002897. S# and E# denote the order of search and expansion calls, respectively, through which a method discovers each answer. Gold Answer
Title
2505.23268
Unsupervised Transcript-assisted Video Summa- S4 search: keyframe selection query E7 expand 2407.04258 S1 search: survey query rization and Highlight Detection Reinforcement Learning for Unsupervised S3 search: action-selection query S1 search: literature- S1 search: survey query Video Summarization With Reward Generator review query Training Unsupervised multi-latent space reinforcement S2 search: reinforcement learning Missed Missed learning framework for video summarization in query ultrasound imaging Video Summarization Through Reinforcement S2 search: reinforcement learning E32 expand 2412.08357 E10 expand 2410.04449 Learning With a 3D Spatio-Temporal U-Net query DeepQAMVS: Query-Aware Hierarchical E6 expand 1801.00054 S14 search: state-of-the- Missed Pointer Networks for Multi-Video Summarizaart query tion Compare and Select: Video Summarization with S2 search: reinforcement learning Missed Missed Multi-Agent Reinforcement Learning query Ultrasound Video Summarization using Deep S3 search: action-selection query E3 expand 2101.06072 S1 search: survey query Reinforcement Learning Query-Biased Self-Attentive Network for E6 expand 1801.00054 Missed E50 expand 2105.04066 Query-Focused Video Summarization Weakly Supervised Video Summarization by Hi- S1 search: deep reinforcement learn- E2 expand 2405.08890 E8 expand 2101.06072 erarchical Reinforcement Learning ing query Video Storytelling: Textual Summaries for E20 expand 1805.02792 Missed Missed Events Video Summarisation by Classification with S3 search: action-selection query E3 expand 2101.06072 S1 search: survey query Deep Reinforcement Learning Deep Reinforcement Learning for Unsuper- S3 search: action-selection query E2 expand 2405.08890 S1 search: survey query vised Video Summarization with DiversityRepresentativeness Reward
2407.04258
2109.01309
2106.10528 2105.06441
2007.14552 2005.09531 2002.03740 2001.05864 1807.09418 1807.03089 1801.00054
PaperScout Route
SPAR Route
PaSa Route
D
Case Studies
We provide four representative ScholarQuest cases, one for each query intent. Each case shows the query metadata and its complete gold-answer set sorted in descending order by the first arXiv submission date. Case 1: Method-Capability Query (BQ_002897) Query. Which papers explore the use of deep reinforcement learning for video summarization? Domain. cs.CV; Topic seed. Video summarization; Constraint. technique = deep reinforcement learning; Gold answers. 11. Date
arXiv ID
Title
2025-05-29 2024-07-05 2021-09-03
2505.23268 2407.04258 2109.01309
2021-06-19 2021-05-13 2020-07-29 2020-05-19 2020-01-12 2018-07-25 2018-07-09 2017-12-29
2106.10528 2105.06441 2007.14552 2005.09531 2001.05864 1807.09418 1807.03089 1801.00054
Unsupervised Transcript-assisted Video Summarization and Highlight Detection Reinforcement Learning for Unsupervised Video Summarization with Reward Generator Training Unsupervised multi-latent space reinforcement learning framework for video summarization in ultrasound imaging Video Summarization through Reinforcement Learning with a 3D Spatio-Temporal U-Net DeepQAMVS: Query-Aware Hierarchical Pointer Networks for Multi-Video Summarization Compare and Select: Video Summarization with Multi-Agent Reinforcement Learning Ultrasound Video Summarization using Deep Reinforcement Learning Weakly Supervised Video Summarization by Hierarchical Reinforcement Learning Video Storytelling: Textual Summaries for Events Video Summarisation by Classification with Deep Reinforcement Learning Deep Reinforcement Learning for Unsupervised Video Summarization with DiversityRepresentativeness Reward
Case 2: Claim-Comparison Query (BQ_000815) Query. Which papers report that transfer learning outperforms random initialization in model performance? Domain. cs.LG; Topic seed. Transfer learning; Constraint. comparison = outperforms random initialization; Gold answers. 15. Date
arXiv ID
Title
2025-11-06 2024-10-10 2024-08-01 2022-06-20 2021-09-29
2511.11622 2410.08194 2408.00695 2206.09872 2109.14536
2021-06-09
2106.04995
2019-09-24 2019-08-26 2018-11-03 2018-10-15
1909.10955 1908.09883 1811.01137 1810.06282
2018-09-02 2018-02-05 2017-10-11
1809.00357 1802.01483 1710.05726
2017-06-02 2016-04-08
1706.00712 1604.02201
Small Vocabularies, Big Gains: Pretraining and Tokenization in Time Series Models Features are fate: a theory of transfer learning in high-dimensional regression Accelerating Full Waveform Inversion By Transfer Learning A Neural Network Based Method with Transfer Learning for Genetic Data Analysis PINNup: Robust neural network wavefield solutions using frequency upscaling and neuron splitting Crosslingual Embeddings are Essential in UNMT for Distant Languages: An English to IndoAryan Case Study Efficiently Reusing Old Models Across Languages via Transfer Learning Transfer learning for scalability of neural-network quantum states Transfer Learning in Multilingual Neural Machine Translation with Dynamic Vocabulary Feature Representation Analysis of Deep Convolutional Neural Network using Two-stage Feature Transfer -An Application for Diffuse Lung Disease ClassificationTrivial Transfer Learning for Low-Resource Neural Machine Translation Explicit Inductive Bias for Transfer Learning with Convolutional Networks Convolutional Neural Networks for Histopathology Image Classification: Training vs. Using Pre-Trained Networks Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning? Transfer Learning for Low-Resource Neural Machine Translation
Case 3: Scope-Control Query (BQ_001592) Query. I am looking for papers on software maintenance tools that exclude static analysis tools from their scope. Domain. cs.SE; Topic seed. Software maintenance tools; Constraint. exclude_filtering = static analysis tools; Gold answers. 12. Date
arXiv ID
Title
2025-06-30 2025-06-16
2506.24015 2506.13182
2024-10-18 2024-04-08 2023-04-24 2023-02-02 2022-02-22 2021-12-03 2021-04-16 2020-08-04 2020-02-10 2018-05-18
2410.14393 2404.05520 2304.12015 2302.01215 2202.10868 2112.02125 2104.08308 2008.01676 2002.03968 1805.07475
Hierarchical Knowledge Injection for Improving LLM-based Program Repair From Empirical Evaluation to Context-Aware Enhancement: Repairing Regression Errors with LLMs Debug Smarter, Not Harder: AI Agents for Error Resolution in Computational Notebooks The Fact Selection Problem in LLM-Based Program Repair ITER: Iterative Neural Repair for Multi-Location Patches Fixing Hardware Security Bugs with Large Language Models Neural Program Repair: Systems, Challenges and Solutions Examining Zero-Shot Vulnerability Repair with Large Language Models Neural Transfer Learning for Repairing Security Vulnerabilities in C Code Anchor: Locating Android Framework-specific Crashing Faults E-APR: Mapping the Effectiveness of Automated Program Repair Learning to Repair Software Vulnerabilities with Generative Adversarial Networks
Case 4: Setting-Anchored Query (BQ_001790) Query. Which papers study or evaluate privacy and security issues, policies, or privacy-preserving techniques in the healthcare domain? Domain. cs.CY; Topic seed. Privacy policies; Constraint. application_domain = healthcare; Gold answers. 19.
E
Date
arXiv ID
Title
2025-11-14
2511.11347
2025-09-18 2025-02-04 2024-10-18 2024-05-28 2023-12-15 2023-11-09 2023-07-24 2023-06-20 2023-06-10
2509.14581 2502.02749 2410.14607 2405.17971 2312.10214 2311.05404 2307.12542 2306.11557 2306.06448
2023-06-09 2023-02-08 2022-11-21 2022-05-06
2306.06033 2302.04208 2211.11434 2205.03168
2021-06-26 2021-06-11 2020-09-17 2020-08-13 2019-10-02
2106.13973 2106.06136 2009.08294 2008.05864 1910.00962
Privacy Challenges and Solutions in Retrieval-Augmented Generation-Enhanced LLMs for Healthcare Chatbots: A Review of Applications, Risks, and Future Directions Can I Trust This Chatbot? Assessing User Privacy in AI-Healthcare Chatbot Applications Unveiling Privacy and Security Gaps in Female Health Apps Evaluating Privacy Measures in Healthcare Apps Predominantly Used by Older Adults A Qualitative Analysis Framework for mHealth Privacy Practices Healthcare Policy Compliance: A Blockchain Smart Contract-Based Approach A Survey on Privacy of Health Data Lifecycle: A Taxonomy, Review, and Future Directions Client-Level Differential Privacy via Adaptive Intermediary in Federated Medical Imaging A Comparative Audit of Privacy Policies from Healthcare Organizations in USA, UK and India HIPAAChecker: The Comprehensive Solution for HIPAA Compliance in Android mHealth Apps SoK: Analysis of User-Centered Studies Focusing on Healthcare Privacy & Security Exploratory Analysis of Federated Learning Methods with Differential Privacy on MIMIC-III Privacy in Practice: Private COVID-19 Detection in X-Ray Images (Extended Version) Defending against Reconstruction Attacks through Differentially Private Federated Learning for Classification of Heterogeneous Chest X-Ray Data Benchmarking Differential Privacy and Federated Learning for BERT Models Security and Privacy for Healthcare Blockchains Robust Aggregation for Adaptive Privacy Preserving Federated Learning in Healthcare An Empirical Evaluation of GDPR Compliance Violations in Android mHealth Apps Privacy-preserving Federated Brain Tumour Segmentation
Prompt Templates
To make the LLM-assisted components of ScholarQuest transparent, we provide the main prompt templates used in query construction, answer discovery, agentic search, and relevance filtering. The templates serve the following roles: • Query Generation Prompt. Generates one taxonomy-guided paper-retrieval query for each intent type from a given topic seed. • Query Rewrite Prompt. Rewrites a benchmark query into multiple first-round search anchors while preserving the original retrieval intent. • Scorer Prompt. Asks a relevance judge to assess whether a candidate paper satisfies the user query based on its title and abstract. • Strict Batch Filter Prompt. Performs conservative batched relevance filtering for final answer curation, requiring explicit satisfaction of all core query constraints. Query Generation Prompt You generate paper retrieval queries from one topic seed. Return strict JSON only. The response must be one valid JSON object with this schema: { "seed_id": "PTS_000001", "topic_seed": "large language model agents", "domain": "cs.AI", "queries": [ { "category": "method_capability", "constraint_kind": "training_paradigm", "constraint_value": "reinforcement learning", "query": "Which papers study large language model agents trained with reinforcement learning?", "rationale": "reinforcement learning is a searchable technical constraint", "risk_flags": []
} ] } Generate exactly one query for each query category: - method_capability - setting_anchor - claim_comparison - scope_control Rules: - Produce exactly four query objects, one per category. - Each query must be a paper retrieval request. - Keep each query concise, natural, and one sentence. - Include one strong executable retrieval constraint per query. - Stay faithful to the topic seed, domain, and ACM id. - Avoid temporal wording such as recent, latest, since 2020, after 2020, or similar time constraints. - Do not ask for analysis, advice, or long-form synthesis. - Do not use fragment wording such as "Papers on ...". - Vary the surface form across the four queries when natural.
Query Rewrite Prompt You rewrite one paper-search query into first-round search anchors. Requirements: - Return exactly 10 English queries. - Keep every query close to the original intent. - Vary the semantic angle, retrieval scope, terminology, method, setting, task, evidence type, or comparison focus. - Make the 10 queries suitable as independent first-round search inputs for finding answer papers. - Keep each query concise and retrieval-oriented. - Avoid duplicates and near-duplicates. - Avoid drifting into a different research problem. - Do not mention citations, references, hops, or tool usage. - Do not output markdown or explanations. Return strict JSON only: {"queries": ["...", "..."]}
Scorer Prompt You are an elite researcher in the field of AI, conducting research on {user_query}. Evaluate whether the following paper fully satisfies the detailed requirements of the user query and provide your reasoning. Ensure that your decision and reasoning are consistent. Searched Paper: Title: {title} Abstract: {abstract} User Query: {user_query} Output format:+++++++++++++++++ Decision: True/False Reason: ... Decision:
Strict Batch Filter Prompt You are a strict relevance judge for academic paper filtering. Your task is to evaluate whether each candidate paper strictly matches the same user query. Judge based on the user query, paper title, and paper abstract. Scoring rules: - 2 = Strict match. The paper fully matches every core requirement in the query, including the target topic, method/task, setting, constraints, and any specified conditions. All important points in the query must be clearly supported by the title or abstract. - 1 = Partial match or missing/violated constraint. The paper is related to part of the query, but at least one core requirement is missing, unclear, too broad, or contradicted. If a query constraint is not explicitly matched, assign 1 rather than 2. - 0 = Mismatch. The paper does not match the query, or the main subject/intent differs from the query. Confidence: - high = The title and abstract provide enough evidence for the decision. - medium = The evidence is somewhat incomplete but the decision is still reasonably supported. - low = The decision is uncertain due to limited or ambiguous information. Be conservative. Assign 2 only when the match is explicit and complete. Do not infer missing constraints from vague similarity. Output valid JSON only: { "results": [ { "paper_index": 1, "arxiv_id": "...", "reason": "brief English reason based on the query, title, and abstract", "strict_score": 0, "confidence": "low" } ] } Return exactly one result for each input paper. Preserve the input paper_index and arxiv_id. Within each result, place "reason" before "strict_score", and place "confidence" last. Do not output markdown or extra text. USER QUERY: {query} PAPER CANDIDATES JSON: {papers_json}