IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. XX, NO. XX, XXXX 2026
PathAgentBench: Benchmarking Evidence-Seeking Vision-Language Models on Whole-Slide Pathology Images
arXiv:2607.19261v1 [cs.CV] 21 Jul 2026
Dankai Liao, Tianyi Zhang, Yufeng Wu, Xinyue Zhang, Qiaochu Xue, Zeyu Liu, Dachun Zhao, Linghan Cai, and Yueming Jin, Member, IEEE
Abstract— Whole-slide image (WSI) diagnosis requires identifying diagnostically relevant regions, examining them across magnifications, and integrating multi-scale evidence. However, most existing pathology benchmarks evaluate models on pre-cropped patches or pre-extracted slide features, leaving their ability to acquire evidence directly from gigapixel WSIs largely untested. We introduce PathAgentBench, a benchmark for evaluating evidence-seeking vision-language models (VLMs) across four complementary capabilities: image-to-text matching for evidence interpretation, text-to-image retrieval for evidence verification, diagnostic-region localization for evidence acquisition, and multi-scale reasoning for evidence integration. The benchmark is organized as a diagnostic tree that links nested regions across magnifications with scalespecific findings and path-level diagnoses. It contains 1,822 TCGA WSIs and 17,135 diagnostic paths annotated by ten board-certified pathologists. An additional private cohort of 190 breast cancer WSIs with detailed annotations is used to evaluate autonomous whole-slide exploration. We evaluate 20 general-purpose, medical, and pathologyspecialized models. Leading open-weight models achieve over 93% accuracy in multi-scale reasoning and over 50% accuracy in both cross-modal matching tasks. In contrast, diagnostic-region localization remains challenging: the best text-guided mean intersection-over-union is below 0.09, underperforming a simple center-based heuristic. During autonomous exploration, the unconditional hit rate This work was supported by the Ministry of Education Tier 1 grant, Singapore (24-1250-P0001), and the Ministry of Education Tier 2 grant, Singapore (T2EP20224-0028). This work was powered by the UnPuzzle & PuzzleCloud Platform (https://puzzlelogic.com/unpuzzle) and supported by PuzzleLogic Pte Ltd, Singapore. Dankai Liao and Tianyi Zhang contributed equally to this work. Corresponding Author: Dachun Zhao (e-mail: [email protected]), Linghan Cai (e-mail: [email protected]), and Yueming Jin (email: [email protected]) Qiaochu Xue, and Yueming Jin are with the Department of Biomedical Engineering, National University of Singapore, Singapore 117417, Singapore (e-mails: {dankai.liao, e1352520}@u.nus.edu, [email protected]) Dankai Liao, Tianyi Zhang and Yueming Jin are with the Department of Electrical and Computer Engineering, National University of Singapore, Singapore 117417 (e-mails: {e0556735, zhangtianyi}@u.nus.edu; [email protected]). Yufeng Wu, Xinyue Zhang, Zeyu Liu, and Linghan Cai are with PuzzleLogic Pte Ltd, Singapore 229594 (e-mails: {yufengwu, xinyuezhang, zeyuliu, linghancai}@puzzlelogic.com). Linghan Cai is also with School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, China (e-mail: [email protected]). Dachun Zhao is with the Department of Pathology, Peking Union Medical College Hospital, Beijing, China (e-mail: [email protected]).
decreases from 0.522 at low magnification to 0.185 at intermediate magnification and 0.020 at high magnification. These results reveal a pronounced gap between reasoning over curated evidence and acquiring that evidence directly from WSIs. PathAgentBench provides a unified framework for measuring and improving evidence-seeking pathology models. The benchmark is accessible at Github. Index Terms— Whole-slide image, pathology benchmark, vision-language model, agentic AI
I. I NTRODUCTION
P
ATHOLOGICAL diagnosis from whole-slide images (WSIs) is inherently an evidence-seeking process. A pathologist first surveys the slide at low magnification to identify suspicious tissue, then examines selected regions at progressively higher magnifications. Observations of tissue architecture, growth patterns, and cellular morphology are subsequently integrated into a diagnostic conclusion [1]. Reliable WSI analysis therefore requires more than interpreting a given image region. It also requires deciding where to look, which regions to examine further, and how to combine evidence acquired across locations and magnifications. The need to evaluate this complete workflow has become increasingly pressing as vision-language models (VLMs) are applied to computational pathology. General-purpose models can interpret histopathology patches, while pathologyspecialised VLMs support morphology recognition, visual question answering, and diagnostic dialogue [2]. More recently, pathology agents have combined VLMs with tool use, hierarchical zooming, and language-based planning to emulate the diagnostic workflow of pathologists [3]. Together, these developments mark a transition from passive interpretation of selected images toward active examination of gigapixel slides. Yet, the evaluation of these systems has not progressed at the same pace. Despite this shift toward active WSI examination, most existing pathology benchmarks assess models only after the relevant visual evidence has already been selected. Patch-level benchmarks provide a cropped field of view [4], while slidelevel benchmarks commonly rely on pre-extracted features, aggregated representations, or manually selected regions [5]. These settings measure whether a model can interpret supplied evidence, but not whether it can acquire that evidence from a
1
2
IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. XX, NO. XX, XXXX 2026
Current Evaluation
Real-World Diagnosis
PathAgentBench: Close the Gap Task 1: Evidence Interpretation Query: • Global architecture • Growth pattern • Cytoplasm
2 𝒎𝒎
2 𝒎𝒎
What is the pathological grade of this liver pathological image? Slide-level benchmarks: WSIBench, Slide-Bench, PathBench …
Evidence seeking
50 𝒖𝒎
Task 2: Evidence verification Query: • Lesion presence • Invasion boundary
Task 3: Evidence acquisition Query: Growth pattern?
200 𝒖𝒎
What is the tissue category of this H&E image patch? ROI-level benchmarks: Path-MMU, PathVQA, Quilt-VQA …
200 𝒖𝒎
Task 4: Evidence integration
200 𝒖𝒎
Evidence is provided, not acquired
Fig. 1: Motivation for PathAgentBench. Current pathology benchmarks predominantly evaluate interpretation of preselected evidence, whereas real-world WSI diagnosis requires active evidence seeking across locations and magnifications. PathAgentBench is designed to bridge this gap.
WSI. This distinction is critical because a model may reason accurately over a diagnostic region while remaining unable to locate that region independently. Consequently, strong performance on conventional visual question answering or slidelevel reasoning does not establish that a model can perform end-to-end diagnostic exploration. To make this missing capability explicit and measurable, we formulate WSI examination as a hierarchical, multimagnification diagnostic tree. At low magnification, the model should survey the global tissue context and localize suspicious regions; at intermediate magnification, it should assess tissue architecture and growth patterns; and at high magnification, it should inspect cellular and nuclear morphology. Each regionselection decision constrains which areas are examined at subsequent stages. In this formulation, the root corresponds to the complete slide, each node represents a region viewed at a particular magnification, and each branch encodes a possible diagnostic traversal. Diagnostic performance therefore depends on two coupled capabilities: selecting an informative path through the tree and reasoning over the evidence accumulated along that path. Existing benchmarks primarily assess the latter, while largely overlooking the former—the ability to navigate the slide and acquire diagnostically informative evidence. Building on this distinction between evidence acquisition and reasoning, we introduce PathAgentBench, a benchmark designed to evaluate both processes within a unified evidenceseeking framework. As illustrated in Figs. 1 and 2, PathAgentBench operationalizes four stages of WSI diagnosis as four corresponding benchmark tasks: evidence interpretation through image-to-text matching, evidence verification through text-to-image retrieval, evidence acquisition through diagnostic region localization, and evidence integration through multiscale diagnostic reasoning. Image-to-text matching assesses whether a model can interpret the morphology within a diagnostic region, whereas text-to-image retrieval tests whether it can verify a textual hypothesis against candidate image evidence. Diagnostic region localization evaluates evidence acquisition in two complementary settings: text-guided lo-
calization and autonomous whole-slide exploration. Finally, multi-scale diagnostic reasoning assesses whether findings collected across magnifications can be integrated into a coherent diagnosis. Evaluating these tasks jointly reveals where a model fails within the diagnostic workflow, rather than reducing its performance to a single aggregate score. To instantiate this framework at scale, PathAgentBench draws on 2,012 WSIs from The Cancer Genome Atlas (TCGA) [6] and an in-house breast cohort, spanning 16 organ types, together with 17,135 diagnostic paths selected by ten board-certified pathologists. Each path contains nested bounding boxes, findings at three magnifications, and a pathlevel diagnostic conclusion. Using this benchmark, we evaluate 20 models, including general-purpose, general-medical, and pathology-specialised models, across the four stages. This evaluation reveals a clear capability asymmetry. Leading models achieve over 93% accuracy when integrating pre-specified multi-scale findings, yet all models perform poorly when required to locate diagnostic evidence. Even the strongest closed-source models obtain a mean intersection-over-union below 0.09 in text-guided localization, while autonomous exploration suffers substantial coverage loss across successive magnifications. These findings identify evidence acquisition, rather than reasoning over preselected evidence, as the principal bottleneck in current pathology agents. The main contributions of this work are threefold. • We formulate pathology-agent evaluation as an evidenceseeking process on a multi-scale diagnostic tree, explicitly separating evidence acquisition from evidence reasoning. • We present PathAgentBench, which instantiates interpretation, verification, acquisition, and integration as four complementary evaluation tasks using pathologistauthored annotations. • Extensive evaluation and analyses on PathAgentBench show that current pathology vision language models can interpret and integrate supplied evidence but remain unreliable at spatially grounded, multi-step WSI navigation. II. R ELATED W ORK A. Vision-Language Model for Pathology General-purpose VLMs have substantially improved multimodal perception, instruction following, and structured reasoning [7]–[9]. Open-weight model families have further extended these capabilities across a wide range of parameter scales and computational budgets [10]–[17]. General-medical models, including MedGemma [18] and Lingshu [19], introduce medicaldomain supervision while retaining the broad instructionfollowing capabilities of general-purpose models. However, these advances are primarily demonstrated on images or image collections that can be directly presented to the model. Their applicability to WSIs remains constrained by the gigapixel scale and multi-resolution structure of slides. Pathology-specific VLMs can be broadly divided according to the spatial scale at which evidence is processed. Patch-level models learn representations or generate responses from individual fields of view. Representative systems include PathChat [2], PathAsst [20], Quilt-LLaVA [21], LLaVA-
DANKAI LIAO, et al.: PATHAGENTBENCH
3
Evidence accumulates along a root-to-leaf diagnostic path
Pathologist Annotation Workflow Select diagnostic regions / paths Write per-node findings
Can VLMs Diagnose Like Pathologists?
T1
T2
40x
Review
Reasoning
Cellular feature
Compose path -level diagnosis Supervisor review
T3
VLM
10x
T1 – Image-to-Text Matching (Interpretation) Match a diagnostic patch to its correct description among sameorgan, same-magnification distractors. ” Which diagnosis-related description matches this patch?”
T4
Growth pattern
T4 – Multi-Scale Diagnostic Reasoning (Integration) 2.5x
Integrate findings across 2.5×→10×→40× into a single diagnosis.
Global architecture
2.5×
A: Nuclear pleomorphic and hyperchromasia
10×
40×
Diagnosis & Grade
B: Grandular crowding C: Normal colonic mucosa
T3 – Diagnostic Region Localization (Acquisition)
WSI root D: Mucinous background
T2 – Text-to-Image Retrieval (Verification) Retrieve the correct H&E patch for a written diagnostic finding, among same-scale candidates. "The lamina propria shows loose edema and focal inflammatory cell infiltration; the muscular layer is not seen.."
A. Text-guided localization
B. Autonomous whole-slide exploration
"High-density malignant epithelioid cell population”
Agentic search and path construction
Dataset Overview 10 board-certified pathologists annotation 1,822 TCGA WSIs 16 organ types 71,735 four-way MCQs
17,135 diagnostic paths 190 private breast-cancer WSIs for autonomous whole-slide exploration 20 evaluated models
Fig. 2: Diagnostic-tree formulation of evidence seeking. A WSI is the root; at each magnification, selected regions become child nodes for further zoom-in while the remainder is pruned. Paths accumulate evidence through interpretation, verification, acquisition, and integration, instantiated by the four PathAgentBench tasks.
Med [22], and Patho-R1 [23]. These models have demonstrated promising capabilities in morphology recognition, visual question answering, and diagnostic dialogue. In contrast, slide-level models such as SlideChat [24], WSI-LLaVA [5], and PathReasoner [25] aggregate information from multiple regions to support WSI-level question answering or diagnostic reasoning. Patch-level models therefore emphasize local evidence interpretation, whereas slide-level models emphasize the integration of distributed evidence. Together, they cover the interpretation and integration endpoints of WSI analysis, but do not by themselves characterize the intervening process of region selection.
models on preselected image regions. Slide-level benchmarks such as WSI-VQA [37], SlideBench [24], WSI-Bench [5], and PathBench [38] extend evaluation to aggregated WSI representations but still determine region selection before inference. HealthAgentBench [39] introduces interactive WSI exploration, although pathology is represented by a single tumor-localization task. Across these settings, no existing benchmark spans evidence interpretation, verification, acquisition, and integration within a common diagnostic-tree framework. PathAgentBench provides this stage-wise evaluation using pathologist-annotated diagnostic trees. III. PATH AGENT B ENCH C ONSTRUCTION
B. Agentic Whole-Slide Image Analysis Agentic systems couple multimodal models with planning and external tools, enabling iterative visual inspection rather than single-step inference [26]–[28]. Recent pathology agents extend this paradigm to WSIs through hierarchical zooming, region extraction, and multi-step reasoning. CPathAgent [29] and PathAgent [3] model coarse-to-fine diagnostic trajectories; TissueLab [30] incorporates specialised tools; and MMNavAgent [31] and PathFound [32] emphasize multi-magnification navigation and active evidence seeking. Related systems further explore reasoning-enhanced diagnostic workflows [33], [34]. However, heterogeneous evaluation settings make existing pathology agents difficult to compare and their failure modes difficult to isolate. C. Benchmarks for Pathology Image Understanding Patch-level benchmarks such as PathVQA [35], QuiltVQA [21], PathMMU [4], and OmniPathoVQA [36] evaluate
In this section, we first describe the diagnostic-tree representation, data sources, annotation workflow, and then introduce the evaluation protocols used to construct PathAgentBench. A. Diagnostic Tree and Evidence-Seeking Decomposition A WSI s induces a diagnostic tree Ts = (Vs , Es ) over its image pyramid. Each node v = (s, ℓ, b) pairs a magnification ℓ ∈ L = {2.5×, 10×, 40×} with a box b ⊆ Ωs on the fullslide canvas; the root rs = (s, ⊥, Ωs ) is the thumbnail. Every non-root node v has a unique annotated parent pas (v) at the preceding view, with next defined by ⊥7→ 2.5× 7→ 10× 7→ 40× and c(b(v)) ∈ b(pas (v)) for box centre c(b). Thus, Es = (pas (v), v) : v ∈ Vs \ {rs } , (1) which remains a tree even when overlapping boxes admit more than one geometric container. A root-to-leaf path π = (rs , v1 , v2 , v3 ) with (ℓ(v1 ), ℓ(v2 ), ℓ(v3 )) = (2.5×, 10×, 40×)
4
IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. XX, NO. XX, XXXX 2026
Task 1: Evidence Interpretation
a) Tasks and evaluation structure
2.5× patches 10× patches
b) TCGA WSI distribution by organ (1,822 slides) 40× patches (5,256)
40× patches
Lung (579)
10× patches (3,796)
Task 2: Evidence Verification 2.5× patches
Mode A - 2.5× guided (queries) (16,975)
10× patches (3,788)
10× patches
40× patches (5,256)
40× patches
Triage stage (17,056)
Diagnose stage
T2
PathAgentBench
Liver (194)
T4 Pleura (1) Sarcoma (7) Unknown (8) Esophagus (8)
T3
Task 4: Evidence Integration Mode A - 2.5× guided (queries) Mode A - 10× guided (queries)
Stomach (223)
Mode A - 10× guided (queries) (16,975)
Diagnose stage (17,055)
Refine stage
Colorectum (228)
T1
Task 3: Evidence Acquisition Triage stage
Breast (465)
2.5× patches (1,232)
2.5× patches (1,240)
Refine stage (17,056)
Mode A - 40× guided (queries)
Mode A - 40× guided (queries) (16,933)
Uterus (29) Brain (20) Bladder (10) Extrahepatic bile duct (10) Head and neck (10)
Kidney (10) Thyroid (10) Lymphatic system (10)
Mode B - autonomous navigation (WSIs) (190)
Mode B - autonomous navigation (WSIs)
Fig. 3: Overview of PathAgentBench. (a) Four tasks and their evaluation axes; numbers denote task-specific evaluation budgets. (b) TCGA WSI distribution by organ (top 10 plus Other). The separate private breast cohort used for Mode B is not shown.
represents one diagnostic search route; each node carries a pathologist-authored finding d(v) and each path carries a pathologist-composed summary D(π). Existing patch- and slide-level VQA benchmarks measure reasoning given evidence that has already been supplied. To formalise the additional search component, let Π denote a single path sampled from a model’s diagnostic policy and let Y denote its diagnostic output. Marginalising over the latent path gives X pθ (y | s) = pθ (y | Π = π, s) pθ (Π = π | s). (2) π∈P(Ts )
The two terms represent evidence reasoning and acquisition, respectively; a set-valued Π gives the corresponding multibranch formulation. Conventional VQA fixes Π implicitly, whereas PathAgentBench evaluates both components. B. Data Sources PathAgentBench draws on two complementary sources. The first is a TCGA-based diagnostic tree dataset of 1,822 WSIs supporting tasks 1, 2, 3A, and 4. Of these, 1,814 carry one of 16 standardised organ labels (Lung, Breast, Colorectum, Stomach, Liver dominate); the remaining 8 are labelled “Unknown”. The per-organ distribution and task structure are shown in Fig. 3. The second is a private 190-WSI breast-cancer cohort used for Mode B (whole-slide diagnostic exploration); each slide carries a clinically-annotated tumor polygon (Nottingham grading) that identifies the diagnostically critical region but has no multi-scale path structure, which is what restricts its use to Mode B rather than Mode A. C. Pathologist Annotation Workflow All per-node bounding boxes, per-magnification findings, and path-level diagnoses in the TCGA dataset are pathologistauthored. Ten board-certified pathologists used an internal tool to select diagnostic regions and record the unique parent relation in Eq. (1). For each slide, one annotator selected the
critical root-to-leaf paths π, wrote a finding d(v) for every node along each path, and composed a path-level summary D(π). Each annotated slide was then reviewed by a second supervising pathologist; flagged paths, findings, or summaries were returned to the original annotator and the slide re-entered review until it passed. Only passed slides are released. The workflow yielded 17,135 paths and 51,363 per-magnification node descriptions over the 1,822 WSIs. D. Evaluation PathAgentBench formulates the WSI diagnostic workflow as four research questions about VLMs. Each task directly probes a component defined in §III-A: Tasks 1, 2, and 4 hold annotated evidence fixed and evaluate evidence reasoning, whereas Task 3 evaluates the evidence-acquisition policy. In Tasks 1, 2, and 4, all four options are presented jointly and the parsed choice is scored by exact-match accuracy. 1) Evidence Interpretation:
Q. Can a vision-language model read the pathological content of a single node of the diagnostic tree? As a node-level probe of the evidence-reasoning term in Eq. (2), image-to-text matching presents a node image with four candidate descriptions and asks the model to select the single pathologist-authored finding. All four candidates in a question come from the same organ–magnification cell of the diagnostic tree so that neither tissue type nor magnification leaks the answer. Distractors are mined by encoding each finding in the cell using the Sentence-Transformers model all-MiniLM-L6-v2 [40], [41] and greedily selecting the next candidate that minimises the maximum cosine similarity against already-chosen items, rejecting any set whose internal pairwise similarity exceeds 0.6. Each four-element set yields four questions by rotating the image slot; letter positions are shuffled under a fixed seed; magnification prefixes (“40×:”) are stripped to remove a trivial leakage cue. The released
DANKAI LIAO, et al.: PATHAGENTBENCH
task contains 10,284 four-option questions across the three magnifications. 2) Evidence Verification:
Q. Can a vision-language model verify whether a textual finding matches the morphology actually present in a tissue region? As the complementary node-level probe of the same evidence-reasoning term, text-to-image retrieval presents a pathologist-authored finding with four candidate images and asks the model to select the matching region. The four-element pool is mined with the same group-and-greedy procedure as T1 so the distractor difficulty is comparable across the two modality directions and any T1↔T2 gap reflects a directional alignment property of the model. The textual stem is LMrewritten into “Which of the following images shows. . . ” phrasing for natural medical wording. The released task contains 10,284 four-option questions. 3) Evidence Acquisition:
Q. Can a vision-language agent actively acquire diagnostically relevant evidence by navigating a WSI across scales under different levels of guidance? This task directly evaluates the evidence-acquisition policy in Eq. (2). Accordingly, we design two complementary modes that differ in the guidance available to the agent. Mode A supplies an explicit textual target and asks the agent to localize the corresponding evidence, whereas Mode B provides no target-specific guidance and asks the agent to determine what is worth examining while exploring an unfamiliar WSI. Mode A: text-guided localization.: Given a slide and a target description, the agent returns a predicted box at the requested magnification. We report mean intersection-overunion (mIoU) with the pathologist-annotated box and the fraction of queries with IoU at least 0.3. A LangGraph state machine exposes three tools—get image info(), extract roi(x,y,w,h,reason) in level-0 pixels, and finish and report(conclusion, bbox, confidence)—under an exploration budget of (10, 7, 5) ROIs at (2.5×, 10×, 40×). At 10× and 40×, the search space is constrained to the annotated parent box to follow the hierarchy in Eq. (1). The predicted box is constrained to the ground-truth dimensions so IoU isolates position rather than extent. Each trajectory issues many tool calls against a gigapixel pyramid, so we evaluate on a 50-slide stratified subset (three queries per slide, identical across models). Mode A is reported for the three VLMs and PathoR1-7B—other open-weight checkpoints did not emit valid bounding-box tool calls—against five non-VLM baselines and an exploration-budget sweep (Table III). Mode B: whole-slide diagnostic exploration.: The agent scores all 2.5× tiles with the tumor-suspicion prompt, then recursively scores the children of retained tiles; at each level, the top-K tiles classified “Yes” are retained per parent. A tile is tumor-positive when more than 5% of its area overlaps the pathologist-annotated tumor region. Conditional hit rate
5
measures recall among positive tiles reachable from branches retained at the preceding level, whereas unconditional hit rate measures recall among all positive tiles at that magnification, so unreached branches count as misses. Both rates microaverage tile counts across the cohort. The protocol uses a retention schedule of (6, 3, 2) at (2.5×, 10×, 40×) on the 190WSI breast cohort. Mode B is reported for the 15 successfully evaluated models against non-VLM baselines and a branching (K) sweep with a ground-truth oracle (Table IV). 4) Evidence Integration:
Q. Can a vision-language model integrate the permagnification findings along a diagnostic route into a single diagnostic conclusion? As a path-level probe of the evidence-reasoning term in Eq. (2), multi-scale diagnostic reasoning presents the organ, the three pathologist-authored findings along a diagnostic route, and a four-option Diagnose, Triage, or Refine question. For Diagnose, the correct option is the pathologistcomposed route summary and distractors come from other paths of the same organ; Triage maps the summary to a broader pathological category, whereas Refine targets a finegrained detail such as subtype, invasion, grade, or marker. The text-only input isolates integration from upstream perception and acquisition. Distractors are selected using the same all-MiniLM-L6-v2-based greedy low-similarity procedure as in T1/T2, with a cosine-similarity threshold of 0.6. Triage and Refine stems are LM-generated with retry-andvalidate guards. The released task contains 51,167 four-option questions. IV. E XPERIMENTAL R ESULTS AND A NALYSES A. Experimental Setup We evaluate 20 model configurations stratified by training domain (general-purpose / pathology-specialised / generalmedical) and scale; MoE rows report total / active parameters, and undisclosed closed-source sizes use public analyst estimates). The general-purpose group comprises two closed-source VLMs: GPT-5.2 [7] and Gemini-3-Flash [8],together with the open-weight multimodal models Qwen-3.5Flash [11], Kimi-K2.5 [13], InternVL2.5-8B and InternVL2.526B [10], LLaMA-3.2-11B-Vision [42], Qwen2.5-VL-7B [43], Phi-3-Vision-4.2B [15], LLaVA-1.5-7B [16], and LLaMA3LLaVA-Next-8B [17], as well as the text-only DeepSeek-V4Flash [44]. The pathology-specialised group consists of PathoR1-7B [23], Quilt-LLaVA-7B [21], and LLaVA-Med-1.57B [22]. The general-medical group comprises MedGemma1.5-4B, MedGemma-4B, and MedGemma-27B [18], together with Lingshu-7B and Lingshu-32B [19]. The T4 sub-columns decompose the clinical reasoning chain: Triage (broad pathology class), Diagnose (integrated conclusion), and Refine (subtype, grade, marker, or morphology). Table I reports results on the full benchmarks for Tasks 1, 2, and 4 (10,284, 10,284, and 51,167 questions, respectively), a 50-slide subset for Task 3 Mode A (three queries per slide; 150 queries per model), and the full 190-slide cohort for Mode B.
6
IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. XX, NO. XX, XXXX 2026
TABLE I: Per-model performance across the four PathAgentBench tasks. T1/T2/T4 report accuracy (%); T3 reports mean IoU (Mode A) or conditional and unconditional hit rate (Mode B). Model (scale)
T1: image-to-text Acc. (%)
T2: text-to-image Acc. (%)
T3-A mean IoU
T3-B cond. hit (%)
T3-B uncond. hit (%)
T4: diagnostic reasoning Acc. (%)
2.5/10/40 | All
2.5/10/40 | All
2.5/10/40
2.5/10/40
2.5/10/40
Tri/Diag/Ref | All
Trivial baselines Random Choice Frequent Choice
25.00/25.00/25.00 | 25.00 27.92/26.03/26.22 | 26.35
25.00/25.00/25.00 | 25.00 27.42/26.16/25.36 | 25.78
—/—/— —/—/—
—/—/— —/—/—
—/—/— —/—/—
25.00/25.00/25.00 | 25.00 25.00/25.00/25.00 | 25.00
Human reference Expert Performance
92.5/93.8/94.5 | 93.6
91.0/92.4/93.6 | 92.3
—/—/—
—/—/—
—/—/—
98.0/95.5/96.2 | 96.6
Non-VLM navigation reference baselines Random sampling Grid centroid Tissue density Parent-box center CONCH retrieval
—/—/— | — —/—/— | — —/—/— | — —/—/— | — —/—/— | —
—/—/— | — —/—/— | — —/—/— | — —/—/— | — —/—/— | —
0.038/0.073/0.031 0.049/0.141/0.163 0.072/0.032/0.021 0.061/0.247/0.283 0.038/0.105/0.060
33.47/16.62/7.40 —/—/— 28.60/17.87/6.42 —/—/— 30.08/18.30/9.04
33.47/9.16/0.87 —/—/— 28.60/9.68/1.04 —/—/— 30.08/7.32/0.67
—/—/— | — —/—/— | — —/—/— | — —/—/— | — —/—/— | —
General-purpose VLMs (closed-source) GPT-5.2 (2–5 T‡ ) Gemini-3-Flash (∼1.2 T‡ )
41.56/50.08/56.28 | 52.23 57.31/63.94/64.59 | 63.48
50.16/58.05/65.60 | 60.96 62.82/66.53/69.77 | 67.74
0.086/0.056/0.066 0.038/0.089/0.068
46.81/22.50/7.94 52.17/24.08/8.43
46.81/16.42/1.83 52.17/18.48/2.02
96.99/79.37/95.43 | 89.20 97.64/82.62/95.95 | 91.18
General-purpose VLMs (open-weight) Qwen-3.5-Flash (35 B / 3 B-A) Kimi-K2.5 (1 T / 32 B-A) DeepSeek-V4-Flash† (284 B / 13 B-A) InternVL2.5-26B (26 B) LLaMA-3.2-11B-Vision (11 B) InternVL2.5-8B (8 B) Qwen2.5-VL-7B (7 B) LLaMA3-LLaVA-Next-8B (8 B) LLaVA-1.5-7B (7 B) Phi-3-Vision-4.2B (4.2 B)
45.05/52.77/57.69 | 54.36 51.14/56.74/60.67 | 58.08 —/—/— | — 32.87/36.67/40.62 | 38.23 26.30/26.82/27.80 | 27.26 33.69/39.01/41.97 | 39.89 28.57/32.35/34.93 | 33.22 24.19/26.55/26.20 | 26.09 28.00/26.26/26.05 | 26.36 24.43/25.55/25.61 | 25.45
50.00/57.26/64.44 | 60.05 —/—/— | — —/—/— | — 31.21/33.63/38.64 | 35.90 —/—/— | — 29.19/29.91/33.49 | 31.65 31.69/34.77/39.86 | 37.00 27.10/26.48/24.03 | 25.30 26.85/26.06/25.06 | 25.64 —/—/— | —
0.051/0.078/0.083 —/—/— —/—/— —/—/— —/—/— —/—/— —/—/— —/—/— —/—/— —/—/—
42.32/20.33/7.77 —/—/— —/—/— 46.92/23.32/8.52 —/—/— 34.07/16.98/6.84 25.41/17.70/7.94 7.03/6.41/4.84 37.30/18.09/7.82 11.11/9.01/7.74
42.32/14.57/1.55 —/—/— —/—/— 46.92/15.28/1.58 —/—/— 34.07/10.64/1.06 25.41/7.56/0.81 7.03/0.62/0.04 37.30/10.77/1.03 11.11/1.79/0.16
97.10/83.31/95.43 | 90.98 —/—/— | — 96.34/83.64/93.76 | 90.49 97.78/90.01/93.73 | 93.16 96.92/80.41/91.87 | 88.86 97.63/90.89/93.43 | 93.44 95.25/85.29/86.12 | 88.26 97.34/76.43/89.46 | 85.20 42.21/30.92/47.28 | 40.59 93.72/90.45/76.52 | 86.24
Pathology-specialised VLMs Patho-R1-7B (7 B) Quilt-LLaVA-7B (7 B) LLaVA-Med-1.5-7B (7 B)
20.29/19.10/20.97 | 20.20 6.25/6.06/7.91 | 7.03 20.37/20.44/20.55 | 20.49
21.61/22.10/21.80 | 21.89 6.53/7.55/6.22 | 6.75 18.79/18.66/18.61 | 18.65
0.000/0.000/0.000 —/—/— —/—/—
12.62/8.62/6.11 37.83/18.15/8.36 —/—/—
12.62/2.23/0.22 37.83/10.38/1.01 —/—/—
81.82/60.85/73.23 | 70.08 69.02/35.08/63.88 | 56.02 72.16/57.27/35.67 | 53.97
General-medical VLMs Lingshu-32B (32 B) MedGemma-27B (27 B) Lingshu-7B (7 B) MedGemma-4B (4 B) MedGemma-1.5-4B (4 B)
40.67/46.02/49.89 | 47.36 27.76/27.08/28.54 | 27.91 44.24/51.32/56.72 | 53.23 29.14/31.09/32.61 | 31.63 30.52/34.51/39.08 | 36.37
36.94/42.05/47.51 | 44.22 22.42/22.62/24.89 | 23.76 47.10/51.45/58.43 | 54.49 29.03/32.89/34.02 | 33.00 24.44/30.62/32.61 | 30.89
—/—/— —/—/— —/—/— —/—/— —/—/—
—/—/— 49.16/24.05/8.70 30.47/17.06/6.99 48.90/23.11/8.50 46.92/22.43/8.27
—/—/— 49.16/16.52/1.80 30.47/10.01/1.04 48.90/17.01/1.80 46.92/15.11/1.58
97.55/83.34/94.33 | 90.65 97.34/85.88/93.98 | 91.83 97.64/90.62/93.36 | 93.41 96.62/83.63/89.97 | 89.06 89.61/73.27/76.09 | 78.13
Bold and underlining mark the best and second-best model per eligible column; “—” denotes not evaluated. † Text-only LLM. ‡ Analyst estimate. Mode A uses an exploration budget of (10, 7, 5) ROIs at (2.5×, 10×, 40×). Mode B retains (6, 3, 2) branches per parent across the three magnifications.
Table II uses 1,071 questions each for the T1/T2 distractorencoder ablations and 1,500 questions for the T4 magnification ablation. Table III reuses the 50-slide Mode A subset (150 queries per model per budget), while Table IV uses 45 slides. B. Patch-Level Recognition (Tasks 1 and 2) Tasks 1 and 2 share samples and distractor pools (Table I). Gemini-3-Flash leads both (T1 63.5%, T2 67.7%), and the strongest five T1 models span 52.2–63.5%. The pathologyspecialised checkpoints fall below the 25% random baseline: Patho-R1 and LLaVA-Med reach only 19–22%, while QuiltLLaVA falls to about 7%. The distractor-encoder columns in Table II show that encoder choice changes absolute difficulty in a task-dependent direction but preserves model ordering (Spearman ρ = 0.95–0.99), indicating that the main ranking is not a MiniLM-specific shortcut.
controlled K-sweep subset, Gemini and Qwen reach 54.8% and 51.8% unconditional hit rate at 2.5×, versus 30.7% for random sampling. Current VLMs are thus useful as evidence scorers, not autonomous planners. Constrained scoring still suffers severe end-to-end attrition: Gemini’s directly measured unconditional hit rate falls 52.2% → 18.5% → 2.02% from 2.5× to 40×, and PathoR1-7B remains below random under the default schedule. The controlled sweep now exposes the same mid-scale loss directly: under (6, 3, 2), Gemini falls 54.82% → 19.89% → 2.24% and Qwen falls 51.75% → 18.46% → 2.06%. Branching budget is a real but incomplete remedy (Table IV): expanding Qwen from (6, 3, 2) to (10, 5, 3) raises 40× coverage from 2.06% to 6.25%, while the ground-truth oracle rises from 3.01% to only 8.26%. The bottleneck therefore couples early pruning with scorer quality; retaining more tiles buys recall with compute but still does not yield the precise localization a clinical workflow needs.
C. Diagnostic Region Localization (Task 3) Both Task 3 regimes collapse, but the two interfaces invert the agent’s role (Table I). In Mode A the agent has full agency to choose where to look, and there the closed-source VLMs never exceed IoU 0.09: a parameter-free parent-box-center heuristic (IoU 0.25–0.28 at 10×/40×) beats every VLM by 3–4×, the pathology-specialised checkpoints fail to emit valid bounding-box calls (IoU ≡ 0), and enlarging the exploration budget degrades IoU further—the sweep is an inverted U (Table III). In Mode B the model is demoted to a pertile scorer inside a fixed hierarchical top-K pipeline; on the
D. Multi-Scale Diagnostic Reasoning (Task 4) T4 is easy once observations are supplied (Table I): twelve models exceed 87%, with the top three open-weight ≤26 B checkpoints (InternVL2.5-8B 93.4%, Lingshu-7B 93.4%, InternVL2.5-26B 93.2%) beating every closed-source VLM. Among the strongest models, most remaining errors occur on Diagnose: broad triage is nearly saturated, but deriving the integrated conclusion is less reliable. Multi-scale evidence generally helps. For 15 of the 16 models in the magnification ablation, the full three-scale input exceeds the best single
DANKAI LIAO, et al.: PATHAGENTBENCH
7
TABLE II: Benchmark-construction ablations for distractor encoder (T1/T2) and magnification subset (T4). Bold marks the easiest encoder per model/task or the best single magnification. Model
Encoder–T1
Encoder–T2
Single magnification
Magnification pair
All three
MiniLM/BiomedCLIP/CONCH
MiniLM/BiomedCLIP/CONCH
2.5/10/40
2.5+10/10+40/2.5+40
2.5+10+40
General-purpose VLMs GPT-5.2 Gemini-3-Flash Qwen-3.5-Flash InternVL2.5-26B InternVL2.5-8B Qwen2.5-VL-7B LLaMA3-LLaVA-Next-8B LLaVA-1.5-7B
61.2/66.8/59.1 73.3/70.8/70.5 65.6/60.3/63.3 56.0/45.3/45.9 58.3/44.9/44.4 52.0/42.2/41.6 41.0/31.1/35.9 31.6/26.8/28.8
63.4/64.1/62.6 68.0/70.5/71.6 62.5/65.5/64.9 37.2/38.3/36.6 33.8/37.2/32.5 28.9/27.9/28.7 24.8/26.1/24.6 25.7/25.7/25.7
82.6/85.5/83.7 83.7/87.7/85.3 85.7/88.3/85.9 86.3/90.9/88.2 88.1/90.9/89.6 78.1/83.7/79.2 77.5/82.7/81.5 37.7/37.2/38.1
88.5/88.5/88.7 89.8/89.5/89.9 90.3/90.6/91.0 92.2/92.8/92.6 92.1/92.3/92.3 85.3/86.1/84.3 86.1/86.9/85.3 43.0/43.3/44.3
90.5 91.6 92.7 94.5 93.2 87.3 88.5 43.3
Pathology-specialised VLMs Patho-R1-7B Quilt-LLaVA-7B LLaVA-Med-1.5-7B
43.7/31.6/38.2 10.5/9.5/12.4 27.7/27.6/23.2
24.6/25.9/24.8 8.3/8.9/8.7 26.1/26.1/26.0
38.5/45.0/49.5 60.3/60.9/59.8 59.2/66.9/64.0
44.1/47.8/44.5 67.5/65.1/65.6 62.5/66.9/63.7
51.8 67.1 65.0
General-medical VLMs Lingshu-7B MedGemma-27B MedGemma-4B MedGemma-1.5-4B
69.8/60.7/62.6 47.4/38.7/42.1 55.9/39.6/40.4 53.3/41.8/46.7
58.2/59.5/60.2 29.1/30.5/30.6 35.4/36.1/35.3 36.8/38.8/38.8
87.7/91.5/89.3 84.0/89.5/84.7 82.9/86.7/86.3 66.4/71.9/70.3
92.1/92.9/92.1 90.9/90.5/90.9 88.6/89.0/87.2 73.3/73.0/71.3
93.8 93.1 90.2 75.4
TABLE III: Task-3 Mode A navigation ablation. Explorationbudget sweep: mean IoU per magnification under a Tight, the default Base, and a Loose ROI budget. Budget
2.5×
10×
40×
GPT-5.2 Tight (5,3,2) Base (10,7,5) Loose (20,15,10)
0.001 0.086 0.006
0.004 0.056 0.033
0.000 0.066 0.003
Gemini-3-Flash Tight (5,3,2) Base (10,7,5) Loose (20,15,10)
0.007 0.038 0.043
0.006 0.089 0.023
0.000 0.068 0.012
Qwen-3.5-Flash Tight (5,3,2) Base (10,7,5) Loose (20,15,10)
0.005 0.051 0.017
0.022 0.078 0.013
0.010 0.083 0.011
magnification by 2.3–6.3 pt. The 10× view is the strongest single scale for 14 models, whereas Patho-R1 and LLaVA1.5 perform best at 40×. LLaVA-Med is the sole exception to the multi-scale gain, scoring 66.9% at 10× versus 65.0% with all three scales. Pathology-specialised models span 54–70%, while LLaVA-1.5 remains a 40.6% prompt-format outlier.
E. Closed-Source Inference Efficiency We further evaluate the inference efficiency of GPT-5.2 and Gemini-3-Flash on fixed subsets comprising 20 shared questions per task for Tasks 1, 2, and 4, the same five slides at each magnification for Task 3 Mode A, and five shared slides for Mode B. Task 3 accounted for 96.5% and 92.3% of the total cost for GPT-5.2 and Gemini-3-Flash, respectively: MCQ inference cost $0.00025–$0.00284 per question, compared with $0.0043–$0.0297 per slide/magnification for Mode A and $0.195–$0.224 per slide for Mode B. Mode B further required 20.7–46.8 min per slide, showing that iterative evidence acquisition dominates the operational burden.
F. Per-Organ Robustness Before distilling the cross-task findings, we check that the task-axis story is not confounded by an organ-axis effect. Fig. 4 averages each task metric across the task-specific evaluated methods/models and breaks the result down by organ. For T3A, the evaluated set comprises three VLMs and five nonVLM baselines: center, CONCH, grid, random, and tissueweighted sampling. The five high-coverage organs that carry the bulk of the cohort (Lung, Breast, Colorectum, Stomach, Liver) remain tightly grouped within each MCQ task: their per-organ means span 3.4 pt on T1, 3.1 pt on T2, and 4.9 pt on T4. Colorectum is the one clear head-organ outlier on T3A (IoU 0.17 vs. the 0.08 panel mean), plausibly because its glandular crypt architecture is more spatially distinctive than diffuse parenchyma. Larger swings occur mainly in the rare tail, where the per-organ evaluation pool drops to tens or single digits (e.g. Sarcoma n=4 on T1/T2 and Pleura n=2 on T4); these low-count organs also pull the unweighted T4 panel mean below the high-coverage cluster. The bottlenecks isolated by the per-task results are therefore properties of the task, interpretation, acquisition, integration, rather than of which high-coverage organ a slide comes from, and the cross-task findings below carry across the bulk of the cohort. G. Cross-Task Findings We distil the per-task results into four findings; Fig. 5 summarises the per-model profiles. Finding 1 – Once multi-scale evidence is supplied, integration is no longer the bottleneck. The open-weight leaders and narrow residual spread among the leading models on T4 contrast with the much weaker recognition and navigation profiles, indicating that structured findings remove much of the capacity advantage seen when models must perceive or acquire evidence and that supplied-evidence leaderboards can overstate end-to-end readiness. Finding 2 – Evidence interpretation remains biased toward high magnification, while pathology-specialised checkpoints fail differently. Stronger models benefit from cellular detail,
8
IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. XX, NO. XX, XXXX 2026
TABLE IV: Task-3 Mode B branching-schedule ablation. Directly measured unconditional hit rate (%) is reported at 2.5×, 10×, and 40×; bold marks the default (K2.5 , K10 , K40 ) = (6, 3, 2). Random (K2.5 , K10 , K40 ) (1,1,1) (3,3,3) (6,3,2) (6,5,3) (10,5,3)
0.5
Patho-R1
Qwen
Gemini
Oracle
2.5
10
40
2.5
10
40
2.5
10
40
2.5
10
40
2.5
10
40
6.58 18.42 30.70 33.77 47.37
0.72 4.74 9.15 12.62 18.18
0.02 0.64 0.87 1.84 2.52
4.72 10.37 7.92 12.40 9.84
0.54 1.81 2.92 3.76 3.19
0.03 0.23 0.30 0.54 0.43
11.40 28.95 51.75 51.75 69.74
1.43 10.08 18.46 27.93 37.63
0.08 1.65 2.06 4.62 6.25
11.40 31.58 54.82 70.37 —
1.38 11.13 19.89 38.99 —
0.08 1.82 2.24 5.92 —
17.98 42.98 67.11 67.54 84.21
2.20 15.87 24.68 37.47 45.84
0.14 2.87 3.01 6.78 8.26
0.40
0.40
Task 1 - Image-to-Text Matching
0.5
Task 2 - Text-to-Image Retrieval
panel mean = 0.35
0.4
0.38
0.36
0.38
0.35
0.4
0.37
0.35
0.33
0.35
0.32
0.32
0.3
0.2
0.1
0.38
0.38
0.38
0.36
0.38
0.34
0.33
0.3
0.2
0.0 n Lu
g
ea Br
st
r ve
h
m
tu
c re lo
Co
ac
s
r
n
ai
u er
Li
m
o St
Br
Ut
ad
Bl
de
r xt
p.
e ah
E
BD
ad He
&
ck Ne
ey
dn
Ki
ph
h st ng um ac ea Lu ct Br om re St lo o C
s a d oi gu om yr ha rc op Sa Es
ic
at
Th
m Ly
Task 3A - Text-Guided Localization
1.0
r ve
Li
Ut
us
n ai
er
Br
r a d ic ck us ey BD oi de at om ag dn yr Ne p. ad ph rc Ki & ph Th Bl he Sa d ym so ra L a t E He Ex
Task 4 - Multi-Scale Diagnostic Reasoning
panel mean = 0.08 0.21
0.82
0.8 0.16
0.15
0.11
0.10
0.10 0.08
0.08
0.07
0.06
0.05
0.05
0.05
0.07
0.06
0.84
0.85
0.86
0.84
0.83
0.80
panel mean = 0.73
0.87
0.84
0.81 0.73 0.67
0.64
0.65
0.67
0.6 0.47
0.4
0.05
0.21
0.2
0.04
0.03
Balanced accuracy
0.20
Mean IoU
0.37
0.37 0.33
0.1
0.0
0.25
0.37 0.35
0.33
Accuracy
Accuracy
0.39
0.38
0.36
0.35
panel mean = 0.37 0.40
0.02
0.00
0.0 n Lu
g
h st um ac ea ct Br om re St lo o C
r ve
Li
s
u er
Ut
er
n
ai
Br
D
dd
a Bl
ra xt
E
.B
p he
ad He
&
ck Ne
ey
dn
Ki
m Ly
ph
ic
at
a n us ow om ag rc kn Sa Un
d
i ro
y Th
Es
h op
t h ng as um ac Lu Bre ct om re St lo o C
r ve
Li
Ut
us
er
r s a d n D ic ra ck ey w oi de gu at .B om leu dn yr Ne no ad ha ph rc P ep Ki & Th Bl nk m op Sa d ah U y s r L a t E He Ex
n ai
Br
Fig. 4: Per-organ performance across the four PathAgentBench tasks, averaged over the task-specific evaluated models and baselines. Mode B is omitted because its breast-only cohort has no organ axis. Saturated and faded bars denote high-coverage and rare-tail organs, respectively; dashed lines mark unweighted organ means.
whereas the two reported pathology-specialised models remain flat and below random across scales; their stable ordering across distractor encoders is consistent with weak use of scalespecific morphology and answer-format misalignment rather than a single construction shortcut, separating the availability of pathology knowledge from its scale-appropriate use. Finding 3 – Evidence acquisition remains the primary bottleneck in hierarchical search. Mode A shows that target guidance alone cannot produce reliable spatial plans, while Mode B shows that restricting the model to tile scoring improves early selection but not fine localization; retaining more candidates recovers recall without overcoming errors made early in an irrevocable search, so interface constraints help without solving acquisition. Finding 4 – Domain pre-training does not transfer uniformly across evidence-seeking stages. At a matched model scale, broad medical instruction tuning can improve recognition, whereas localization is not monotonic in parameter count
and depends strongly on interface-specific spatial grounding, tool use, and confidence ranking; integration is comparatively less sensitive to scale. This stage-dependent pattern shows that neither larger models nor narrow pathology corpora are universal substitutes for agentic competence. V. D ISCUSSION Collectively, the cross-task findings isolate evidence acquisition as the primary bottleneck and motivate the mechanisms and design requirements examined below. A. Why These Patterns Hold The two T3 modes fail through complementary mechanisms. Mode A exposes spatial-planning and tool-use failures, whereas Mode B exhibits cascading coverage loss through hierarchical pruning. The two ablations expose complementary failure modes: overly tight or loose Mode A budgets destabilise localization, while larger Mode B schedules recover recall at rapidly growing compute cost. Neither model scale nor
DANKAI LIAO, et al.: PATHAGENTBENCH
9
Ground Truth Tumor Region MedGemma-27B
InternVL2.5-26B
6.5% mIoU
GPT-5.2
Autonomous Diagnostic Navigation (%)
Circle area ∝ log (total parameters)
6.9% mIoU
Quilt-LLaVA-7B
MedGemma-1.5-4B
Gemini-3-Flash MedGemma-4B
Qwen3.5-Flash 35B/3B activate
LLaVA-1.5-7B Qwen2.5-VL-7B
InternVL2.5-8B
Oracle
30B
Lingshu-7B
2/2 -> 6/12 -> 12/177
2/3
Gemini-3-Flash
300B
2/2 -> 5/12 -> 9/177
Phi-3-Vision-4.2B
Patho-R1-7B
Patho-R1-7B
AUTONOMOUS NAVIGATION NOT EVALUATED
Scale Encoding LLaMA-3.2-11B-Vision
2/3
Kimi-K2.5 1T/32B active 1/3
0/2 -> 0/12 -> 0/177
1T/reference max
LLaVA-NeXT-8B
LLaVA-Med-7B
Selected Trajectories
4B
7.1% mIoU
Lingshu-32B
DeepSeek-V4-Flash 284B/13B active
1/3
1/3
6.5% mIoU
Selected at 2.5x
1 of 3 evidence components available
Selected at 10x
6.5% mean IoU of Text-guided localization task
Selected at 40x
Pathology Evidence Competence (%)
Fig. 5: Cross-task capability landscape. Evidence competence (horizontal axis) combines Tasks 1, 2, and 4 with weights of 25%, 25%, and 50%; autonomous navigation (vertical axis, log scale) is the product of the three conditional Task 3 Mode B hit rates. Logo area encodes total parameters; the shaded band marks models without Mode B results.
search budget alone resolves the interaction among confidence ranking, branching capacity, and irrevocable pruning. The two modes therefore isolate complementary components of navigation. Mode A asks one model to plan, transform coordinates, call tools, and decide when to stop; additional actions can amplify spatial drift rather than improve coverage. Mode B removes those planning demands but converts weakly discriminative tile scores into structural pruning through topK retention. Their shared failure suggests that planning and scoring should first be evaluated and trained separately before being recombined in an end-to-end agent. The strongest models’ T2 advantage over T1 suggests that verifying a textual hypothesis against candidate images is easier than matching an image to a textual interpretation, supporting hypothesis-driven search as a natural WSI agent loop. This asymmetry motivates agents that formulate differential diagnoses and retrieve evidence to test them rather than scan without an explicit hypothesis. Operationally, such a loop could alternate broad candidate generation with targeted verification, updating the search frontier as evidence changes the differential. The transfer pattern across tasks further indicates that navigation depends on tool use, coordinate-space reasoning, and discriminative confidence ranking—competencies not guaranteed by medical or pathology corpora. Domain knowledge and agentic competence are orthogonal axes that must be trained jointly. B. Implications for Pathology Agent Design Fig. 6 illustrates this interface gap: pre-tiling makes local ranking tractable, but turning tile scores into a diagnostic trajectory still requires coordinate selection, reliable confidence ranking, and recovery from early errors, motivating four design requirements. (i) Backtracking is a hard requirement. Greedy navigation is bounded by irrevocable hierarchical pruning; explicit errorrecovery—beam search, re-entry, or sibling reconsideration— is needed so a single bad branch does not foreclose the subtree below.
Fig. 6: Qualitative case study of Task 3 Mode B navigation on one WSI. Yellow denotes pathologist-annotated tumor ground truth; black, blue, and orange boxes show the multi-scale regions retained by the GT-scored oracle, Gemini3-Flash, and Patho-R1-7B, respectively.
(ii) Reliable confidence ranking is a prerequisite for perlevel pruning. Top-K selection is only as good as its ranking signal; near-uniform scores cannot reliably order candidate tiles. A discriminative confidence signal is therefore a structural requirement, not merely a fine-tuning enhancement. (iii) Hybrid verification pairs VLM navigation with classical CPath classifiers. Tumor/stroma segmentation, foundationmodel embeddings, or task-specific detectors can veto evidently-wrong child selections without themselves solving the planning problem. We conjecture that the broadly medical and strongly instruction-tuned recipes represented in the top Lingshu/InternVL T4 cluster, plus explicit agentic posttraining along these architectural axes, may help close the navigation gap. (iv) Deployment must jointly address efficiency and privacy. Multi-minute Mode B latency and repeated transmission of WSI tiles and clinical context to external services raise practical concerns about turnaround time, cost, and privacy. Onpremises or hybrid inference should therefore be evaluated to keep sensitive data within institutional infrastructure, while benchmarking end-to-end latency and throughput under realistic hardware constraints. C. Limitations The autonomous-navigation protocol currently covers only breast-cancer WSIs, so generalisation across organs and tissue architectures remains untested.Mode A is restricted to three VLMs and Patho-R1 because the remaining models either lacked tool-calling support or used the localization tools too unreliably to produce meaningful results, limiting the generalizability of our findings beyond tool-capable agents. Both localization modes also use fixed budgets or branching schedules, and adaptive policies may change absolute performance. Although findings and diagnoses are pathologistauthored, distractors rely on MiniLM similarity; rank stability across the encoder ablation reduces, but does not eliminate, this concern. The 93% T4 result is also bounded by the four-
10
IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. XX, NO. XX, XXXX 2026
option format and should not be interpreted as open-ended diagnostic competence. Survival prediction, treatment response, and molecular subtyping remain outside the diagnostic-tree decomposition.
[20] Y. Sun et al., “PathAsst: A Generative Foundation AI Assistant towards Artificial General Intelligence of Pathology,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 5, pp. 5034–5042, Mar. 2024. [21] M. S. Seyfioglu, W. O. Ikezogwo, F. Ghezloo, R. Krishna, and L. Shapiro, “Quilt-LLaVA: Visual instruction tuning by extracting localized narratives from open-source histopathology videos,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2024, pp. 13 183–13 192. [22] C. Li et al., “LLaVA-med: Training a large language-and-vision assistant for biomedicine in one day,” in Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. Curran Associates, Inc., 2023, pp. 28 541–28 564. [23] W. Zhang et al., “Patho-R1: A Multimodal Reinforcement LearningBased Pathology Expert Reasoner,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 33, pp. 28 418–28 426, Mar. 2026. [24] Y. Chen et al., “SlideChat: A large vision-language assistant for wholeslide pathology image understanding,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2025, pp. 5134–5143. [25] S. Jiang, F. Liu, Z. Wang, L. Cai, and Y. Zhang, “PathReasoner-R1: Instilling Structured Reasoning into Pathology Vision-Language Model via Knowledge-Guided Policy Optimization,” Jan. 2026. [26] S. Yao et al., “ReAct: Synergizing Reasoning and Acting in Language Models,” Mar. 2023. [27] L. Wang et al., “A survey on large language model based autonomous agents,” Frontiers of Computer Science, vol. 18, no. 6, p. 186345, Mar. 2024. [28] T. Tanida, P. Müller, G. Kaissis, and D. Rueckert, “Interactive and explainable region-guided radiology report generation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2023, pp. 7433–7442. [29] Y. SUN et al., “CPathAgent: An Agent-based Foundation Model for Interpretable High-Resolution Pathology Image Analysis Mimicking Pathologists’ Diagnostic Logic,” in Advances in Neural Information Processing Systems, vol. 38. Curran Associates, Inc., 2025, pp. 101 673–101 731. [30] S. Li et al., “A co-evolving agentic AI system for medical imaging analysis,” Sep. 2025. [31] Z. Xu et al., “MMNavAgent: Multi-Magnification WSI Navigation Agent for Clinically Consistent Whole-Slide Analysis,” Mar. 2026. [32] S. Hua et al., “PathFound: An agentic multimodal model activating evidence-seeking pathological diagnosis,” Medical Image Analysis, vol. 113, p. 104200, Sep. 2026. [33] L. L. Weishaupt et al., “Evidence-based diagnostic reasoning with multiagent copilot for human pathology,” Mar. 2026. [34] Z. Xu et al., “A Versatile Pathology Co-pilot via Reasoning Enhanced Multimodal Large Language Model,” Aug. 2025. [35] X. He, Y. Zhang, L. Mou, E. Xing, and P. Xie, “PathVQA: 30000+ Questions for Medical Visual Question Answering,” Mar. 2020. [36] K. Chen et al., “OmniPathoVQA: Benchmarking pathology vision– language models with Encyclopedia-scale knowledge,” Medical Image Analysis, vol. 113, p. 104196, Sep. 2026. [37] P. Chen, C. Zhu, S. Zheng, H. Li, and L. Yang, “WSI-VQA: Interpreting Whole Slide Images by Generative Visual Question Answering,” Oct. 2024. [38] Y. Sun et al., “Pathbench: Advancing the benchmark of large multimodal models for pathology image understanding at patch and whole slide level,” IEEE Transactions on Medical Imaging, vol. 44, no. 10, pp. 4087–4097, Oct. 2025. [39] Q. Liu et al., “HealthAgentBench: A Unified Benchmark Suite of Realistic Agentic Healthcare Environments for Challenging Frontier AI Agents,” Jun. 2026. [40] Sentence-Transformers, “Sentence-transformers/all-MiniLM-L6-v2,” 2021. [41] N. Reimers and I. Gurevych, “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLPIJCNLP), K. Inui, J. Jiang, V. Ng, and X. Wan, Eds. Hong Kong, China: Association for Computational Linguistics, Nov. 2019, pp. 3982–3992. [42] Meta, “Llama 3.2 vision model card,” Sep. 2024. [43] S. Bai et al., “Qwen2.5-VL Technical Report,” 2025. [44] DeepSeek-AI et al., “DeepSeek-V4: Towards Highly Efficient MillionToken Context Intelligence,” Apr. 2026.
VI. C ONCLUSION In this work, we introduced PathAgentBench, a diagnostictree benchmark that evaluates pathology vision-language models (VLMs) as evidence-seeking systems rather than passive patch classifiers. Across 20 models, leading systems exceed 93% accuracy when integrating supplied multi-scale observations, yet text-guided localization remains below 0.09 mean IoU and autonomous tumor coverage falls from 52.2% at 2.5× to 2.02% at 40×. This asymmetry redirects future research from stronger reasoning over curated evidence toward reliable evidence acquisition. Progress requires spatially grounded training for coordinate reasoning and tool use; confidencecalibrated adaptive search with backtracking to recover from pruning errors; and hybrid verification with segmentation or task-specific detectors. These mechanisms should be tested across organs and in open-ended diagnostic settings, alongside privacy-preserving deployment with acceptable end-toend latency and cost, before pathology agents are considered clinically ready. R EFERENCES [1] S. Chakraborty et al., “Decoding the Visual Attention of Pathologists to Reveal Their Level of Expertise,” Oct. 2024. [2] M. Y. Lu et al., “A multimodal generative AI copilot for human pathology,” Nature, vol. 634, no. 8033, pp. 466–473, Oct. 2024. [3] J. Chen et al., “PathAgent: Toward Interpretable Analysis of Wholeslide Pathology Images via Large Language Model-based Agentic Reasoning,” Nov. 2025. [4] Y. Sun et al., “PathMMU: A Massive Multimodal Expert-Level Benchmark for Understanding and Reasoning in Pathology,” Oct. 2024. [5] Y. Liang et al., “WSI-LLaVA: A multimodal large language model for whole slide image,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Oct. 2025, pp. 22 718–22 727. [6] The Cancer Genome Atlas Research Network et al., “The Cancer Genome Atlas Pan-Cancer analysis project,” Nature Genetics, vol. 45, no. 10, pp. 1113–1120, Oct. 2013. [7] OpenAI, “Update to GPT-5 system card: GPT-5.2,” OpenAI, System Card, Dec. 2025. [8] Google DeepMind, “Gemini 3 Flash: Model Card,” Google DeepMind, Model Card, Dec. 2025. [9] xAI, “Grok 4 Model Card,” xAI, Model Card, Aug. 2025. [10] Z. Chen et al., “Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling,” Sep. 2025. [11] Qwen Team, “Qwen3.5: Towards Native Multimodal Agents,” https://qwen.ai/blog?id=qwen3.5, Feb. 2026. [12] J. Bai et al., “Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond,” Oct. 2023. [13] Kimi Team et al., “Kimi K2.5: Visual Agentic Intelligence,” Feb. 2026. [14] A. Grattafiori et al., “The Llama 3 Herd of Models,” Nov. 2024. [15] M. Abdin et al., “Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone,” Aug. 2024. [16] H. Liu, C. Li, Y. Li, and Y. J. Lee, “Improved baselines with visual instruction tuning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2024, pp. 26 296–26 306. [17] B. Li et al., “LLaVA-NeXT: Stronger llms supercharge multimodal capabilities in the wild,” May 2024. [18] A. Sellergren et al., “MedGemma Technical Report,” Apr. 2026. [19] LASA Team et al., “Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning,” 2025.