ToolSciVer: Multimodal Scientific Claim Verification with Visual Tool Augmented Reinforcement Learning

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

ToolSciVer: Multimodal Scientific Claim Verification with Visual Tool Augmented Reinforcement Learning

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

artificialintelligence knowledgerepresentation reasoning

artificial intelligence, reasoning, knowledge representation

ToolSciVer: Multimodal Scientific Claim Verification with Visual Tool Augmented Reinforcement Learning

Binglin Zhou1 Peng Shi2 Ryo Kamoi1 Nan Zhang1 Rui Zhang1∗ 1 2 The Pennsylvania State University University of Waterloo {bbz5169, rmz5227}@psu.edu

arXiv:2607.16131v1 [cs.CL] 17 Jul 2026

Abstract Multimodal Scientific Claim Verification (MSCV) requires models to verify scientific claims using visually grounded evidence from papers, including figures, tables, charts, and textual context. However, existing methods often fail because they struggle to locate decisive visual evidence, accurately read structured scientific visuals, and integrate multimodal observations into reliable reasoning. We introduce ToolSciVer, the first tool-augmented framework for MSCV to our knowledge. ToolSciVer equips a VLM with three type-aware visual tools, table row/column focus, chart-to-structure parsing, and high-resolution region zoom, which convert dense scientific visuals into explicit, claim-facing evidence, and trains the policy with Group Relative Policy Optimization (GRPO) under a composite reward of answer correctness, format validity, length control, tool-use efficiency, and toolvalidity penalties. Experiments on S CI V ER and M U S CI C LAIMS datasets on five VLMs from three model families (Qwen, InternVL, Gemma) demonstrate that our method achieves superior performance compared to four competitive baselines including prompting-based and RL-based tool-use methods, highlighting the effectiveness of learned, type-aware tool use for scientific claim verification.2

Introduction

Multimodal scientific claim verification (MSCV) is critical for building trustworthy scientific information systems [13, 29]. In MSCV, a model determines whether a claim is supported or refuted by evidence that spans both textual descriptions and scientific visuals. Unlike traditional textual fact verification [1, 24, 26–28], MSCV requires integrating heterogeneous modalities with different structures by addressing two key challenges: extracting localized, structured visual evidence and performing multi-step reasoning over that evidence to verify the claim. Despite growing interest, existing approaches to MSCV remain limited in addressing these challenges. Recent studies highlight that frontier proprietary models, such as GPT, Claude, and Gemini, still struggle particularly when correct verification depends on precise visual interpretation with tables, charts, and figures [6, 7, 13, 29]. They often fail to localize correct visual evidence and struggle to aggregate information across modalities via reliable grounded multi-step reasoning [7, 13, 29]. Meanwhile, while recent tool-augmented VLMs propose external visual operations for focused or extracted visual states [4, 22, 34], these methods are developed for general-domain visual question answering and rely on a narrow repertoire of image-editing operations (e.g., cropping, zooming, highlighting), leaving them poorly suited to the claim-conditioned evidence localization and crossfigure aggregation over scientific tables, charts, and multi-panel figures that MSCV demands. We summarize their limitations in Table 1. ∗ Corresponding author. 2 Code is available at https://github.com/psunlpgroup/Tool-Sciver.

Preprint.

User Input

Tool Call

ResNet-50 achieves higher Top-1 accuracy than DenseNet-121 on CIFAR-100, and the validation-loss curve in Figure 3 decreases monotonically from epoch 1 to epoch 5.

Image (Table 2) Model VGG-16 DenseNet-121 ResNet-50 EfficientNet-B0 Swin-T

Top-1 Acc. (%) 71.5 74.1 76.1 77.1 81.3

Table 2: Top-1 accuracy(%) on CIFAR-100.

Image (Figure 3)

Table Focus

Region Zoom

Tool Executor: Get the tool result for each turn Result 1 Tool Call: Table Focus(img1,row=“4”)

Reasoning: Need ResNet-50 performance. Use Table Focus for Table 2.

Tool Result(text): Row: ResNet-50|Top-1 Acc.(%) 76.1

Tool Call: Table Focus(img1,row=“4”)

Turn 2

Result 1

Reasoning: Need DenseNet-121 performance. Use Table Focus for Table 2.

Tool Call: Table Focus(img1,row=“3”) Tool Result(text): Row: DenseNet-121|Top-1 Acc.(%) 74.1

Tool Call: Table Focus(img1,row=“3”)

Result 1

0.4

3 Epoch

Context + Evidence

Reasoning Claim 1 is supported:

ResNet-50 76.1 > DenseNet-121 74.1

Claim 2 is not supported:

The validation-loss curve is not monotonically decreasing because it rises from epoch 3 (0.70) to epoch 4 (0.73).

Because the overall claim is claim1 and claim2, the full claim is false.

Verification Result

No (not supported)

Tool Call: Chart Parse(img2)

Reasoning: Need to inspect the validation-loss curve.

0.8

Claim Verification

or Yes (supported)

Turn 3

1.2

Validation Loss

Chart Parse

Turn 1

Text / Caption Table 2 reports Top-1 accuracy(%) on the CIFAR-100 test set for various models. Figure 3 shows validation loss across training epochs.

0.0

Evidence Extraction

Available Tool Set

Claim

Tool Result(text): Epoch:[1,2,3,4,5] Val_loss:[0.90, 0.78, 0.70, 0.73,0.65]

Tool Call: Chart Parse(img2)

Final Answer: No (not supported)

Figure 3:Validation loss across epochs.

Figure 1: Overview of ToolSciVer framework for evidence-grounded tool-augmented scientific claim verification. Given a claim, textual context, and scientific visual evidence, the model can either answer directly or iteratively call type-aware visual tools to acquire claim-relevant evidence. The collected tool observations are added to the augmented context and used for final claim verification.

In this paper, we introduce ToolSciVer, a type-aware tool-augmented reinforcement learning framework for MSCV. To address the above challenges, ToolSciVer includes two major innovations compared to prior work: (1) we design a suite of tools that are specialized to handle complex multimodal evidence in scientific papers, (2) we propose to use reinforcement learning to train models with a combination of rewards to efficiently use these tools and perform multi-step reasoning over extracted evidence. To be specific, as shown in Figure 1, our framework equips a VLM with three specialized tools matched to the visual types most common in scientific evidence: an OCR-backed Table Focus tool that returns localized row or column evidence from tables, a Chart Parse tool that converts charts and plots into structured textual observations, and a Region Zoom tool that returns a high-resolution crop for general or multi-panel figures. Furthermore, effective use of these tools requires the model to jointly decide whether a tool is needed, which tool matches the visual type, what arguments identify the relevant evidence, and how to integrate the returned observation into its judgment. Therefore, we train the model with Group Relative Policy Optimization (GRPO) [21] to learn selective, valid, and reliable tool use. Our reward combines final-answer correctness, output-format validity, length control, tool-use efficiency, and penalties for malformed or failed tool interactions, encouraging the model to acquire the right visual evidence only when needed. To demonstrate the effectiveness of our method, we conduct a comprehensive set of experiments on two recent challenging benchmarks on MSCV, S CI V ER [29] and M U S CI C LAIMS [13], which contains complex claims to verify with diverse evidence modality, scientific subdomains, and reasoning structures. We apply ToolSciVer on three backbone model familites (Qwen, InternVL, Gemma) and compare against four competitive baselines: non-tool CoT inference, prompt-only tool-use CoT inference, and two RL-based multimodal tool-use baselines VT OOL -R1 [34] and O PEN T HINK IMG [22]. Across both benchmarks, ToolSciVer consistently improves over non-tool CoT inference and promptonly tool access, and achieves stronger overall performance than the RL-based tool-use baselines. Our further analyses and ablation study show that our method improves the performance by promoting more accurate visual evidence localization, more appropriate tool selection, and more reliable reasoning over extracted evidence across tables, charts, and general scientific figures. Our contributions are summarized as follows: • We study Multimodal Scientific Claim Verification through the lens of visual evidence extraction and identify observation-stage visual evidence acquisition as a key bottleneck. 2

Claim-Facing Visual Evidence Tools Method

Target Task

Table Evid. Chart Evid. Region Insp. Struct. Obs.

Learning

Eff. Reward

VisProg [5] ViperGPT [23] MM-ReAct [35] Visual Sketchpad [8] ReFocus [4] DePlot/MatCha [14, 15] VTool-R1 [34] OpenThinkIMG [22]

Gen. visual reasoning Gen. visual reasoning Gen. multimodal reasoning Visual/math reasoning Struct. img. underst. Chart understanding Struct. visual reasoning Chart/tool reasoning

✗ ✗ ✗ ✗ ⃝ ✗ ⃝ ✗

✗ ✗ ✗ ✗ ⃝ ✓ ⃝ ⃝

⃝ ⃝ ⃝ ⃝ ⃝ ✗ ⃝ ⃝

✗ ✗ ✗ ✗ ✗ ✓ ✗ ✗

Training-free Training-free Prompt Prompt Prompt/Sup. Sup./Pretrain RL RL

✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗

ToolSciVer (Ours)

MSCV

✓

Table 1: Comparison of ToolSciVer with representative tool-augmented and visual reasoning methods. ✓ denotes a dedicated claim-facing evidence extractor or parser for the corresponding visual type; ⃝ denotes partial or generic support through visual editing, sketching, cropping/zooming, OCR, or code-based visual tools, without a dedicated scientific-evidence extractor; ✗ denotes no explicit support. Struct. Obs. refers to structured observations directly usable as verification evidence, such as table rows/columns or chart values, rather than raw OCR text, bounding boxes, or edited images.

• We propose ToolSciVer, the first visual tool augmented framework for MSCV to the best of our knowledge. It consists of a type-aware visual tool suite that exposes multimodal, claim-relevant scientific evidence, trained with reinforcement learning for selective, valid, and efficient tool use. • We empirically validate the effectiveness of ToolSciVer across challenging MSCV benchmarks in multiple scientific domains and reasoning difficulties, achieving state-of-the-art performance on different backbone model families compared with strong tool-use baselines.

Related Work

Scientific and multimodal claim verification. Claim verification is commonly formulated as evidence-grounded prediction, where a system retrieves evidence and predicts whether it supports or refutes a claim. In the scientific domain, S CI FACT and follow-up work study verification against paper abstracts, full-document context, open-domain retrieval, and LLM-based zero-shot settings [1, 26–28]. Scientific table verification further shows that claims over scientific evidence require grounding, compositional reasoning, and careful treatment of ambiguity [16]. Recent benchmarks extend this line to multimodal evidence: S CI V ER evaluates multimodal scientific claim verification over scientific documents, while M U S CI C LAIMS focuses on figure-centric scientific claims [13, 29]. Related work studies multi-hop multimodal claims, verification over visual and textual table representations, and unified multimodal fact verification [12, 30, 37]. Recent cross-modal scientific verification studies also show that current multimodal models remain weak on table-, chart-, and figure-based evidence [6, 7]. Our work builds on these benchmarks, but focuses on the mechanism behind the errors: acquiring claim-relevant visual evidence before making the final verification decision. Visual evidence extraction from structured scientific visuals. Structured visual understanding has been studied for tables, charts, plots, and document-like images. For tabular evidence, TabFact, SciTab, and M2 -TabFact show that table-based verification requires accurate grounding and compositional reasoning [2, 16, 37]. For charts and plots, ChartQA highlights visual and logical reasoning over chart content, while DePlot and MatCha show the value of converting charts into explicit intermediate representations through plot-to-table translation or chart derendering [14, 15, 17]. These works motivate our use of structured evidence, but fixed OCR, parsing, or cropping alone does not decide whether a tool is needed, which tool to call, or which row, column, chart element, or region is relevant to the claim. We therefore treat visual evidence extraction as part of a learned verification policy rather than as fixed preprocessing. Tool-augmented VLMs and reinforcement learning. Tool use has been widely studied for language models. ReAct interleaves reasoning and acting through prompting, while Toolformer learns API use from self-supervised signals [20, 36]. Visual programming and tool-use systems such as VISPROG, ViperGPT, and MM-ReAct call external vision modules or generate executable programs without task-specific training [5, 23, 35]. Visual Sketchpad and ReFocus further show that intermediate visual operations can expose focused visual states for multimodal reasoning [4, 8]. Closest to our 3

work, VT OOL -R1 and O PEN T HINK IMG train VLMs to use visual tools with reinforcement learning [22, 34], while broader tool-learning work emphasizes reward design, execution reliability, and action efficiency [9, 18, 31]. Unlike these general visual tool-use settings, our method targets MSCV specifically: tools are organized by scientific visual type, their outputs are claim-facing evidence, and GRPO training encourages selective, valid, and efficient tool use for scientific verification.

Our Method

3.1

Task Formulation

We study multimodal scientific claim verification (MSCV), where a model verifies a scientific claim using textual context and scientific visual evidence. Each instance is denoted as x = (q, V, t), where q is the claim, V = {vi }N i=1 is the set of associated scientific visuals, and t denotes optional textual context, such as captions, surrounding paragraphs, or benchmark metadata. The model outputs a normalized prediction ŷ in the benchmark label space. In our experiments, both benchmarks are cast into binary verification with Y = {supported, refuted}. 3.2

Framework Overview

The key challenge in MSCV is that decisive evidence is often visually localized and structurally constrained, requiring the model to read specific table entries, recover chart trends, or inspect local regions of dense scientific figures. We therefore formulate MSCV as an evidence acquisition problem and propose ToolSciVer, a tool-augmented framework that selectively extracts claim-relevant visual evidence before making the final verification decision. As shown in Figure 1, given a claim, textual context, and scientific visual evidence, the policy either answers directly from the original input or issues a type-aware visual tool call. A scheduler validates and executes each call, appends the returned observation to the context, and allows the model to continue reasoning over the augmented evidence. The resulting trajectory is written as τ = (u1 , o1 , u2 , o2 , . . . , uT , ŷ), where ut denotes a model turn, ot denotes an optional tool observation, and ŷ is the final answer. Each turn contains either one tool call or the final answer, which keeps the action space narrow and grounds the interaction in visual evidence extraction rather than unrestricted agentic search. This design follows three principles: tools should expose claim-relevant evidence rather than generic image descriptions; tool use should be type-aware across tables, charts, and general figures; and tool calls should be selective, invoked only when the original input is insufficient for reliable verification. 3.3

Type-Aware Visual Tool Suite

We design a compact tool suite for three common evidence structures in MSCV: tables, charts, and general scientific figures. Each tool call targets one image through an image_path argument, and the policy decides which tool to call and what arguments to provide. Detailed tool interfaces are provided in Appendix A.1. Table Focus. Scientific claims often depend on localized tabular evidence, such as a method’s performance on a specific dataset or the value of a particular metric. For table images, we first convert the image into a structured CSV-like representation using OCR. We then expose focus_row and focus_column, which return only the selected row or column. This reduces irrelevant context and makes the returned evidence easier to compare against the claim. Chart Parse. Claims over charts or plots often require recovering trends, relative ordering, thresholds, or numerical values. For chart images, parse_content converts visual chart content into structured textual observations by using stronger VLM, such as JSON-like entries of chart elements and values. Region Zoom. Many scientific figures are dense, multi-panel, or diagrammatic, and the decisive evidence may occupy only a small local region. For these cases, image_zoom_in crops and enlarges a model-selected region, returning a zoomed-in image. Coordinates are normalized to a [0, 1000] scale, making the interface resolution-invariant. 4

3.4

Reward Design

Our reward aligns the policy with the deployment goal of MSCV: produce a correct final decision, follow the required output format, avoid overlong responses, and use tools only when they help acquire relevant visual evidence. For a rollout τ , we define R(τ ) = seff (τ ) (rans (τ ) + λfmt rfmt (τ )) + λlen rlen (τ ) + λtool rtool (τ ) (1) In our experiments, we set λfmt = λlen = λtool = 0.1. Answer reward. The answer reward measures final task correctness: rans (τ ) = I[ŷ = y] where y is the ground-truth label and ŷ is the normalized model prediction. Format reward. The format reward encourages the model to produce a parseable final response: rfmt (τ ) = 0.5 Ireason (τ )+0.5 Ianswer (τ ), where Ireason (τ ) indicates whether the final response contains the required reasoning segment under the prompt template, and Ianswer (τ ) indicates whether the response contains exactly one valid normalized final-answer marker. This term stabilizes training and ensures that final predictions can be reliably extracted. Tool-efficiency coefficient. To encourage selective rather than excessive tool use, we introduce an OTC-style tool-efficiency coefficient. Let m(τ ) be the number of tool calls in rollout τ . Let C(x) be the set of strict-correct rollouts in the GRPO group, where a rollout is strict-correct if it predicts the correct label and ends with a valid final answer. If C(x) is non-empty, we estimate the group-local optimal tool budget as n⋆ (x) = minτi ∈C(x) m(τi ). If no strict-correct rollout exists, we set seff (τ ) = 1. Otherwise, for maximum tool budget cmax ,  ⋆ 1,   n = 0, m = 0,   mπ ⋆ cos 2m+cmax , n = 0, m > 0, seff (τ ) = (2)  0, n⋆ > 0, m = 0,    sin mπ , n⋆ > 0, m > 0. m+n⋆ This coefficient rewards correct trajectories that use close to the group-local minimal number of tool calls, while penalizing unnecessary tool use when a direct answer is sufficient. We set cmax = 5. Length reward. We use a soft overlong penalty to discourage unnecessarily long generations. Let L(τ ) denote the completion length in tokens, let Lmax be the maximum completion length, and let C denote the soft-cache length. We define the soft threshold as Lsoft = Lmax − C. The length reward is rlen (τ ) = − max(L(τC)−Lsoft ,0) . Thus, the penalty is zero below the soft threshold and decreases linearly as the response approaches the hard length limit. In our experiments, Lmax = 4096 and C = 768. Tool-interaction penalty. The tool-interaction penalty discourages malformed tool turns and execution failures. Let eturn (τ ) be the number of malformed tool turns, let eexec (τ ) be the number of tool-execution failures, and let Ibad-end (τ ) indicate whether the rollout contains at least one tool-related error and still fails to terminate with a valid final answer. We compute r̃tool (τ ) = − 0.20 I[eturn ≥ 1] − 0.10 max(eturn − 1, 0) − 0.30 I[eexec ≥ 1] − 0.15 max(eexec − 1, 0) (3) − 0.40 Ibad-end (τ ), and clip the result: rtool (τ ) = clip[−1.5,0] (r̃tool (τ )) . This gives the model dense feedback for invalid tool behavior, rather than treating all tool failures as a single sparse error. 3.5

Learning Selective Tool Use with GRPO

Prompt-only tool use is brittle because the model must decide whether a tool is needed, select the appropriate tool family, provide valid arguments, and use the returned observation in the final decision. To learn this behavior, we train the policy with Group Relative Policy Optimization (GRPO) [21]. For each input x, we sample a group of G rollouts G(x) = {τi }G i=1 from the current policy. After rollout completion, we assign a scalar reward Ri based on equation 1 and compute the group-relative Ri −meanG

j=1 advantage Ai = . The policy is then optimized with the standard GRPO objective, stdG j=1 Rj encouraging high-reward trajectories while staying close to the reference policy. G 1 XX JGRPO (θ) = min (ri,t (θ)Ai , clip(ri,t (θ), 1 − ϵ, 1 + ϵ)Ai ) − βDKL (πθ ∥πref ) (4) G i=1 t

ri,t (θ) is the token-level policy ratio, πref is the reference policy, and β controls the KL penalty. 5

Experimental Setup

Datasets. We evaluate on two MSCV benchmarks: S CI V ER [29] and M U S CI C LAIMS [13]. S CI V ER contains scientific claims paired with multimodal evidence from scientific documents and reports four reasoning subsets: Analytic, Direct, Parallel, and Sequential. M U S CI C LAIMS contains figure-centric scientific claims from biology, physics, and chemistry. We use the binary verification setting for both datasets and report subset/domain accuracy together with sample-weighted overall accuracy. Exact train/test split sizes are reported in Appendix B.1. Models. We evaluate five open-source VLM backbones from three model families: Qwen3.54B and Qwen3.5-9B [19], InternVL3.5-4B and InternVL3.5-8B [32], and Gemma4-E4B [3]. All trainable methods are initialized from the same backbone and trained on the same MSCV training splits. Baselines. We compare our method with four competitive baselines. Non-tool CoT inference [33] prompts the model to reason step by step over the claim, context, caption, and image, without access to external tools. Prompt-only tool-use CoT inference [25] gives the model access to the same MSCV tool APIs and scheduler as our method, but without tool-use training. Dataset-adapted VT OOL -R1 [34]. and dataset-adapted O PEN T HINK IMG [22] retain the original tool suites and reward designs of VT OOL -R1 and O PEN T HINK IMG, respectively, but we train them on the same MSCV training splits as our method. These baselines test whether general trained visual tool-use recipes are sufficient for MSCV. Finally, we also report advanced proprietary models, including GPT-4o, GPT-5.4, and Claude Sonnet 4.6, as external references under the non-tool CoT setting; they are not part of the controlled tool-training comparison.

Results and Analysis

We organize our results and analysis around the central question of whether ToolSciVer improves MSCV by helping VLMs acquire and use claim-relevant visual evidence. Section 5.1 first reports the main MSCV results across datasets, model families, and baselines. To better understand why our method works, we conducted four diagnostic analyses on Qwen3.5-4B. • Evidence acquisition (Section 5.2): Does the model obtain more claim-relevant visual evidence? • Tool selection (Section 5.3): Does the model learn to select tools according to the image type? • Efficiency (Section 5.4): Does the model solve MSCV with fewer tool calls and shorter responses? • Evidence-based reasoning (Section 5.5): Once evidence is acquired, does the model reason more reliably over it? 5.1

Main Results

Table 2 shows three main trends. First, ToolSciVer improves over non-tool C OT on both benchmarks across all five open-source backbones, with particularly clear gains on the information-dense M U S CI C LAIMS benchmark. Second, prompt-only tool access is unstable: on S CI V ER, Tool Prompt underperforms non-tool C OT for four of five backbones, showing that simply exposing tools does not reliably improve verification. Third, ToolSciVer matches or improves over VT OOL -R1 in all completed overall comparisons, including the Qwen3.5-4B tie on S CI V ER, and outperforms O PEN T HINK IMG on M U S CI C LAIMS and all completed S CI V ER settings except Qwen3.5-9B. These results suggest that MSCV benefits from task-aligned visual-evidence tools beyond general visual tool-use training. The proprietary C OT references further show that advanced closed-source VLMs are not uniformly reliable on MSCV: Claude Sonnet 4.6 performs best among them on M U S CI C LAIMS, whereas GPT-5.4 is substantially weaker on S CI V ER. This underscores the difficulty of MSCV and motivates explicit visual-evidence acquisition for open-source VLMs. 5.2

Claim-Relevant Visual Evidence Acquisition

Model

S CI V ER

M U S CI C LAIMS

Backbone

Method

Analytic Direct Parallel Sequential Overall

Qwen3.5-4B Qwen3.5-4B Qwen3.5-4B Qwen3.5-4B Qwen3.5-4B