arXiv CSopen access

SpecVQA: A Benchmark for Spectral Understanding and Visual Question Answering in Scientific Images

SpecVQA: A Benchmark for Spectral Understanding and Visual Question Answering in Scientific Images

arXiv:2604.28039v1 [cs.AI] 30 Apr 2026

Jialu Shen, Han Lyu, Suyang Zhong, Hanzheng Li, Haoyi Tao, Nan Wang, Changhong Chen, and Xi Fang⋆ DP Technology

Abstract. Spectra are a prevalent yet highly information-dense form of scientific imagery, presenting substantial challenges to multimodal large language models (MLLMs) due to their unstructured and domainspecific characteristics. Here we introduce SpecVQA, a professional scientific-image benchmark for evaluating multimodal models on scientific spectral understanding, covering 7 representative spectrum types with expert-annotated question–answer pairs. The aim comprises two aspects: spectra scientific QA evaluation and corresponding underlying task evaluation. SpecVQA contains 620 figures and 3100 QA pairs curated from peer-reviewed literature, targeting both direct information extraction and domain-specific reasoning. To effectively reduce token length while preserving essential curve characteristics, we propose a spectral data sampling and interpolation reconstruction approach. Ablation studies further confirm that the approach achieves substantial performance improvements on the proposed benchmark. We test the capability of prominent MLLMs in scientific spectral understanding on our benchmark and present a leaderboard. This work represents an essential step toward enhancing spectral understanding in multimodal large models and suggests promising directions for extending visual–language models to broader scientific research and data analysis. Keywords: Spectral Understanding · Visual QA · Multimodal Large Models · Scientific Image Benchmark

1

Introduction

The emergence of Multimodal Large Language Models (MLLMs) has marked a new milestone in artificial intelligence, demonstrating remarkable advancements in visual–language understanding, cross-modal reasoning, and zero-shot generalization across diverse domains [8, 29, 37]. Both proprietary systems such as GPT-5 [35], Gemini-3 [19], Claude-4.5 [3] and advanced open-source architectures such as Qwen3-VL [5] exemplify an increasingly unified understanding of the world, continuously pushing the boundaries of machine perception [39, 49]. ⋆

Corresponding author

2

Shen et al.

Fig. 1: Overview of data curation and the proposed SpecVQA benchmark. Stage 1 automatically harvests high-quality figure–caption pairs from peer-reviewed journals. Stage 2 performs multi-stage filtering and spectrum-type classification, yielding a curated set of 20k spectrum–caption pairs under expert supervision. Stage 3 leverages MLLMs to distill QA pairs from the curated figures; domain experts then re-annotate representative cases and assign them to two semantic categories. Stage 4 synthesizes an additional 201k underlying samples to support model evaluation and to assess the effectiveness of our sampling strategy.

The potential to translate this broad, general intelligence into domain-specific scientific discovery is substantial, particularly in fields that depend heavily on complex data visualization. Despite strong general-purpose reasoning abilities, current MLLMs still struggle with scientific spectral figures, where dense curves, precise numerical reading, and domain knowledge are tightly coupled. It requires a synthesis of pattern recognition and deep mechanistic understanding [16, 21]. In spectral image understanding, MLLMs typically target two complementary tasks: Scientific QA Task and Underlying Construction Task. (1) Scientific QA Task: expert-level interpretation and reasoning based solely on the visual characteristics of the spectral plot and the domain knowledge implicitly encoded within it. The fundamental limitations of current MLLMs in spectral understanding are summarized as follows. – Mislocalization of Fine Features. Inability to accurately locate or distinguish subtle spectral details, such as minor impurity peaks, small chemical shifts, or the precise positions of crystalline reflections. – Inaccurate Quantitative Extraction. Errors in reading key numerical values from axes or curves, including integration ratios, maximum absorption wavelengths (λmax ), and precise binding energies. – Flawed Mechanistic Reasoning. Logical inconsistencies when connecting visual evidence to scientific principles, resulting in incorrect structure elucidation, phase identification, or interpretation of fragmentation pathways. (2) Underlying Construction Task: extracting and reconstructing the highdensity numerical data points that define the spectral curve, often for a specific

SpecVQA

3

sub-figure within a composite visualization. Relative early works include ChartOCR [30] and Plot2Spectra [23]. This task is closely related to chart data extraction and plot digitization, but differs in that it focuses on dense spectral curve reconstruction from sub-figures embedded in composite scientific visualizations, requiring accurate curve localization, separation, and interpolation. We show some bad cases of these two tasks in the Appendix. In the scientific QA task, it is noted that these limitations cannot be accurately assessed or diagnosed by existing benchmarks. Standard VQA datasets [20] primarily evaluate general object recognition, whereas chart-based VQA datasets [31, 32] focus on elementary data retrieval without requiring scientific knowledge grounding. To address this gap, we propose SpecVQA (Spectral Visual Question Answering)1 , a professional scientific-image benchmark for evaluating multimodal models on scientific spectral understanding. It encompasses the seven most prevalent and analytically important types of spectra, which focus enables a quantitative evaluation of MLLMs in terms of both scientific accuracy and their capacity to leverage embedded domain knowledge, and thereby guiding the development of more robust, domain-aware multimodal models. In the underlying task, high-resolution spectral data present a major tokenefficiency challenge for MLLMs, especially in tasks requiring reconstruction or regression of the underlying curves. Unlike general charts with sparse key points, spectra comprise thousands of data points, each representing critical fluctuations that models are required to capture with high accuracy. Whether learning from or predicting these points, MLLMs will tokenize them into a longer sequence compared with those in other tasks, which might impose heavy computational and memory burdens. Attempts to compress the data often lead to truncation or oversimplification and making fine-tuning on large spectral datasets impractical. This challenge arises regardless of input image length, as it depends on the minimum token count required to faithfully represent the spectral curve. To overcome the underlying task challenges, we introduce an efficient data sampling and interpolation reconstruction strategy. This method aims to drastically reduce output token length while preserving the spectral features, alleviating the token bottleneck and enabling large-scale MLLMs fine-tuning on high-density scientific data. The resulting test set also forms part of our benchmark, which it used to quantitatively evaluate the performance of several large models on the spectral underlying task. In summary, this paper makes the following major contributions: A High-quality Benchmark (SpecVQA). We introduce a comprehensive, domain-specific VQA benchmark rigorously curated by scientific experts, specifically targeting the seven most critical types of scientific spectra. An Efficient Data Sampling Strategy. We propose Efficient Data Sampling and Interpolation Reconstruction strategy that employs adaptive sampling and interpolation to substantially reduce text token length, improving training efficiency while preserving spectral fidelity. A Comprehensive Leaderboard. We establish a detailed leaderboard that 1

https://huggingface.co/datasets/UniParser/SpecVQA

4

Shen et al.

quantitatively evaluates and compares major MLLMs on the SpecVQA benchmark, identifying their limitations and providing actionable insights for future domain-specialized multimodal model development.

2

Related Work

Our work is positioned at the intersection of general MLLMs, specialized visual question answering, and advanced computational methods for scientific data. 2.1

Foundations of MLLMs

The foundation of MLLMs is the integration of powerful vision encoders with large language models (LLMs) [11, 45]. Early approaches demonstrated the feasibility of aligning visual and linguistic feature spaces through contrastive learning [28, 37], while subsequent models introduced sophisticated projection layers to bridge the gap between frozen image encoders and LLMs [1, 27]. The recent surge of instruction-tuned MLLMs, such as LLaVA [26, 29] and Qwen-VL [4, 6], has further advanced zero-shot and few-shot reasoning capabilities across a wide range of multimodal tasks [44, 50]. Trained on massive web-scale corpora [43], these models exhibit remarkable generalization ability. However, such generalized training inherently dilutes model sensitivity to the highly specialized, formalized, and quantitative visual semantics found in scientific spectra [39]. 2.2

Specialized VQA and Scientific Reasoning

As general VQA benchmarks mature, research attention has increasingly shifted toward specialized visual reasoning. Chart and Data Visualization VQA. Efforts in data visualization understanding such as PlotQA [32] and ChartQA [31] focus on interpreting standard business and statistical charts (e.g., bar, line, and pie charts). While these tasks involve axis reading and arithmetic comparison, they primarily deal with discrete or low-density data points and emphasize simple data retrieval rather than scientific interpretation. They fall short of supporting advanced reasoning, such as distinguishing π–π ∗ from n–π ∗ electronic transitions in UV–Vis spectra or inferring chemical environments from peak multiplicity in NMR spectra. The specialized nature of spectral interpretation thus demands a distinct category of reasoning beyond elementary data extraction. Scientific Document and Diagram Understanding. Broader scientific benchmarks such as ScienceQA [41] and other datasets for figure understanding, integrate textual and visual modalities. However, their visual elements are typically schematic or illustrative (e.g., flowcharts or simple diagrams), lacking the continuous, high-density data and strict analytical constraints that characterize experimental spectra. Our work is distinguished by its requirement for expertlevel quantitative analysis and knowledge-driven deduction applied directly to images of raw scientific measurements.

SpecVQA

2.3

5

Computational Methods for Spectral Data and Long Context VLM

Traditional Spectral Analysis. Spectral analysis has long been a cornerstone of computational spectroscopy and chemometrics. Techniques ranging from Partial Least Squares (PLS) to modern deep learning models (CNNs and RNNs) have been employed on raw numerical data for tasks such as classification, compound identification, and property prediction [16,21,25]. These approaches serve as predictive tools operating purely on numerical inputs, which fundamentally differ from our goal of achieving visual language grounding based on spectral images. They do not address the need to bridge MLLMs’ visual perception with scientific language understanding. Data Sampling and Dimensionality Reduction. Efficiently processing the high-density data generated by modern instruments poses a longstanding challenge. Reducing the number of sampled points while retaining essential curve characteristics is critical for computational efficiency. Classical simplification methods such as the Ramer–Douglas–Peucker (RDP) algorithm [12, 38] achieve this through adaptive piecewise linear approximation, whereas more recent approaches leverage Topological Data Analysis (TDA) [47] and persistent homology to capture scale-invariant geometric features. Although these techniques are well-established in signal processing and computer graphics, their targeted integration into multimodal large-model pipelines to alleviate the long-tokenization burden of high-fidelity scientific curves remains unexplored.

3

Method

To address the token length crisis resulting from high-density spectral data, we propose the Data Sampling and Interpolation Reconstruction Strategy. This technique significantly reduces the length of the output text tokens without compromising the scientific fidelity required for accurate interpretation. 3.1

Adaptive Key-point Sampling

Unlike uniform downsampling which risks discarding narrow and critical signal peaks, our method employs a three stage strategy to maximize the preservation of scientific features while achieving a high reduction ratio R. The original highresolution spectral curve is defined by N data points: Sori = {(xi , yi )}N i=1 . Step 1: Signal Smoothing. The raw data is first subjected to a Savitzky–Golay (S-G) filter [42] to effectively reduce high-frequency noise without distorting the characteristic peak shapes and widths. This preprocessing step enhances the robustness of subsequent feature extraction. Step 2: Uniform Baseline Sampling. To ensure coverage of baseline and low-feature regions, we uniformly sample approximately 5% of the points from the smoothed curve. This guarantees a minimal token representation. Step 3: Critical Feature Sampling. We apply the RDP algorithm [12, 38] to

6

Shen et al.

(a)

(b)

(c)

(d)

Fig. 2: Comparison of the noisy real data curve, sampled points, and the reconstructed smooth curve. Four examples from the experiments are presented.

the smoothed curve. RDP is a line simplification algorithm that recursively identifies and keeps only the most crucial points required to approximate the curve within a specified maximum distance threshold ϵ. After this step, the sampled points inherently encode the most significant bends and features of the spectral curve, preserving peak summits and critical changes in slope. The final sampled set Ssampled is the union of points from step 2 and step 3.

3.2

Interpolation Reconstruction

The sparse sampled point set Ssampled is then used to reconstruct a new smooth visual representation Irecovered . We employ cubic spline interpolation since it ensures second-order continuity, which is crucial for preserving the natural shape and smoothness of spectral curves that MLLMs must interpret. I_{\text {recovered}} = \text {Image}(\text {CubicSpline}(\mathcal {S}_{\text {sampled}}))

(1)

The resulting Irecovered reconstruct the original data line well and drastically shortening the input token length, thus mitigating memory and truncation issues. Figure 2 shows the comparsion of real data line and reconstruction data line. 3.3

Fidelity Evaluation

To quantitatively evaluate the efficiency of our strategy in preserving the spectral curve’s shape and fidelity after downsampling, we employ several metrics.

SpecVQA

7

Chamfer Distance (dCD ) [9, 14] measures the average squared Euclidean distance between the nearest points of two point sets. Let Ctrue denote the original high-resolution point cloud and Crec denote the reconstructed point cloud obtained from sampled points. A lower value of dCD indicates better preservation of geometric fidelity between the reconstructed and original point clouds. d_{CD}\left (C_{\text {true }}, C_{\text {rec }}\right ) = \frac {1}{\left |C_{\text {true }}\right |} \sum _{p \in C_{\text {true }}} \min _{q \in C_{\text {rec }}}\|p-q\|^2 +\frac {1}{\left |C_{\text {rec }}\right |} \sum _{q \in C_{\text {rec }}} \min _{p \in C_{\text {true }}}\|q-p\|^2. Hausdorff Distance (dHD ) [13, 22] measures the maximum distance from a point in one point set to its nearest neighbor in the other set. We define Ctrue as the original high-resolution point cloud and Crec as the reconstructed point cloud from sampled points. A lower dHD indicates that the worst-case geometric deviation between the reconstructed and original point clouds is smaller, implying better structural preservation. d_{HD}\left (C_{\text {true }}, C_{\text {rec }}\right )= \max \{\max _{p \in C_{\text {true }}} \min _{q \in C_{\text {rec }}}\|p-q\|, \max _{q \in C_{\text {rec }}} \min _{p \in C_{\text {true }}}\|q-p\|\}.

Wasserstein Distance (dW D ) [40, 46] measures the minimal transportation cost required to transform one point distribution into another. We define Ctrue as the original high-resolution point cloud and Crec as the reconstructed point cloud from sampled points. A lower dW D indicates that the reconstructed point cloud better matches the overall spatial distribution of the original point cloud. d_{WD}\left (C_{\text {true}}, C_{\text {rec}}\right )= \frac {1}{N}\sum _{i=1}^N\|p_i-q_i\|.

4

The SpecVQA Benchmark

To address the limitations of general MLLMs in interpreting scientific figures and to provide a systematic evaluation framework, we introduce SpecVQA (Spectral Visual Question Answering). SpecVQA is designed to emulate the complex, visually grounded, and knowledge-integrated reasoning process that human experts employ when interpreting experimental spectral data. 4.1

Benchmark Scope

SpecVQA focuses on seven widely utilized types of spectra frequently encountered in scientific research. This specialization ensures that the benchmark’s evaluation results are directly relevant to practical scientific analysis. The spectrum types and their core interpretation requirements are as Table 1.

8

Shen et al. Table 1: The seven spectrum types and their core interpretation requirements Spectrum Type

Description Involves interpreting chemical shifts (δ), integration NMR (Nuclear Magnetic areas, coupling constants (J), and multiplicity to Resonance) deduce molecular structures. Involves identifying characteristic vibrational IR (Infrared Absorption frequencies (cm−1 ) to determine the presence or Spectroscopy) absence of specific functional groups. Requires analyzing diffraction peak positions (2θ), intensities, and widths to infer crystal structures, XRD (X-ray Diffraction) lattice parameters, and crystallite sizes. Focuses on interpreting Raman-active vibrational modes to probe molecular symmetry and bonding Raman (Raman characteristics, particularly in materials such as Spectroscopy) carbon-based systems. Involves analyzing molecular ion peaks and MS (Mass Spectrometry). fragmentation patterns to determine molecular weight and identify structural fragments. Requires extracting maximum absorption UV-Vis (Ultraviolet-Visible wavelengths (λmax ) and quantifying concentrations Spectrophotometry). based on absorption intensity. Involves analyzing core-level peak binding energies XPS (X-ray Photoelectron to determine elemental composition and chemical Spectroscopy). states at the sample surface.

4.2

Data Curation

High-Quality Data Acquisition. SpecVQA is constructed from high-quality, real-world scientific data, emphasizing both accessibility and authoritative provenance. The raw image corpus is primarily aggregated from Open Access (OA) literature, the arXiv preprint server, and large-scale technical reports. Refer to Uniparser [15] and OmniScience 2 for the process of acquiring literature data. Specifically, to ensure the data authority and impact, we prefer figures extracted from papers with high citation counts or published in high-impact factor journals. We will give more consideration to literature from the last 20 years to ensure image clarity, while also ensuring the integrity of the dataset by not omitting classic literature in related fields. Systematic Image Segmentation Pipeline. The figure corpus was systematically extracted using a dedicated pipeline. Initial figure-text pairs were extracted coarsely from the literature corpus using keywords corresponding to the seven spectrum types. Then the article provenance including journal and discipline was cross-referenced to perform a layered case extraction, ensuring a diverse, broad 2

https://huggingface.co/datasets/UniParser/OmniScience

SpecVQA

9

coverage of domain knowledge within the chemical and material sciences. Image segmentation and extraction are performed the same as OmniScience. Refined Figure-Label Pair Selection Rules. During the final manual and semi-automated refinement, the following rules were applied to ensure semantic independence and data quality. Of the 60k figure-caption obtained, 30k were automatically selected by regularization method and Seed1.5-VL, and then 20k spectrum-label pairs were retained through manual screening. – Sufficient Description. Only figures where the accompanying text description was sufficient to convey the essential physical or chemical information were retained. Text requiring excessive external context or providing minimal description was discarded. – Single-Type Spectrum. If the figure contained only one spectrum and the text description was complete and context-independent, the figure-text pair was saved, typically named "filename + A". – Comparison Cases. If the figure consisted of multiple sub-figures of the same type, and the text explicitly emphasized their contrast or comparison, the entire original figure was retained without splitting, supporting comparative analysis tasks. – Resolution Consideration. Recognizing the prevalence of low-resolution images in real-world applications, we did not limit selection to only highresolution spectra, thus ensuring the high practical generalization of the dataset. – Semantic Independence. A core consideration during selection was ensuring that different spectrum sub-figures exhibit weak association, thus ensuring the semantic independence of individual samples. 4.3

QA Benchmark

Expert screening. To ensure the scientific validity and representativeness of the benchmark, a team of domain experts manually curated a subset of 620 spectral figures from an initial pool of over 20k candidates collected from peerreviewed journals and open-access scientific databases. After taking into account both time cost and answer quality, all candidate images were used to generate QA pairs via distillation using GPT-o4-mini and Gemini-2.5-Pro, with five QA pairs produced for each image. The selection process was guided by following criteria. Figures were chosen to cover a broad range of visual configurations, including single-spectrum plots, multi-panel figures, and both homogeneous and heterogeneous subplot arrangements, enabling evaluation of MLLMs on complex figure layouts and visual hierarchies. Each figure was also screened to ensure that the accompanying captions or contextual text contained sufficient experimental and interpretive information to support grounded question–answer generation. In addition, experts examined the semantic relationships among subplots, distinguishing between independent

10

Shen et al.

and tightly coupled panels to facilitate multi-instance reasoning. The dataset further emphasizes diversity across journals, research fields, and data sources to reduce bias and improve generalizability. Finally, both high and low resolution images were included to reflect practical scenarios and evaluate the visual robustness of MLLMs. During the selection process, the QA pairs distilled from the chosen images were also reviewed by experts. In addition to verifying factual correctness, the experts were required to assess the semantic relevance of each QA pair. We expected the questions and answers to reflect issues that are genuinely of interest in their research and pose challenges to MLLMs, rather than trivial or nonsensical visual Q&As. If any distilled pair failed to meet these criteria, the experts revised it until it satisfied the requirements. Through this expert screening pipeline, SpecVQA achieves a carefully balanced collection of spectral figures that combines scientific rigor, visual diversity, and contextual completeness—forming a reliable foundation for multimodal evaluation in specialized scientific reasoning. Multi-Stage Bias-Averse QA Generation. The construction of the SpecVQA question-answering pairs employs a robust, multi-stage pipeline designed to maximize question diversity, minimize MLLM generation bias, and ensure scientific accuracy. The final 3,100 QA pairs are classified into two critical categories based on the required cognitive effort. We provided both Chinese and English version to test the scientific performance of the large model in different languages. – Category 1 (L0): Descriptive Question (2,080 QAs). This category evaluates the ability to directly understand and interpret visual information in spectral figures. It includes tasks such as extracting textual elements (e.g., titles, labels, legends, and axis information), identifying key values like maxima or minima and peak locations, recognizing patterns or entities that satisfy specific conditions, understanding the layout of multi-panel subplots, and classifying visual features such as peak shapes. – Category 2 (L1): Reasoning Question (1,020 QAs). This category focuses on higher-level analytical and reasoning abilities based on the visual content of the figures. It involves comparing multiple entities to draw conclusions, counting elements that meet certain criteria, performing calculations on numerical information in the figure, analyzing trends or predicting changes in spectral patterns, and conducting causal analysis to interpret the scientific phenomena reflected in the data. Dataset Statistics. This part of SpecVQA benchmark comprises 620 unique scientific spectral figures. Through rigorous expert-level generation and filtering, we constructed a total of 3,100 high-quality Question-Answer pairs (an average of 5 QA pairs per figure). The underlying, raw numerical data associated with these spectral images is made publicly available as a part of the benchmark. The critical distinction of SpecVQA is that it requires models to interpret the spectrum rather than merely read the graph, specifically addressing the high data density and expert knowledge required for curve-based scientific analysis.

SpecVQA

Q:The position of the rightmost peak in the Pt(0) peak corresponds to approximately which binding energy (eV)? A: About 71 eV. Q: Which has a larger peak area, Pt(II) or Pt(0)? A: Pt(0) has a larger peak area. Q: Approximately what binding energy (eV) corresponds to the position of the Ni 2p peak? A: About 855 eV. Q: Which of the two peaks with the highest intensity in the left and right subgraphs is stronger? A: Subplot b. Q: How many peaks appear in Pt(II) in Figure a? A: 2.

11

Q: What is the approximate binding energy of the Pd 3d5/2 peak of Pd(0)? A: About 335 eV. Q: What is the binding energy shift (ΔE) of the Cu 2p3/2 peak? A: -0.7 eV. Q: What chemical valence states are Pd and Cu respectively? A: Pd(0) and Cu(0). Q: Which element has a worse signal-to-noise ratio in its raw data spectral lines? A: Cu. Q: What is the peak located at approximately 341 eV called? A: 3d₃/₂.

Fig. 3: Two examples from the Scientific QA part of the SpecVQA benchmark. Each example contains one spectral figure and five QA pairs in different languages. Here we show the English QA pairs.

4.4

Underlying Benchmark Construction

Based on the 20k selected real spectral images, we generated a total of 200k corresponding image datasets across the seven categories. We also designed QA pairs that samples key points from these images, which served as the training set for the underlying task. The prompt format for the QA pairs is shown in the Appendix, and a comparison between the generated data and the real data is shown in Figure 4. Specifically, we first generated the data based on the style and format of 20k real spectra. We then continuously optimized the generation method based on feedback from human scientists, iterating through versions until it was approved by them. Next, we generated batches of 10k data points each, from which human experts randomly selected 10% for quality control. A batch was accepted only if its pass rate exceeded 95%; otherwise, it was discarded. This procedure was repeated until 20 qualified batches were obtained.

5

Experiments and Results

5.1

Experimental Setup

We conduct a large-scale evaluation on the following set of several prominent MLLMs: Gemini-3-Flash-Preview [18], Gemini-3-Pro-Preview [19], Claude-4.5Sonnet [3], Claude-4-Sonnet [2], Doubao-seed-1-6-flash [7], DeepSeek-VL2 [48], Qwen3-VL-8B Thinking [5], Qwen3-VL-4B-Instruct [5], Qwen3-VL-32B-Instruct [5], GPT-4o [34], GPT-o4-mini [33], GPT-5 [35], GPT-o3 [36], Gemini-2.5-Pro [10], and Gemini-2.5-Flash [17], and the Qwen3-VL-4B Backbone used for our method. Our proposed model variants are ours without the sampling strategy (Qwen3-VL-4B trained on 20k QA data and 200k original underlying data,

12

Shen et al.

(a) ms-real

(b) uv-real

(c) raman-real

(d) ms-generated

(e) uv-generated

(f ) raman-generated

Fig. 4: Comparison between real spectral images and generated spectral images. The first row shows real images, while the second row shows generated spectral images of the same categories.

serving as the absolute baseline) and ours with the sampling strategy (Qwen3-VL-4B trained on 20k QA data and 200k sampling underlying data). The benchmark data does not overlap with the training set. This experimental phase of our proposed model is supervised fine-tuning (SFT), designed to adapt the selected visual-language backbone model to the domain of our constructed SpecVQA benchmark, thereby improving the model’s scientific understanding and quantification capabilities in spectral image question answering tasks. By selectively freezing the vision tower and multimodal projector, the fine-tuning strategy preserves the robustness and structural coherence of the pretrained visual representations, thereby promoting stable convergence and enhancing the model’s ability to capitalize on its established visual priors during downstream adaptation. 5.2

Evaluation Metrics

We used the following evaluation metrics in the two types of experimental tasks respectively. Scientific Accuracy. We report results separately for L0 (Descriptive Question) and L1 (Reasoning Question). Each answer is scored by GPT-o4-mini with reference to the standard answers. If the error rate falls within the predefined error tolerance of 5 percentage points, the answer is deemed correct; otherwise, it is regarded as incorrect. The accuracy rate is then calculated accordingly. Point Cloud Distance Score. Used to quantify the fidelity of the data sampling and interpolation reconstruction strategy. We normalized it in actual application and compute the score as (2). Here d means the three kind of distance

SpecVQA

13

Table 2: Performance comparison of models on the SpecVQA benchmark. Model

Think

Weight

API-Version

Gemini-3-Flash-Preview Gemini-3-Pro-Preview Gemini-2.5-Pro Gemini-2.5-Flash GPT-5(high) GPT-o4mini GPT-5(medium) GPT-o3 GPT-5(low) GPT-5.1 GPT-5.2 Claude-4.5-Sonnet Doubao-seed-1-6-flash Claude-4-Sonnet Qwen3-VL-8B-Thinking Doubao-seed-1-6-250615 Qwen3-VL-8B-Instruct Qwen3-Omni-30B-A3B-Thinking Doubao-seed-1-6 DeepSeek-VL2

✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ × × ✓ ✓ ✓ ✓ ✓ × ✓ × ×

Proprietary Proprietary Proprietary Proprietary Proprietary Proprietary Proprietary Proprietary Proprietary Proprietary Proprietary Proprietary Proprietary Proprietary Open Proprietary Open Open Proprietary Open

20251217 20251119 20250617 20250617 20250807 20250416 20250807 20250416 20250807 20251113 20251211 20250929 20250828 20250514 20250615 20250615 -

English (en) Chinese (zh) Overall Descriptive Reasoning Descriptive Reasoning Question (L0) Question (L1) Question (L0) Question (L1) 0.7782 0.7759 0.8047 0.7900 0.7872 0.7718 0.7759 0.7939 0.7731 0.7787 0.7645 0.7486 0.7758 0.7571 0.7615 0.7395 0.7043 0.7473 0.7185 0.7274 0.7017 0.6959 0.7115 0.7147 0.7059 0.7144 0.6836 0.7237 0.6996 0.7054 0.6923 0.6977 0.7154 0.7100 0.7039 0.6953 0.7100 0.7027 0.7043 0.7031 0.6825 0.6968 0.7066 0.6987 0.6961 0.6776 0.6347 0.6899 0.6290 0.6578 0.6776 0.6328 0.6899 0.6158 0.6540 0.6148 0.5518 0.6315 0.5782 0.5941 0.6060 0.5687 0.6148 0.5574 0.5867 0.5947 0.5282 0.5957 0.5565 0.5688 0.5864 0.5348 0.5805 0.5508 0.5631 0.5721 0.5499 0.5736 0.5452 0.5602 0.5721 0.4256 0.5927 0.4727 0.5158 0.5442 0.4868 0.5368 0.4765 0.5111 0.5530 0.4435 0.5697 0.4586 0.5062 0.4657 0.3183 0.4092 0.3079 0.3753

presented in Section 3.3. \label {score} Score = 1 -\overline {d}\left (C_{\text {true }}, C_{\text {rec }}\right ) = 1-\frac {{d}\left (C_{\text {true }}, C_{\text {rec }}\right )}{ \max _{x, y \in C_{\text {true }} \cup C_{\text {rec }}}\|x-y\|^2}.

(2)

During the experiment, it was noticed that the number of lines output by some MLLMs differed from the original image. To address this, Hungarian algorithm [24] was employed to match the model’s output with the real data, calculate the score for each line after matching, and then take the average of the valid data. 5.3

SpecVQA Benchmark Leaderboard

Performance evaluations of different MLLMs were conducted on our VQA benchmark. Table 2 presents the leaderboard, featuring the performance of MLLMs, with results clearly delineated across the two primary question categories. Ours model achieves the higher Sci-Acc across both Information Extraction and Reasoning categories in both Chinese and English, demonstrating its superior ability to handle both fine-grained data retrieval and complex scientific deduction. This confirms that optimizing the input visual representation is significantly effective for this domain. Additionaly, it is clearly indicated that there is a substantial performance boost from our Data Sampling technique. 5.4

Efficiency and Fidelity Analysis of Data Sampling strategy

This section rigorously evaluates the impact of our data strategy in terms of input fidelity and final performance. The result is showed in Table 3.

14

Shen et al.

Table 3: Fidelity Comparison of the Data Sampling and Interpolation Reconstruction Strategy. Model GPT-4o Deepseek-VL2 Ours(without sampling strategy) Qwen3-VL-32B-Instruct Qwen3-VL-4B-Instruct GPT-o4-mini GPT-o3 GPT-5 Gemini-2.5-Pro Gemini-2.5-Flash Ours(with sampling strategy) Testset with sampling strategy

ScoreScoreScoreChamfer Distance Hausdorff Distance Wasserstein Distance 0.3996 0.4102 0.4146 0.7575 0.7606 0.8229 0.8330 0.8389 0.8893 0.8953 0.9776 0.9899

0.3645 0.3207 0.3827 0.7485 0.7500 0.7819 0.7988 0.7988 0.7042 0.7994 0.9595 0.9794

0.4100 0.4366 0.4159 0.7784 0.7827 0.8413 0.8460 0.8516 0.9051 0.9109 0.9810 0.9904

Fidelity Evaluation. We first assess the quality of the point reconstruction using three distance scores. Specifically, we evaluate the fidelity of the reconstructed curve (Crec ) obtained by our approach relative to the original real data curve. The results indicate a high reconstruction accuracy, with a score approaching 0.99, which is shown as model line Testset with sampling strategy. During experiments, we observe that most MLLMs perform a truncation operation on the input data. Some models such as Deepseek-VL2 handle excessively long tokens by directly replacing the remaining content with an ellipsis, rather than truncating or interrupting the processing. Our strategy substantially reduce visual token length to 6.7 percentage of the original, improving training efficiency and avoiding the cut-off of long data while preserving spectral fidelity. Ablation Study on Performance and Efficiency. Then we compare the performance of our model (ours with the sampling strategy) against the baseline (ours without the sampling strategy) and other MLLMs. The higher score for our strategy validates that our adaptive sampling strategy substantially preserves the spatial configuration and characteristic features of the original high-resolution spectrum, which is essential for accurate scientific analysis.

6

Conclusion

This work introduces SpecVQA, a comprehensive benchmark dedicated to evaluating the scientific visual intelligence of Multimodal Large Language Models across the seven most critical types of scientific spectra. By focusing on identifying and categorizing critical cases where general MLLMs fail, SpecVQA provides a necessary benchmark for advancing scientific MLLM research. Furthermore, our proposed Efficient Data Sampling and Interpolation Reconstruction strategy effectively addresses the inherent token length crisis associated with highresolution spectral data, significantly improving the efficiency and feasibility of

SpecVQA

15

training domain-specific models. The establishment of a comprehensive leaderboard based on SpecVQA offers a clear and quantitative measure of current MLLM performance, guiding future research toward building truly expert-level scientific visual systems. In the future, we will further unify the two parts of the benchmark data and build professional scientific image datasets in a wider range of fields.

16

Shen et al.

References 1. Alayrac, J.B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al.: Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems (NeurIPS) 35, 23716–23736 (2022) 2. Anthropic: Claude 4 sonnet. https://www.anthropic.com/claude (2025), accessed: 2026 3. Anthropic: Claude 4.5 sonnet. https://www.anthropic.com/claude (2025), accessed: 2026 4. Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., Zhou, J.: Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966 (2023) 5. Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., Ge, W., Guo, Z., Huang, Q., Huang, J., Huang, F., Hui, B., Jiang, S., Li, Z., Li, M., Li, M., Li, K., Lin, Z., Lin, J., Liu, X., Liu, J., Liu, C., Liu, Y., Liu, D., Liu, S., Lu, D., Luo, R., Lv, C., Men, R., Meng, L., Ren, X., Ren, X., Song, S., Sun, Y., Tang, J., Tu, J., Wan, J., Wang, P., Wang, P., Wang, Q., Wang, Y., Xie, T., Xu, Y., Xu, H., Xu, J., Yang, Z., Yang, M., Yang, J., Yang, A., Yu, B., Zhang, F., Zhang, H., Zhang, X., Zheng, B., Zhong, H., Zhou, J., Zhou, F., Zhou, J., Zhu, Y., Zhu, K.: Qwen3-vl technical report (2025), https://arxiv.org/abs/2511.21631 6. Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al.: Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923 (2025) 7. ByteDance: Doubao seed 1.6 flash multimodal model. https://www.volcengine. com/ (2025), accessed: 2026 8. Changpinyo, S., Sharma, P., Ding, N., Soricut, R.: Conceptual 12m: Pushing webscale image-text pre-training to recognize long-tail visual concepts. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 3558–3568 (2021) 9. Chazelle, B., Dobkin, D.P.: Optimal convex decompositions. In: Machine Intelligence and pattern recognition, vol. 2, pp. 63–133. Elsevier (1985) 10. DeepMind: Gemini 2.5 pro (model page). https://deepmind.google/models/ gemini/pro/ (2025), official model information / capabilities page. Accessed 202506-05 11. Dosovitskiy, A., Beyer, L., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. In: International Conference on Learning Representations (ICLR) (2021) 12. Douglas, D.H., Peucker, T.K.: Algorithms for the reduction of the number of points required to represent a digitized line or its caricature. Cartographica: the international journal for geographic information and geovisualization 10(2), 112–122 (1973) 13. Dubuisson, M.P., Jain, A.K.: A modified hausdorff distance for object matching. In: Proceedings of 12th international conference on pattern recognition. vol. 1, pp. 566–568. IEEE (1994) 14. Fan, H., Su, H., Guibas, L.: A point set generation network for 3d object reconstruction from a single image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 605–613 (2017) 15. Fang, X., Tao, H., Yang, S., Huang, C., Zhong, S., Lu, H., Lyu, H., Li, X., Zhang, L., Ke, G.: Uni-parser technical report. arXiv preprint arXiv:2512.15098 (2025)

SpecVQA

17

16. Fine, J.A., Rajasekar, A.A., Jethava, K.P., Chopra, G.: Spectral deep learning for prediction and prospective validation of functional groups. Chemical science 11(18), 4618–4630 (2020) 17. Google Cloud / Vertex AI: Gemini 2.5 flash (product / vertex ai docs). https:// cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/2-5-flash (2025), model card / Vertex AI documentation. Accessed 2025-04-09 18. Google DeepMind: Gemini 3 flash: Fast and efficient multimodal model. https: //ai.google.dev/gemini-api/docs/models/gemini (2025), accessed: 2026 19. Google DeepMind: Gemini 3 pro: Frontier multimodal model. https://ai.google. dev/gemini-api/docs/models/gemini (2025), accessed: 2026 20. Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., Parikh, D.: Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 6904–6913 (2017) 21. Houhou, R., Bocklitz, T.: Trends in artificial intelligence, machine learning, and chemometrics applied to chemical data. Analytical Science Advances 2(3-4), 128– 141 (2021) 22. Huttenlocher, D.P., Klanderman, G.A., Rucklidge, W.J.: Comparing images using the hausdorff distance. IEEE Transactions on pattern analysis and machine intelligence 15(9), 850–863 (2002) 23. Jiang, W., Li, K., Spreadbury, T., Schwenker, E., Cossairt, O., Chan, M.K.: Plot2Spectra: an automatic spectra extraction tool. Digital Discovery 1(5), 719– 731 (2022) 24. Kuhn, H.W.: The hungarian method for the assignment problem. Naval Research Logistics Quarterly 2(1–2), 83–97 (1955) 25. Lansford, J.L., Vlachos, D.G.: Infrared spectroscopy data-and physics-driven machine learning for characterizing surface microstructure of complex materials. Nature communications 11(1), 1513 (2020) 26. Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., et al.: Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326 (2024) 27. Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pretraining with frozen image encoders and large language models. In: Proceedings of the International Conference on Machine Learning (ICML). pp. 19730–19742. PMLR (2023) 28. Li, J., Selvaraju, R., Gotmare, A., Joty, S., Xiong, C., Hoi, S.C.H.: Align before fuse: Vision and language representation learning with momentum distillation. Advances in Neural Information Processing Systems (NeurIPS) 34, 9694–9705 (2021) 29. Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in Neural Information Processing Systems (NeurIPS) 36, 34892–34916 (2023) 30. Luo, J., Li, Z., Wang, J., Lin, C.Y.: Chartocr: Data extraction from charts images via a deep hybrid framework. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision. pp. 1917–1925 (2021) 31. Masry, A., Do, X.L., Tan, J.Q., Joty, S., Hoque, E.: Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In: Findings of the association for computational linguistics: ACL 2022. pp. 2263–2279 (2022) 32. Methani, N., Ganguly, P., Khapra, M.M., Kumar, P.: Plotqa: Reasoning over scientific plots. In: Proceedings of the ieee/cvf winter conference on applications of computer vision. pp. 1527–1536 (2020) 33. OpenAI: GPT-4 Mini and Related Variants. OpenAI Platform Documentation (Conceptual Citation) (2024)

18

Shen et al.

34. OpenAI: Gpt-4o: General-purpose multimodal model (2024), https://openai. com/index/gpt-4o-system-card/ 35. OpenAI: Introducing gpt-5. https://openai.com/index/introducing- gpt- 5/ (2025), official product page / announcement. Accessed 2025-08-07 36. OpenAI: Introducing openai o3 and o4-mini. https : / / openai . com / index / introducing-o3-and-o4-mini/ (2025), official announcement (o3 and o4-mini). Accessed 2025-04-16 37. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: Proceedings of the International Conference on Machine Learning (ICML). pp. 8748–8763. PmLR (2021) 38. Ramer, U.: An iterative procedure for the polygonal approximation of plane curves. Computer graphics and image processing 1(3), 244–256 (1972) 39. Roberts, J., Han, K., Houlsby, N., Albanie, S.: Scifibench: Benchmarking large multimodal models for scientific figure interpretation. Advances in Neural Information Processing Systems (NeurIPS) 37, 18695–18728 (2024) 40. Rubner, Y., Tomasi, C., Guibas, L.J.: The earth mover’s distance as a metric for image retrieval. International journal of computer vision 40(2), 99–121 (2000) 41. Saikh, T., Ghosal, T., Mittal, A., Ekbal, A., Bhattacharyya, P.: Scienceqa: A novel resource for question answering on scholarly articles. International Journal on Digital Libraries 23(3), 289–301 (2022) 42. Savitzky, A., Golay, M.J.: Smoothing and differentiation of data by simplified least squares procedures. Analytical chemistry 36(8), 1627–1639 (1964) 43. Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al.: Laion-5b: An open largescale dataset for training next generation image-text models. Advances in Neural Information Processing Systems (NeurIPS) 35, 25278–25294 (2022) 44. Tan, H., Bansal, M.: Lxmert: Learning cross-modality encoder representations from transformers. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP) (2019) 45. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS) 30 (2017) 46. Villani, C.: Optimal Transport: Old and New. Springer (2009) 47. Wasserman, L.: Topological data analysis. Annual review of statistics and its application 5(2018), 501–532 (2018) 48. Wu, Z., Chen, X., Pan, Z., Liu, X., Liu, W., Dai, D., Gao, H., Ma, Y., Wu, C., Wang, B., et al.: Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302 (2024) 49. Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., Chen, E.: A survey on multimodal large language models. National Science Review 11(12), nwae403 (2024) 50. Zhou, L., Palangi, H., Zhang, L., Hu, H., Corso, J., Gao, J.: Unified vision-language pre-training for image captioning and vqa. In: Proceedings of the AAAI conference on artificial intelligence. vol. 34, pp. 13041–13049 (2020)

Appendix

1

Cases in spectral image understanding by MLLMs

In spectral image understanding, MLLMs typically target two complementary tasks: Scientific QA Task and Underlying Construction Task. Here we show some bad cases of these two tasks. 1.1

Mislocalization of Fine Features

Inability to accurately locate or distinguish subtle spectral details, such as minor impurity peaks, small chemical shifts, or the precise positions of crystalline reflections.

Fig. 1: Case 1

Q: In panel a, how does the bandwidth of the absorption peak at approximately 350 nm change over time (from day 0 to day 28)?

20

A: Remains essentially unchanged. GPT-5: Decreases . Gemini-2.5-pro: Increases. Qwen3-VL-32B-Instruct: Not visible. 1.2

Inaccurate Quantitative Extraction

Errors in reading key numerical values from axes or curves, including integration ratios, maximum absorption wave lengths (λmax ), and precise binding energies.

Fig. 2: Case 2

Q: In Figure a, at approximately which wavelength is the absorption edge of the Co-CN sample (green curve) located? A: Approximately 380 nm. GPT-5: 450 nm. Gemini-2.5-pro: 500 nm. Qwen3-VL-32B-Instruct: Not visible. 1.3

Flawed Mechanistic Reasoning

Logical inconsistencies when connecting visual evidence to scientific principles, resulting in incorrect structure elucidation, phase identification, or interpretation of fragmentation pathways. Q: What are the primary diffraction peaks in the X-ray diffraction (XRD) pattern, and to which material do these peaks correspond?? A: The major diffraction peaks correspond to β-MnO2 and Zn-buserite. GPT-5: β-MnO2 . Gemini-2.5-pro: Zn-buserite, β-MnO2 , Zn(CF3 SO3 )2 , InZn(CF3 SO3 )2 electrolyte, Ti. Qwen3-VL-32B-Instruct: (001), (002), (003), (004), (005) – Zn-buserite.

Appendix

21

Fig. 3: Case 3

1.4

Underlying Task

Whether learning from or predicting these points, MLLMs will tokenize them into a longer sequence compared with those in other tasks, which might impose heavy computational and memory burdens.

Fig. 4: Case 4

22

Deepseek-VL2: <subplot A> <line 1>[530.00,0.12],[532.50,0.30],...,[542.50,1.80],[545.00,0]< ⌋ ,→ /line> <line 2>[530.00,0.00],[532.50,0.00],...,[542.50,0.00],[545.00,0.0 ⌋ ,→ 0]</line> <line 3>[530.00,0.00],[532.50,0.00],...,[542.50,0.00],[545.00,0.0 ⌋ ,→ 0]</line> <line 4>[530.00,0.00],[532.50,0.00],...,[542.50,0.00],[545.00,0.0 ⌋ ,→ 0]</line> <line 5>[530.00,0.00],[532.50,0.00],...,[542.50,0.00],[545.00,0.0 ⌋ ,→ 0]</line> <line 6>[530.00,0.00],[532.50,0.00],...,[542.50,0.00],[545.00,0.0 ⌋ ,→ 0]</line> <line 7>[530.00,0.00],[532.50,0.00],...,[542.50,0.00],[545.00,0.0 ⌋ ,→ 0]</line> <line 8>[530.00,0.00],[532.50,0.00],...,[542.50,0.00],[545.00,0.0 ⌋ ,→ 0]</line> <line 9>[530.00,0.00],[532.50,0.00],...,[542.50,0.00],[545.00,0.0 ⌋ ,→ 0]</line> </subplot> GPT-o4-mini: <subplot A> <line1>[528.00,0.00],[530.00,0.60],[532.00,0.10],[534.00,0.00],[5 ⌋ ,→ 36.00,0.00],[538.00,0.00],[540.00,0.00],[542.00,0.00],[544.00 ⌋ ,→ ,0.00],[546.00,0.00]</line> <line2>[528.00,0.00],[530.00,0.00],[532.00,0.80],[534.00,1.60],[5 ⌋ ,→ 36.00,0.50],[538.00,0.00],[540.00,0.00],[542.00,0.00],[544.00 ⌋ ,→ ,0.00],[546.00,0.00]</line> <line3>[528.00,0.00],[530.00,0.00],[532.00,0.00],[534.00,0.30],[5 ⌋ ,→ 36.00,1.80],[538.00,2.00],[540.00,1.80],[542.00,0.80],[544.00 ⌋ ,→ ,0.10],[546.00,0.00]</line> <line4>[528.00,0.00],[530.00,0.00],[532.00,0.00],[534.00,0.00],[5 ⌋ ,→ 36.00,0.00],[538.00,0.10],[540.00,0.30],[542.00,0.80],[544.00 ⌋ ,→ ,0.50],[546.00,0.00]</line> <line5>[528.00,0.00],[530.00,0.00],[532.00,0.00],[534.00,0.00],[5 ⌋ ,→ 36.00,0.00],[538.00,0.00],[540.00,0.00],[542.00,0.10],[544.00 ⌋ ,→ ,0.60],[546.00,0.10]</line> <line6>[528.00,0.05],[530.00,0.65],[532.00,0.95],[534.00,1.95],[5 ⌋ ,→ 36.00,2.35],[538.00,2.15],[540.00,2.15],[542.00,1.75],[544.00 ⌋ ,→ ,1.25],[546.00,0.15]</line> <line7>[528.00,0.05],[530.00,0.05],[532.00,0.05],[534.00,0.05],[5 ⌋ ,→ 36.00,0.05],[538.00,0.05],[540.00,0.05],[542.00,0.05],[544.00 ⌋ ,→ ,0.05],[546.00,0.05]</line> </subplot>

Appendix

23

GPT-5: <subplot A>\n<line 1>[529.00,0.30],[531.00,0.80],[533.00,0.30],[5 ⌋ ,→ 35.00,0.05],[537.00,0.01],[539.00,0.00],[541.00,0.00],[543.00 ⌋ ,→ ,0.00],[545.00,0.00]</line>\n<line ,→ 2>[529.00,0.02],[531.00,0.10],[533.00,1.10],[535.00,1.30],[53 ⌋ ,→ 7.00,0.20],[539.00,0.03],[541.00,0.00],[543.00,0.00],[545.00, ⌋ ,→ 0.00]</line>\n<line ,→ 3>[529.00,0.00],[531.00,0.00],[533.00,0.05],[535.00,0.20],[53 ⌋ ,→ 7.00,1.00],[539.00,1.10],[541.00,0.40],[543.00,0.05],[545.00, ⌋ 0.00]</line>\n<line ,→ 4>[529.00,0.00],[531.00,0.00],[533.00,0.00],[535.00,0.05],[53 ⌋ ,→ 7.00,0.10],[539.00,0.20],[541.00,0.50],[543.00,0.40],[545.00, ⌋ ,→ 0.10]</line>\n<line ,→ 5>[529.00,0.00],[531.00,0.00],[533.00,0.00],[535.00,0.00],[53 ⌋ ,→ 7.00,0.05],[539.00,0.10],[541.00,0.60],[543.00,1.10],[545.00, ⌋ ,→ 0.30]</line>\n<line ,→ 6>[529.00,0.42],[531.00,0.99],[533.00,1.53],[535.00,1.67],[53 ⌋ ,→ 7.00,1.42],[539.00,1.48],[541.00,1.54],[543.00,1.58],[545.00, ⌋ ,→ 0.42]</line>\n<line ,→ 7>[529.00,0.10],[531.00,0.09],[533.00,0.08],[535.00,0.07],[53 ⌋ ,→ 7.00,0.06],[539.00,0.05],[541.00,0.04],[543.00,0.03],[545.00, ⌋ ,→ 0.02]</line>\n</subplot> ,→ Gemini-2.5-pro: <subplot A>\n<line 1>[530.13,0.09],[530.51,0.21],[530.93,0.44],[5 ⌋ ,→ 31.41,0.69],[531.75,0.81],[532.22,0.80],[532.68,0.58],[533.15 ⌋ ,→ ,0.33],[533.64,0.15],[534.12,0.06],[534.60,0.03]</line>\n<line ,→ 2>[531.75,0.05],[532.32,0.14],[532.84,0.35],[533.32,0.70],[53 ⌋ ,→ 3.79,1.17],[534.21,1.48],[534.69,1.43],[535.15,1.05],[535.64, ⌋ ,→ 0.60],[536.14,0.28],[536.60,0.10],[537.09,0.03]</line>\n<line ,→ 3>[533.36,0.02],[534.40,0.06],[535.27,0.18],[536.10,0.50],[53 ⌋ ,→ 6.87,1.05],[537.38,1.42],[537.84,1.58],[538.35,1.49],[538.93, ⌋ ,→ 1.13],[539.52,0.71],[540.10,0.40],[540.67,0.20],[541.24,0.09] ⌋ ,→ ,[541.83,0.04],[542.42,0.01]</line>\n<line ,→ 4>[538.45,0.02],[539.11,0.05],[539.75,0.15],[540.36,0.33],[54 ⌋ ,→ 0.88,0.54],[541.34,0.65],[541.80,0.62],[542.33,0.46],[542.86, ⌋ ,→ 0.29],[543.39,0.15],[543.92,0.06],[544.47,0.02]</line>\n<line ,→ 5>[540.23,0.01],[540.82,0.06],[541.36,0.21],[541.88,0.49],[54 ⌋ ,→ 2.35,0.83],[542.72,1.04],[543.18,0.99],[543.66,0.69],[544.15, ⌋ ,→ 0.38],[544.62,0.16],[545.00,0.06]</line>\n

24

<line 6>[529.98,0.36],[530.51,0.55],[531.02,0.77],[531.55,0.96],[ ⌋ ,→ 532.06,1.12],[532.57,1.29],[533.06,1.48],[533.53,1.68],[534.0 ⌋ ,→ 0,1.79],[534.46,1.82],[534.98,1.69],[535.53,1.39],[536.03,1.1 ⌋ ,→ 3],[536.52,1.11],[537.02,1.27],[537.54,1.49],[538.03,1.59],[5 ⌋ ,→ 38.53,1.52],[539.04,1.29],[539.54,1.01],[540.03,0.79],[540.54 ⌋ ,→ ,0.77],[541.04,0.91],[541.55,1.17],[542.06,1.41],[542.54,1.58 ⌋ ],[542.92,1.64],[543.41,1.42],[543.90,0.99],[544.40,0.57],[54 ⌋ ,→ 4.89,0.29]</line>\n<line ,→ 7>[530.00,0.08],[532.50,0.06],[535.00,0.05],[537.50,0.04],[54 ⌋ ,→ 0.00,0.03],[542.50,0.01],[545.00,0.00]</line>\n</subplot> ,→ Qwen3-VL-32B-Instruct: <subplot a>\n<line 1>[530.00,0.00],[530.50,0.15],[531.00,0.30],[5 ⌋ ,→ 31.50,0.45],[532.00,0.60],[532.50,0.75],[533.00,0.85],[533.50 ⌋ ,→ ,0.90],[534.00,0.95],[534.50,1.00],[535.00,1.05],[535.50,1.08 ⌋ ,→ ],[536.00,1.10],[536.50,1.12],[537.00,1.13],[537.50,1.14],[53 ⌋ ,→ 8.00,1.15],[538.50,1.15],[539.00,1.15],[539.50,1.15],[540.00, ⌋ ,→ 1.15],[540.50,1.15],[541.00,1.15],[541.50,1.15],[542.00,1.15] ⌋ ,→ ,[542.50,1.15],[543.00,1.15],[543.50,1.15],[544.00,1.15],[544 ⌋ ,→ .50,1.15],[545.00,1.15],[545.50,1.15],[546.00,1.15],[546.50,1 ⌋ ,→ .15],[547.00,1.15],[547.50,1.15],[548.00,1.15],[548.50,1.15], ⌋ ,→ [549.00,1.15],[549.50,1.15],[550.00,1.15],[550.50,1.15],[551. ⌋ ,→ 00,1.15],[551.50,1.15],[552.00,1.15],[552.50,1.15],[553.00,1. ⌋ ,→ 15],[553.50,1.15],[554.00,1.15],[554.50,1.15],[555.00,1.15],[ ⌋ ,→ 555.50,1.15],[556.00,1.15],[556.50,1.15],[557.00,1.15],[557.5 ⌋ ,→ 0,1.15],[558.00,1.15],[558.50,1.15],[559.00,1.15],[559.50,1.1 ⌋ ,→ 5],[560.00,1.15],[560.50,1.15],[561.00,1.15],[561.50,1.15],[5 ⌋ 62.00,1.15],[562.50,1.15],[563.00,1.15],[563.50,1.15],[564.00 ⌋ ,→ ,→ ,1.15],[564.50,1.15],[565.00,1.15],[565.50,1.15],[566.00,1.15 ⌋ ,→ ],[566.50,1.15],[567.00,1.15],[567.50,1.15],[568.00,1.15],[56 ⌋ ,→ 8.50,1.15],[569.00,1.15],[569.50,1.15],[570.00,1.15],[570.50, ⌋ ,→ 1.15],[571.00,1.15],[571.50,1.15],[572.00,1.15],[572.50,1.15] ⌋ ,→ ,[573.00,1.15],[573.50,1.15],[574.00,1.15],[574.50,1.15],[575 ⌋ .00,1.15],[575.50,1.15],[576.00,1.15],[576.50,1.15],[577.00,1 ⌋ ,→ ,→ .15],[577.50,1.15],[578.00,1.15],[578.50,1.15],[579.00,1.15], ⌋ ,→ [579.50,1.15],[580.00,1.15],[580.50,1.15],[581.00,1.15],[581. ⌋ ,→ 50,1.15],[582.00,1.15],[582.50,1.15],[583.00,1.15],[583.50,1. ⌋ ,→ 15],[584.00,1.15],[584.50,1.15],[585.00,1.15],[585.50,1.15],[ ⌋ ,→ 586.00,1.15],[586.50,1.15],[587.00,1.15],[587.50,1.15],[588.0 ⌋ 0,1.15],[588.50,1.15],[589.00,1.15],[589.50,1.15],[590.00,1.1 ⌋ ,→ ,→ 5],[590.50,1.15],[591.00,1.15],[591.50,1.15],[592.00,1.15],[5 ⌋ ,→ 92.50,1.15],[593.00,1.15],[593.50,1.15],[594.00,1.15],[594.50 ⌋ ,→ ,1.15],[595.00,1.15],[595.50,1.15],[596.00,1.15],[596.50,1.15 ⌋ ,→ ],[597.00,1.15],[597.50,1.15],[598.00,1.15],[598.50,1.15],[59 ⌋ ,→ 9.00,1.15],[599.50,1.15],[600.00,1.15]</line>\n

Appendix

25

<line 2>[530.00,0.00],[530.50,0.02],[531.00,0.05],[531.50,0.09],[ ⌋ ,→ 532.00,0.14],[532.50,0.21],[533.00,0.30],[533.50,0.40],[534.0 ⌋ ,→ 0,0.51],[534.50,0.62],[535.00,0.73],[535.50,0.83],[536.00,0.9 ⌋ ,→ 2],[536.50,1.00],[537.00,1.06],[537.50,1.10],[538.00,1.12],[5 ⌋ ,→ 38.50,1.13],[539.00,1.13],[539.50,1.12],[540.00,1.10],[540.50 ⌋ ,→ ,1.07],[541.00,1.03],[541.50,0.98],[542.00,0.92],[542.50,0.85 ⌋ ,→ ],[543.00,0.78],[543.50,0.71],[544.00,0.64],[544.50,0.57],[54 ⌋ ,→ 5.00,0.51],[545.50,0.45],[546.00,0.40],[546.50,0.35],[547.00, ⌋ ,→ 0.32],[547.50,0.29],[548.00,0.27],[548.50,0.25],[549.00,0.23] ⌋ ,→ ,[549.50,0.22],[550.00,0.20],[550.50,0.19],[551.00,0.18],[551 ⌋ ,→ .50,0.17],[552.00,0.16],[552.50,0.16],[553.00,0.15],[553.50,0 ⌋ ,→ .15],[554.00,0.15],[554.50,0.15],[555.00,0.15],[555.50,0.15], ⌋ ,→ [556.00,0.15],[556.50,0.15],[557.00,0.15],[557.50,0.15],[558. ⌋ ,→ 00,0.15],[558.50,0.15],[559.00,0.15],[559.50,0.15],[560.00,0. ⌋ 15],[560.50,0.15],[561.00,0.15],[561.50,0.15],[562.00,0.15],[ ⌋ ,→ 562.50,0.15],[563.00,0.15],[563.50,0.15],[564.00,0.15],[564.5 ⌋ ,→ 0,0.15],[565.00,0.15],[565.50,0.15],[566.00,0.15],[566.50,0.1 ⌋ ,→ 5],[567.00,0.15],[567.50,0.15],[568.00,0.15],[568.50,0.15],[5 ⌋ ,→ 69.00,0.15],[569.50,0.15],[570.00,0.15],[570.50,0.15],[571.00 ⌋ ,→ ,0.15],[571.50,0.15],[572.00,0.15],[572.50,0.15],[573.00,0.15 ⌋ ,→ ],[573.50,0.15],[574.00,0.15],[574.50,0.15],[575.00,0.15],[57 ⌋ ,→ 5.50,0.15],[576.00,0.15],[576.50,0.15],[577.00,0.15],[577.50, ⌋ ,→ 0.15],[578.00,0.15],[578.50,0.15],[579.00,0.15],[579.50,0.15] ⌋ ,→ ,[580.00,0.15],[580.50,0.15],[581.00,0.15],[581.50,0.15],[582 ⌋ ,→ .00,0.15],[582.50,0.15],[583.00,0.15],[583.50,0.15],[584.00,0 ⌋ ,→ .15],[584.50,0.15],[585.00,0.15],[585.50,0.15],[586.00,0.15], ⌋ ,→ [586.50,0.15],[587.00,0.15],[587.50,0.15],[588.00,0.15],[588. ⌋ ,→ 50,0.15],[589.00,0.15],[589.50,0.15],[590.00,0.15],[590.50,0. ⌋ ,→ 15],[591.00,0.15],[591.50,0.15],[592.00,0.15],[592.50,0.15],[ ⌋ ,→ 593.00,0.15],[593.50,0.15],[594.00,0.15],[594.50,0.15],[595.0 ⌋ ,→ 0,0.15],[595.50,0.15],[596.00,0.15],[596.50,0.15],[597.00,0.1 ⌋ ,→ 5],[597.50,0.15],[598.00,0.15],[598.50,0.15],[599.00,0.15],[5 ⌋ ,→ 99.50,0.15],[600.00,0.15]</line>\n<line ,→ 3>[530.00,0.00],[530.50,0.01],[531.00,0.03],[531.50,0.06],[53 ⌋ ,→ 2.00,0.09],[532.50,0.14],[533.00,0.19],[533.50,0.25],[534.00, ⌋ ,→ 0.31],[534.50,0.37],[535.00,0.43],[535.50,0.48],[536.00,0.53] ⌋ ,→ ,[536.50,0.57],[537.00,0.60],[537.50,0.63],[538.00,0.65],[538 ⌋ ,→ .50,0.66],[539.00,0.67],[539.50,0.67],[540.00,0.66],[540.50,0 ⌋ ,→ .65],[541.00,0.64],[541.50,0.62],[542.00,0.60],[542.50,0.57], ⌋ ,→ [543.00,0.54],[543.50,0.50],[544.00,0.46],[544.50,0.41],[545. ⌋ ,→ 00,0.37],[545.50,0.32],[546.00,0.28],[546.50,0.24],[547.00,0. ⌋ ,→ 21],[547.50,0.18],[548.00,0.16],[548.50,0.14],[549.00,0.12],[ ⌋ ,→ 549.50,0.11],[550.00,0.10],[550.50,0.09],[551.00,0.08],[551.5 ⌋ ,→ 0,0.08],[552.00,0.07],[552.50,0.07],[553.00,0.07],[553.50,0.0 ⌋ ,→ 7],[554.00,0.07],[554.50,0.07],[555.00,0.07],[555.50,0.07],[5 ⌋ ,→ 56.00,0.07],[556.50,0.07],[557.00,0.07],[557.50,0.07],[558.00 ⌋ ,→ ,0.07],[558.5"\ ,→

26

2

Qualitative analysis of benchmarks

Table 1 presents a qualitative comparison between SpecVQA and existing visual question answering benchmarks. Compared with prior datasets that mainly focus on general charts or synthetic plots, SpecVQA specifically targets spectral figures from scientific literature and introduces underlying scientific tasks. Table 1: Comparison of SpecVQA with existing visual question answering benchmarks. Benchmark

Domain

Scale

QA Types

Answer Underlying

Comparison, color Binary matching Counting, value DVQA Bar charts 300K / 3.4M Num/Text lookup Data extraction, PlotQA Scientific plots 224K/28.9M Numeric comparison Reasoning, ChartQA Chart figures 21.9K/32.7K Num/Text aggregation Scientific Scientific SciVQA ∼10K/∼50K Text diagrams reasoning MultiEXAMS-V Exam figures ∼24K/∼100K Knowledge QA choice L0: Descriptive; SpecVQA Spectral figures 620/3100 Num/Text L1: Reasoning FigureQA

3

Synthetic plots

180K/ 2.3M

× × × × × × ✓

Labeling Guidelines

To ensure the scientific validity and representativeness of the benchmark, a PhD team of domain experts manually curated a subset of 620 spectral figures from 20k candidates collected from peer-reviewed journals and open-access scientific databases. The selection process was guided by six key criteria: Spectra Type, Image Structure, Text Completeness, Subplot Correlation, Sample Diversity and Resolution. For each figure, domain experts carefully designed 5 Question-Answer(QA) pairs. We expect the questions and answers to reflect issues that are genuinely of interest in their research and pose challenges to MLLMs, rather than trivial or nonsensical visual QAs. Therefore, these pairs were further refined through multiple rounds of rigorous review and revision to ensure both clarity and scientific accuracy. The final 3,100 QA-pairs are classified into two critical categoriesbased on the required cognitive effort. We provided both Chinese version and English version to test the scientific performance of models in different languages. Category 1 (L0): Descriptive Question

Appendix

27

This category focuses on the ability to directly understand visual information, including: – Information extraction: extracting titles, labels, legends and x/y-axis information. – Value localization: locating maximum or minimum points (e.g., strongest peaks) and peak location or range. – Pattern recognition: identifying entities that meet specific conditions. – Layout understanding: analyzing multi-panel subplots. – Classification: classifying features (e.g., peak shapes). Category 2 (L1): Reasoning Question This category emphasizes the ability to analyze and reason based on image content, including: – Comparison: comparing multiple entities and drawing conclusions. – Counting: determining the number of elements that satisfy certain conditions. – Calculation: performing computations on data presented in the figure. – Trend analysis: predicting changes in peak shapes or trends. – Causal analysis: analyzing and understanding the scientific problems reflected in the figure. The following is a complete annotation guide for industry experts: For each image, we pre-generated 5 QA pairs using large models such as GPT. All QA pairs are open-ended, and the answers are as concise as possible; that is, the answer may be a number or a phrase, and sentences should be avoided unless absolutely necessary. For each image, the order of work for each expert is as follows: – Determine the image itself: Analyze whether the current image has analytical value. If the result is "meaningful," proceed to the next step; if the result is "meaningless," clear all 5 QA pairs for the current image (keeping the original image), indicating that the current image is meaningless. This step is to prevent the inclusion of some worthless or low-value images that contaminate the data. A meaningful spectrum should include: horizontal and vertical axes and their corresponding units, necessary legends or symbols, relatively complete peaks/signals, and a clear image. – Evaluate each of the 5 QA pairs individually: First, determine if the current Q’s question aligns with the current major and the current image. If any mismatch is found, modify the Q itself. If partial modifications still do not meet the requirements, delete the current Q and rewrite it. Finally, determine if the current A’s answer satisfies the issues just corrected. If not, modify or rewrite the current A until it does. Tips

28

– Due to the limitations of Uniparser’s parsing capabilities, some images may have incorrect or incomplete slices. Encountering such issues is essentially meaningless. – To reflect the capabilities of large-scale models in real-world applications as objectively as possible, we encourage experts to design questions that are more closely aligned with actual research needs. Any questions and answers that can be derived from the graphs are welcome and can be documented; there’s no need to strictly adhere to MLLMs settings, as MLLMs current research capabilities regarding spectra are limited. We encourage industry experts to take initiative and design appropriate assessment criteria. Questions that are particularly challenging for current large-scale models are especially welcome. – If you encounter any other issues with the annotations, please feel free to contact us. Thank you for your valuable questions and suggestions.

4

Prompt Examples

4.1

Generate QA pairs

We use the following prompt template for question answering. Prompt: """You will perform a Vision Question Answering (VQA) task based on a scientific literature image. I will provide you with an image and a question related to it. Requirements: - Answer **only based on information directly visible in the image**, without relying on any prior knowledge or assumptions. - Do **not infer, reason, or speculate**. Only describe content that can be directly observed. - The answer must be **very concise**, preferably a **number, word, or short phrase**, and **not a full sentence**. - If the information is not visible in the image, answer: **Not visible**. Question: {QUESTION} Output **only the answer**, and do not include any other content: """

Appendix

4.2

29

Evaluate model performance on (VQA) tasks

This benchmark evaluates model performance on Visual Question Answering (VQA) tasks. Following the ChartVLM, GPT-o4-mini serves solely as a judge to score the model’s predictions against the ground truth. A predefined error tolerance of 5 percentage points is applied: if the error falls within this range, the answer is considered correct; otherwise, it is marked incorrect. Accuracy is then computed based on these judgments. 1 Prompt: """Given multiple question-answer pairs and the corresponding predictions, evaluate the correctness of predictions. The output should be only 'True' or 'False'. Note that if the groundtruth answer is a numeric value with/without the unit, impose 5 percentage error tolerance to the answer, e.g., the answer of 95 is marked as correct when groundtruth value is 100 million. User: <question> What was the incremental increase in revenue from 2020 to 2021? <groundtruth answer> 5 million $ <answer> 20 </s> A: False User: <question> What percentage of government spending was allocated to infrastructure in 2020? <groundtruth answer> 10 percentage <answer> 14-4=10 </s> A: True User: <question> What is the total production of Wind Energy in the four months from January to April 2021? <groundtruth answer> 2300 MW <answer> The total production of Wind Energy in the four months from January to April 2021 is 2450 MW. A: True User: <question> What is the total of manufactured goods for UK and Germany combined? <groundtruth answer> 5 <answer> Five A: True User: <question> {QUESTION} <groundtruth answer> {GROUND TRUTH} <answer> {PREDICTION} </s> AI: """ 4.3

Extract data points in the underlying task

For large, untuned models, our prompts are as follows. 1

https://github.com/Alpha-Innovator/ChartVLM/blob/main/eval/metric/gpt_acc.py

30

Prompt: """You will be given a composite scientific figure and a question. Your task is to identify the specific subplot referenced in the question and extract the underlying data series from that subplot. For every line in the selected subplot, output its extracted data points in the following strict format: <subplot {{SUBPLOT_NAME}}> <line 1>[x1,y1],[x2,y2],...[xn,yn]</line> <line 2>[x1,y1],[x2,y2],...[xn,yn]</line> ... </subplot> Requirements: - Output must follow the exact format above with no extra text. - {{SUBPLOT_NAME}} will be a letter (A, B, C, ...) provided in the question. - Each data series (line) must be enclosed in <line i></line>. - Each (x,y) pair must be enclosed in brackets [x,y] and values should keep two decimal places. - Do not add explanations, summaries, or comments outside the format. Example Output (for illustration) <subplot A> <line 1>[0.00,1.25],[1.00,2.50],[2.00,3.75]</line> <line 2>[0.00,0.80],[1.00,1.60],[2.00,2.40]</line> </subplot> Here is your question: {QUESTION} """ For our two fine-tuned models, namely the Qwen3-VL-4B model with and without a sampling strategy, we can directly input the Question. Here are examples of the Question and Answer for our Underlying task. Prompt: {'conversations': [{'from': 'human',

Appendix

31

'value': '<image>Underlying data for subplot B:'}, {'from': 'gpt', 'value': '<subplot B><line 0>[393.00,0.12],[393.36,0.14]...[41 ⌋ 4.63,0.00],[415.59,0.00]</line><line 2>[393.00,0.12],[393.36,0.14]...[414.63,0.00],[415.59,0.00]</l ⌋ ine></subplot>'}], 'images': ['data20/199999.png']}

5

SFT parameters

### method stage: sft do_train: true finetuning_type: full freeze_vision_tower: true freeze_multi_modal_projector: true freeze_language_model: false deepspeed: examples/deepspeed/ds_z3_config.json template: qwen3_vl_nothink cutoff_len: 16384 ### train per_device_train_batch_size: 2 gradient_accumulation_steps: 4 learning_rate: 1.0e-5 num_train_epochs: 1.0 lr_scheduler_type: cosine warmup_ratio: 0.1 bf16: true ddp_timeout: 180000000 resume_from_checkpoint: null flash_attn: fa2 enable_liger_kernel: True