SciFigQual-Bench: A Benchmark for Scientific Figure Quality Assessment with Full-Manuscript Context Zihan Deng1†∗† , Chuanzhi Xu2†∗ , Huiqi Liang2 , Haoyang Li2 , Xiaozhen Zhong3 , Lequan Yu1 1
The University of Hong Kong 2 The University of Sydney 3 University of Electronic Science and Technology of China Project Page Scientific Figure Evaluation
arXiv:2607.27084v1 [cs.CV] 29 Jul 2026
Abstract Scientific images are the core elements of presenting experimental conclusions, elaborating system architecture, and supporting comparative arguments in scientific papers. However, existing image quality assessment (IQA) are predominantly designed for natural photographs or AI-generated content, which cannot be directly applied to scientific papers. The few existing studies on scholarly charts remain confined to visualsurface comparisons, failing to verify caption alignment, citation relevance, or visual misleadingness. To address this, we propose SciFigQual-Bench, a full-text contextual benchmark that evaluates scientific image across five dimensions (clarity, layout, caption fit, context relevance, and misleading risk). The data covers the top computer-science conferences from 2020 to 2025, 6,308 images were independently scored by multiple domain experts in five dimensions and aggregated into the gold-standard annotations. Unlike previous scientific figure benchmarks, our dataset binds each image to its caption, citing sentence, and manuscript context. To enable automated evaluation on this benchmark, we designed a staged cross-modal evaluation SFQ-Agent to achieve auditable and refined scoring through the collection and fusion of modal evidence. Multiple mainstream large models were evaluated on the test subset eval1200, and the SFQ-Agent (F3) equipped with GPT-5.6Sol achieved the lowest overall average absolute error (0.418) and the highest ±1-point consistency rate (93.4%), consistently outperforming both direct evaluation and auxiliary (Sidecar) visual language model evaluation schemes. Project page: https://frankdengai.github.io/SciFigQual-Bench/ (code: https://github.com/FrankDengAI/SciFigQual-Bench).
1
Introduction
Natural images arise from light, shadows, and objects in the real world, and their evaluation criteria mostly remain at the level of human visual perception. In Image Quality Assessment (IQA), for example, a landscape photo is often evaluated from the aspects of natural color, clarity, aesthetic perception, etc. There is no "right or wrong" standard at the content level (Li et al. 2025). These types of images are often unrelated to language and text, and can be evaluated solely based on the image itself (Chen et al. 2024). However, scientific images are naturally bound to the complete academic text context, and they must be combined with descriptions, specific data, or conclusions in order to have evaluative value (Tufte 2001; Cleveland and McGill 1984). ∗ †
Equal contribution. Email: [email protected], [email protected].
CTX ❌ Body claim conflicts with figure
CC ❌ Caption-figure mismatch Caption: ❌ Red Line=The highest
PDF Paper
Natural IQA Reference Image
MR ❌ Misleading visual design
Paragraph: ❌ As shown in Fig. 3, Structure significantly outperforms Context...
Paragraph: ❌ no error bars /no baseline
Real paper figures often fail under isolated evaluation
Isolated Schematics
AIGC Alignment Prompt
Image
Figure Understanding
GenAI
PSNR / SSIM / NIQE (Pixel-level/Perceptual Metrics)
CLIP/BLIP VGG-16/ResNet (Feature(Text-to-Image Similarity) based Classification)
QA Accuracy / Reasoning Score
SciFigQual-Bench: full-manuscript, evidence-grounded evaluation Experimental Results
Staged Cross-modal Scoring
VC
Visual Clarity
SL
Structure Layout
CC
Caption Consistency
SFQ-Agent
MR
Misleading Risk
evidence-grounded 5D judging
CTX
Context Consistency
T1: 4.2 As shown in Fig. 3, method A significantly outperforms the baseline... T2: 5.1 Ablation results shown in Fig. 3...
Figure 1: From isolated-figure evaluation to full-manuscript context binding. Prior paradigms evaluate figures without manuscript evidence, whereas SciFigQual-Bench grounds assessment in figure-relevant text and scores five orthogonal dimensions on a unified 1–10 scale.
If there are errors in the figures in a scientific paper that are closely related to the data or conclusions, such as axis truncation, missing baseline data, or inconsistencies between the data in the figures and the description in the text, visual inspection at the pixel level alone cannot identify such defects. In scientific paper peer review, quality assessment of scientific paper is therefore inherently tri-modal: it depends on what is visible in image, what is claimed in caption, and what the manuscript asserts in citing paragraphs. However, existing paradigms for natural IQA and AIGC alignment operate on a single visual modality, inherently decoupling the image from both its caption and the citing manuscript context. As illustrated in Figure 1, four mainstream paradigms highlight this gap. Natural IQA (Mittal, Moorthy, and Bovik 2012; Mittal, Soundararajan, and Bovik 2013; Prashnani et al. 2018; Wang, Chan, and Loy 2023) evaluates perceptual image quality relying on low-level statistics. AIGC alignment (Radford et al. 2021; Xu et al. 2023; Wu et al. 2023; Hu
Comparison dimension Real figures from published CS papers Source PDF + section binding Full-text citing paragraphs (index-driven) Evaluation dimensions (multi-dim. rubric) End-to-end Acquire→Annotate pipeline Learnable multi-dimensional regressor Auditable scoring rationale Multi-venue × multi-year (CS top-tier) # Images (test bench.) # Text items # Papers (crawl scope)
SIQA SCIEval VisJudge SPUR GENFIG1
Ours
✗ ✗ ✗ ✓ ✗ △ △ △
✗ ✗ ✗ ✓ △ ✓ △ ✗
△ ✗ ✗ ✓ ✗ ✗ △ △
△ △ △ △ ✗ ✗ ✗ △
✗ △ ✗ ✓ △ ✗ △ △
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
2,240 ≈2.2k –
6,000 ≈6.0k –
3,090 ≈3.1k –
1,084 ≈4.3k ≈5k
584 ≈584 ≈584
7,609 ≈355k ≈63.6k
Table 1: Capability and scale comparison against related benchmarks. Symbols: ✓=fully supported; △=partial; ✗=not targeted. et al. 2023; Kirstain et al. 2023) measures text-image similarity for generated content. Schematic understanding (Masry et al. 2022; Methani et al. 2020; Kafle et al. 2018; Liu et al. 2023; Masry et al. 2023) uses the model to implement image classification recognition and visual question answering functions, and the model’s judgment results cannot be directly associated with the quoted paragraphs in the original PDF document. Figure reasoning QA (Siegel et al. 2016; Mathew, Karatzas, and Jawahar 2021) tests whether a model can read a figure but does not offer a rubric for holistic quality or misleading risk. In each case, evaluation proceeds on a detached visual input, decoupled from captions, citing paragraphs, and full-text context. In this paper, we propose SciFigQual-Bench, an evidencegrounded staged judging framework for evaluating humanpublished scientific images under authentic full-manuscript context. We binding each figure image with its caption and the citing paragraph set from the source PDF to form complete experimental context (CTX). Instead of adopting a single end-to-end vision-language model (VLM) prompt, we employ an evidence-grounded staged judging process: first collect multi-modal evidence from each component, then perform cross-modal fusion, and finally output traceable multidimensional scores covering Visual Clarity (VC), Structure Layout (SL), Caption Consistency (CC), Misleading Risk (MR), and Context Consistency (CTX). Unlike faithfulness benchmarks for AI-generated scientific images (Ye et al. 2026; Li et al. 2024a; Hsu, Giles, and Huang 2021), our task evaluates human-published figures in a real manuscript context, forming a complementary evaluation objective. Our contributions can be summarized as follows: • First, we release SciFigQual-Bench: 7,609 curated figures from 1,144 qualified papers, 6,308 expert-aggregated gold instances, and index-driven context records spanning top-tier conference papers (2020–2025). • Second, we propose SFQ-Agent, a staged cross-modal judge that operationalizes full-manuscript figure evaluation by binding figure, caption, and citing text as tri-modal evidence, fusing vision-language for text-grounded dimensions, and aggregating deterministic rubric-aligned scores to ensure each dimension is traceable to structured evidence rather than an opaque VLM verdict. • Third, on the fixed public test subset eval1200
(n=1,200), we benchmark a broad suite of state-of-the-art vision–language models against human gold labels, establishing baselines and showing that SFQ-Agent consistently outperforms direct and auxiliary judging schemes while revealing caption consistency and context consistency as the dominant human–model disagreement axes.
2
Related Work
Existing research that can be applied in scientific figure quality assessment and multimodal figure-text evaluation falls into four major categories: perceptual image quality evaluation, chart comprehension, text-to-figure generation and multimodal corpus construction, and LLM-based automated evaluation frameworks. Perceptual IQA and isolated image quality. Classical noreference IQA (BRISQUE, NIQE) and learned perceptual metrics (PieAPP, CLIP-IQA) estimate quality from low-level statistics or deep features (Mittal, Moorthy, and Bovik 2012; Mittal, Soundararajan, and Bovik 2013); SIQA defines a four-dimensional rubric for schematic scientific figures and trains learned regressors on expert ratings, yet its instances are not drawn from published CS PDFs and carry no indexresolved citing paragraphs (Li et al. 2026). VisJudge-Bench employs MLLM judges for visualization aesthetics and quality on chart and diagram crops (Xie et al. 2026), but likewise omits source-PDF binding and manuscript-level CC/CTX. However, these approaches are confined to visual-only quality assessment on isolated crops, and none grounds its scores on the full manuscript context or caption consistency. Chart and scientific figure understanding. ChartQA and SciFIBench benchmark chart reasoning and multiple-choice scientific figure interpretation (Masry et al. 2022; Roberts et al. 2024). SPUR extends to biomedical experimental images with perception–understanding–reasoning QA and partial figure–text linkage (Ding et al. 2026), yet targets VQA accuracy rather than a calibrated multi-dimensional quality rubric and covers neither CS top-tier venues nor index-driven citing paragraphs at our scale. These benchmarks focus on the analysis of scientific images, but lack a multidimensional quality assessment system: they cannot measure the matching degree between chart titles and images, the consistency between charts and the context of the text, nor can they quantify the risk of misleading information in computer conference official charts.
Generation faithfulness Generation faithfulness. SCIEval trains and benchmarks CLIP/LMM modules on multidimensional faithfulness of generated scientific images, which covering relevance, accuracy, and interpretability in text-to-image and captioning settings (Ye et al. 2026). GENFIG1 focuses on visual summaries automatically produced for scientific papers, challenging VLMs to judge whether such generated overviews faithfully reflect paper content (Guan et al. 2026). At only hundreds of instances, it targets quality control of newly synthesized academic illustrations rather than retrospective assessment of human-authored conference figures. These resources support generation or retrieval, but extraction and captioning alone do not certify figure quality in peer-review terms. LMM-as-judge. Rubric-conditioned VLM evaluators (Prometheus-Vision, VIEScore) enable fine-grained multimodal scoring (Lee et al. 2024; Ku et al. 2024); evidencegrounded NLP cautions against plausible but unfaithful rationales (Jacovi and Goldberg 2020). Monolithic judges conflate visual perception with textual verification and hallucinate evidence when scoring CC, CTX, and MR jointly. SciFigQual-Bench links each figure to its citing paragraphs resolved from the PDF, gathers modality-specific evidence stepwise, and fuses them only after these stages. It is designed for real published figures, not isolated crops, synthetic substitutes, or multiple-choice interpretation tasks. Table 1 compares our benchmark with the five closest 2024–2026 efforts in terms of capability and data scale.
3
SciFigQual-Bench
This section first provides a formal definition of benchmark instances and a five dimensional scoring scale, and then introduces the data construction process and the SFQ-Agent phased scoring framework.
3.1
STEP 1 : Multi-Venue Paper Corpus Collection OpenReview
ACL Anthology
PMLR
63,634 papers 4 venues ·2020-2025 ACL, EMNLP, ICML, NeurIPS
Context Consistency
Caption Consistency
The results presented in Fig. 3 further validate the effectiveness of our approach...
Score
3
7,609 Clean figs
STEP 3 : Index-Driven Context Binding
As shown in Fig.3, our method significantly out performs the base line...
STEP 5: Benchmark Release SciFigQualBench
Structure & Layout
clean figure
raw noisy crop
Fig3. Overall pipeline of the method.
Visual Clarity
2
Five-stage CaptionPDF → quality curation anchored Structured (S1–S5) layout parsing figure indexing
Scientific/CS Papers
STEP 4: EvidenceGrounded 5D Scoring
1
Marker parser 300 DPI
source PDFs
Preprocessing& index
Misleading Risk
STEP 2 : Structure-Aware Figure Extraction & Curation
Bind text → figure, not figure alone
No Isolated figure Human expert annotation
Benchmark Instances
Diagnostic Panel
Figure 2: SciFigQual-Bench construction pipeline: corpus collection, structure-aware extraction, context binding, fivedimensional rubric, and expert validation. vectors si , ŝi ∈ [1, 10]|D| . The gated overall label is yi = P
1
X
d∈D gi (d) d∈D
gi (d) si (d),
(1)
with yi undefined only when both c and T are absent (the instance is excluded). If c=∅, CC is marked null (gi (CC)=0); if T =∅, CTX is null; instances lacking both are excluded from scoring. This L1 evidence gating prevents penalizing missing metadata as low quality (Krippendorff 2011) and mirrors annotation practice in SciCap+ (Yang et al. 2024). MR may still be scored from visual hazards when text is partial.
Task Formulation and Rubric
The benchmark instance is a tuple (I, c, T , m) where I is the figure crop, c the caption (possibly empty), T citing paragraphs (possibly empty), and m lightweight metadata (venue, year, figure index, section tags). Instances are always anchored to a source PDF so that T is resolved by figure index rather than layout heuristics alone. Human experts and SFQAgent assign scores on five orthogonal dimensions, each on [1, 10]: Visual Clarity (VC), legibility of text, marks, and encoding at publication scale; Structure & Layout (SL), composition, panel organization, and chartjunk (Tufte 2001); Caption Consistency (CC), alignment between c and visible content, including subpanels and trends; Context Consistency (CTX), alignment between evidentiary claims in T and what I supports; Misleading Risk (MR), likelihood of reader misinterpretation (higher score = lower risk), including truncated axes or undisclosed baselines (Rückert, Bloch, and Friedrich 2026). Overall quality is the mean over available dimensions, enabling fair comparison when captions or citing text are missing. Let D = {VC, SL, CC, CTX, MR} and gi (d) ∈ {0, 1} indicate whether dimension d is evaluable for instance i under L1 gating. Human and model outputs are
3.2
Benchmark Construction
Figure 2 summarizes our pipeline over CS papers from ACL, EMNLP, ICML, and NeurIPS (2020–2025). We formalize the released benchmark as B = {(Ij , cj , Tj , mj )}N j=1 with N =7,609 figures from P =1,144 qualified papers, of which Nrat =6,308 carry human five-dimensional scores.The remaining figures retain image–caption–context bindings for protocol development and future labeling, but are excluded from gold-based evaluation on eval1200. Each module below is deterministic given source PDFs and fixed preprocessing seeds. Step 1: Corpus acquisition. We acquire the corpus from multiple venues: we crawl PDFs and bibliographic metadata from OpenReview (Idahl and Ahmadi 2025), ACL Anthology (Gildea et al. 2018), and PMLR, tagging each file with (venue, year , paper _id ). Next, we normalize PDFs by linearizing, stripping publisher watermarks where possible, and hashing by content fingerprint h(p) = SHA256(bytes(p)) to remove exact duplicates. Finally, we apply venue–year indexing, discarding records that fail year filters (outside 2020–2025) or missing parseable metadata. The surviving
3.3
raw corpus is 0 C0 = {pk }K k=1 ,
K0 = 62,694,
(2)
subsuming the scale of DocFigure and SciFIBench sources (Jobin, Mondal, and Jawahar 2019; Roberts et al. 2024; Li et al. 2024b) but curated for quality evaluation rather than MCQ or type classification. Step 2: Structure-aware figure extraction. We rasterize each PDF at 300 DPI and parse its layout with Marker, obtaining text blocks, figure bounding boxes, and caption candidates. For each figure, we rank candidate captions using spatial overlap, vertical distance, and a figure-caption cue (Song et al. 2025), selecting the best match above a threshold. The extracted pairs then undergo a five-stage curation pipeline that removes duplicates and non-figures, filters anomalous crops, recovers clipped panels, flags borderline cases, and discards irrecoverable layouts. This yields a clean set of figure–caption pairs from qualified papers. Step 3: Index-driven context binding. We detect figure references in the body text using PyMuPDF (Artifex Software, Inc. 2026) by matching index patterns anchored to figure indices. For each matched reference, we collect its hosting paragraph and merge multi-sentence windows within Sthe same section, yielding per-figure context sets Tk = τ ∈match(k) u(τ ), each capped at Lmax tokens. This corpus-wide process produces approximately 355,000 citingparagraph records—resolved directly from the PDF index rather than relying on abstract or introduction heuristics, distinguishing our approach from caption-only corpora (Wang et al. 2026; Li et al. 2024b). Finally, we apply a quality filter that down-weights paragraphs lacking explicit figure-specific claims and retains those with comparative or trend language for subsequent CTX annotation. Step 4: Five-dimensional human annotation. We first calibrated domain experts with a 50-figure pilot using the rubric in appendix A.1. During formal annotation, each instance received scores under Eq. (1) with L1 gating—CC and CTX were hidden whenever the necessary evidence was absent. Following faithful-NLP guidelines (Jacovi and Goldberg 2020), annotators had to justify every score with specific visible marks, caption phrases, or citing sentences; any rationale lacking evidence was sent back for revision. To ensure quality, we performed dual annotation on an ACL 2025 holdout and measured inter-rater agreement; disagreements exceeding two points triggered adjudication. The final set comprises 6,308 rated figures with venue-balanced splits. Step 5: Release and evaluation packaging. We release all rated instances, context bundles T , and model-score outputs as a unified JSONL dataset. To support fair and reproducible cross-protocol comparisons, we construct a fixed public test split, eval1200, containing 1,200 stratified instances—balanced by venue and figure type. For evaluation, we implement three judge protocols (Direct, Sidecar, and SFQ-Agent), all using the same input fields (I, c, T , m) and outputting scores in the same five-dimensional schema; any performance gap thus reflects the judgment strategy itself, not differences in data representation.
Dataset Statistics
Figure 4 summarizes the scale, diversity, and quality distribution of our dataset relative to prior scientific-figure benchmarks. (a) Radar coverage spans 18 research categories across four venues, covering broader CS topics than chartonly sets like CharXiv (Wang et al. 2024). (b) The funnel yields P =1,144 qualified papers and N =7,609 clean figures from K0 =62,694 raw PDFs (Eq. (2)); 6,308 are human-rated with balanced venue splits, surpassing SIQA-U (2,240) and SPUR (1,084) in comparable settings (Li et al. 2026; Ding et al. 2026). The overall yield η = N/K0 ≈ 12.1% reflects our strict curation standards. (c) Overall scores center near 8.05, indicating solid quality with sufficient variance for effective discrimination. (d) The largest share of rated figures comes from NLP (2,813), followed by ML (1,891) and CV (873). (e) Temporal coverage peaks in 2024–2025 with over 1,400 rated figures per year, supporting evaluation on modern plotting styles. (f) Mean dimension scores show CC (7.42) as the weakest axis versus SL (8.58), confirming that caption–figure mismatches—rather than low-level blur—are the dominant real-world defects, which motivates our manuscript-grounded CC and CTX evaluation.
3.4
SFQ-Agent
Based on the previous statistics, the main bottleneck is the consistency of captions and context, which makes it difficult for a single visual language model to perform end-to-end scoring. To this end, we propose SFQ Agent: under a unified five dimensional scale, collect visual and linguistic evidence in stages and perform cross modal fusion to achieve auditable and refined decision-making. SFQ-Agent (Figure 3) implements evidence-grounded judging in four stages, designed as an auditable alternative to monolithic LMM-asjudge pipelines (Zheng et al. 2023; Lee et al. 2024; Ku et al. 2024). Step 0: Input and gating. The system loads (I, c, T ); figures without caption and citing text are discarded; partial evidence triggers dimension hiding. Step 1: Vision evidence. We extract visual evidence from each figure via PaddleOCR-VL (a compact OCR model) and classical CV descriptors (blur, edge density, color histogram). The OCR module supplies text regions, confidence scores, and layout cues; we also derive side features such as text density, caption-token overlap, and numeric consistency. These are packaged as visual_facts together with axis units and legend notes to support VC and SL scoring. This parallels low-level perceptual auditing in prior work (Wu et al. 2024a; Rückert, Bloch, and Friedrich 2026), but our structured outputs are explicitly designed for downstream fusion. The language module never accesses pixels, ensuring clean modality separation. Step 2: Language evidence. To complement the vision track, an LLM reads only the caption, citing paragraphs, and metadata, without any pixel input. It extracts textual evidence, implied claims, and rule violations to support CC and CTX scoring. This strict separation prevents the LLM from generating plausible but unfaithful rationales, a documented risk in prior work (Jacovi and Goldberg 2020).
SFQ-Agent
Input Initialization Scientific/CS Papers
As shown in Fig.3, our method significantly out performs the base line...
LLM Text facts
Caption implied
SL
Axes units
Rule flags
Context implied
Intrinsic risk band
Metrics
Text MR Adjust MERGE
Ablation comparison
Cross-modal Judge
SL - Structure & Layout Evidence : panels, reading path
CC
MR -Misleading Risk ★
MR
CTX
Reasons
Evidence: axes/units/comparisons text claims
Hide CC (Null)
Hide CTX (Null)
CC -Caption Consistency
Skip figure
Runner
Evidence : caption text figure content
CTX -Context Consistency ★
Results Scores
Evidence: ALL citing paragraphs figure
Panels ref.
[I, C, T]
Evidence : pixels,axes,labels
Neither
Small chips [paper meta | fig map | avail dims]
Legend notes
VC -Visual Clarity
Has Citing paragraphs ? No
VLM
VC
Has caption? No
language Evidence Module
Visual facts
Both
Citing paragraphs T
Vision Evidence Module OCR /CV features
Fig3. Overall pipeline of the method. The results presented in Fig. 3 further validate the effectiveness of our approach...
Caption c
Image I
[I, C, T] OCR+CV
VLM Juddage Sidecar Judge
Misleading Risk (MR) ← fuse(visual risk level, text conflict severity)
Figure 3: SFQ-Agent scoring pipeline: L1 gating, parallel vision and language evidence modules, cross-modal judge, and deterministic Runner aggregation. Direct and Sidecar judges serve as ablations. Step 3: Cross-modal judge. Once both tracks have produced structured evidence, the cross-modal judge fuses them to score CC, CTX, and MR. Specifically, it aligns visual hazard indicators with textual cues and detects conflicts, such as a figure showing an upward trend while the text describes degradation. VC and SL are copied from Stage 1 rather than re-estimated from text summaries, so fluent language evidence cannot re-inflate visual scores. Step 4: Runner. Finally, a deterministic post-processor locks VC and SL to the vision module outputs, imposes rulebased CC/CTX/MR caps informed by visualization guidelines (Rückert, Bloch, and Friedrich 2026; Tufte 2001), and aggregates all available dimensions under L1 gating. This design prevents score inflation from unconstrained LLM arbitration and keeps each final score traceable to its originating evidence track. Frozen prompt contracts and the MR fusion mapping are reported in the supplementary appendix C. Judge protocols. We evaluate three judge protocols along a spectrum of increasing complexity. The Direct Judge serves as the simplest baseline, issuing a single VLM prompt over (I, c, T )—similar to zero-shot judging in Prometheus-Vision and Q-Align (Lee et al. 2024; Wu et al. 2024b). The Sidecar Judge augments this setup with PaddleOCR-VL and CV side features within the same single-pass call, testing whether additional visual cues alone bring gains. The SFQ-Agent implements our full staged pipeline, reusing the same feature bundle across three sequential calls (Stage 1 for vision, Stage 2 for language, Stage 3 for fusion). Together, these variants isolate two design choices (OCR-VL side signals versus explicit evidence staging), especially on CC and CTX, where monolithic judges most often conflate perception with text verification.
4 4.1
Experiments and Results
Experimental Setup
We evaluate on eval1200, a fixed subset of 1,200 humanrated figures stratified by venue (300 per conference) and figure type (plot, diagram, table, composite), with gold labels from Eq. (1) and the same L1 gating applied to model outputs (supplementary appendix B.1 “eval1200 Split Definition”). We test 11 VLM backends (Gemini3.5-Flash, GPT-5.6-Sol, Claude-Sonnet-5, Claude-Opus-4.8, Qwen-VL-Max, GLM-4.6V, Doubao-Seed-2.0-pro, Llama4-Maverick, Pixtral-Large, Nova-Pro, and InternVL3-78B) under three protocols that share inputs (I, c, T , m) and differ only in how evidence is staged. Reproducible run IDs and per-protocol call budgets are listed in the supplementary appendix B.2. Protocol ladder. Direct (1 call/figure) issues a single endto-end VLM prompt over (I, c, T ) and returns all evaluable dimensions jointly. Sidecar (1 call/figure) keeps that interface but injects PaddleOCR-VL and CV side features, testing whether auxiliary visual cues alone help without changing the decision topology. SFQ-Agent (3 calls/figure) follows §3.4: Stage 1 locks VC/SL from vision evidence; Stage 2 extracts caption/context facts without pixels; Stage 3 fuses the two evidence reports for CC, CTX, and MR; the Runner then re-applies visual-score ownership and score caps in code. Except for the heterogeneous F5 pairing (Qwen-VLMax vision + Qwen-Plus language), Agent runs use matched vision/language backends within a configuration. This ladder separates two effects that a monolithic judge conflates: denser OCR-side observation (Sidecar) versus explicit modality separation and fusion (Agent). Prompt templates for all three protocols appear in the supplementary appendix C. Alignment metrics. We measure agreement with gold overall scores using MAE, Within-1 (fraction within ±1), Spearman correlation (Spearman 1904), and signed bias (positive
（ b）
（a）
（c）
（ d）
（e）
（f）
Figure 4: Corpus overview of SciFigQual-Bench. From 62,694 PDFs we retain 7,609 figures (6,308 human-rated) across ACL/EMNLP/ICML/NeurIPS (2020–2025), with NLP/ML/CV as the dominant domains. Overall scores concentrate near 8.05; caption consistency is the weakest axis, motivating manuscript-grounded CC/CTX evaluation. = lenient). For CC, CTX, and MR we additionally report perdimension MAE on L1-evaluable samples only; CTX is further restricted to figures with citing text. These text-grounded errors are expected to exceed MR, because they require verifying claims against c and T rather than reading pixels alone. Unless noted in Table 2, metrics use the full eval1200 split; measurement coverage and any projected rows are documented in the supplementary appendix, Sec. B.3.
4.2
Protocol Ablation Results
Table 2 consolidates our main evaluation and protocol ablation on eval1200: 29 protocol–backend configurations spanning Direct (1 call), Sidecar (1 call + PaddleOCRVL/CV), and SFQ-Agent (3-call staged fusion). We ablate along two axes, judge protocol (Direct, Sidecar, Agent) and VLM backend, under identical inputs (I, c, T ) and L1 gating, so every row in the table is directly comparable. Direct baseline. Claude-Opus-4.8 leads Direct protocols (MAE 0.443, W-1 92.4%, SRCC 0.582), closely followed by GPT-5.6-Sol. Qwen-VL-Max achieves reasonable calibration (W-1 88.8%) but poor ordinal ranking (SRCC 0.416), driven by high CC/CTX errors (MAECC ≈ 0.99, MAECTX ≈ 1.04). Doubao-Seed-2.0-pro shows strong negative bias (−0.120), indicating systematic harshness, while GLM-4.6V lags with the largest errors across dimensions. Sidecar ablation. Adding PaddleOCR-VL and CV features uniformly reduces MAE across all backends, with the largest gain on GLM (∆MAE=0.044), suggesting that explicit text and layout cues particularly benefit weaker encoders on dense scientific plots. GPT-5.6-Sol also sees modest CC/CTX improvements (e.g., CC 0.922→0.912), confirming that Sidecar strengthens text-heavy visual evidence more than holistic reasoning.
SFQ-Agent ablation. Full staged judging monotonically improves all strong backends. GPT-5.6-Sol achieves the best overall result (MAE 0.418, W-1 93.4%, SRCC 0.598, bias +0.012) and lowest MR error (0.678), showing that deterministic fusion and evidence separation enhance both calibration and misleading-risk detection. Claude-Sonnet-5 attains the lowest CTX MAE (0.842) among Agent runs, suggesting language-centric backends excel at manuscript consistency even when overall MAE lags GPT-5.6-Sol. Conversely, Qwen Agent exhibits elevated MR MAE (0.938), indicating that pairing vision and language models does not guarantee cross-modal conflict resolution without explicit fusion.
4.3
Analysis and Discussion
Ablation analysis. The progressive improvement from Direct to Sidecar to Agent confirms that gains stem from protocol–evidence alignment, not model scale. These variants sequentially address three orthogonal failure modes of monolithic VLM judges: Sidecar fixes missed OCR details; Agent tackles perception-text conflation and score inflation via explicit visual-textual separation and fusion. Architecturally, Agent’s edge is not mere multiple calls. Stage 1 locks VC/SL to visual evidence, preventing fluent text from reinflating visual scores; Stage 2 extracts captions and citing claims without pixel access, blocking unfaithful visual rationales for CC/CTX; Stage 3 fuses the two evidence reports only for CC, CTX, and MR; finally, the Runner applies ownership and score caps in code. On matched backends, this yields near-zero bias and largest gains on text-grounded dimensions where Direct fails, not uniform polishing. Our design distinguishes from prior scientific-figure work: SIQA regressors lack index-
Protocol
Backend
MAE↓
W-1↑
SRCC↑
Bias → 0
CC MAE↓
CTX MAE↓
MR MAE↓
Direct
Gemini-3.5-Flash GPT-5.6-Sol Claude-Sonnet-5 Qwen-VL-Max GLM-4.6V Doubao-Seed-2.0-pro Llama-4-Maverick Pixtral-Large Nova-Pro Claude-Opus-4.8 InternVL3-78B
0.464 0.448 0.478 0.520 0.662 0.609 0.486 0.502 0.471 0.443 0.538
91.0% 92.0% 90.2% 88.8% 81.0% 83.2% 89.8% 89.2% 90.6% 92.4% 87.6%
0.532 0.565 0.518 0.416 0.438 0.570 0.512 0.498 0.528 0.582 0.445
0.032 0.022 0.038 0.065 0.102 -0.120 0.035 0.042 0.028 0.015 0.058
0.942 0.922 0.968 0.985 1.082 0.878 0.958 0.972 0.938 0.908 1.012
0.972 0.928 0.862 1.044 1.128 0.918 0.948 0.968 0.912 0.888 1.052
0.724 0.708 0.732 0.758 0.812 0.752 0.728 0.742 0.716 0.692 0.768
Sidecar
Gemini-3.5-Flash + OCR GPT-5.6-Sol + OCR Qwen-VL-Max + OCR GLM-4.6V + OCR Doubao-Seed-2.0-pro + OCR Nova-Pro + OCR Llama-4-Maverick + OCR Pixtral-Large + OCR InternVL3-78B + OCR
0.452 0.436 0.516 0.618 0.596 0.459 0.474 0.490 0.524
91.8% 92.6% 89.1% 83.8% 84.6% 91.2% 90.4% 89.6% 88.2%
0.546 0.578 0.470 0.472 0.584 0.538 0.524 0.508 0.458
0.026 0.018 0.050 0.088 -0.098 0.022 0.030 0.036 0.052
0.928 0.912 0.972 1.048 0.862 0.928 0.948 0.962 0.998
0.958 0.918 0.962 1.088 0.898 0.902 0.938 0.958 1.038
0.712 0.698 0.778 0.792 0.738 0.708 0.718 0.732 0.758
SFQ-Agent
Gemini-3.5-Flash Gemini-3.1-Pro GPT-5.6-Sol Claude-Sonnet-5 Qwen-VL-Max + Qwen-Plus GLM-4.6V Llama-4-Maverick Claude-Opus-4.8 Nova-Pro
0.440 0.428 0.418 0.456 0.511 0.574 0.462 0.424 0.446
92.6% 92.9% 93.4% 91.2% 89.3% 86.4% 91.0% 93.1% 92.2%
0.560 0.572 0.598 0.542 0.519 0.508 0.532 0.588 0.548
0.020 0.016 0.012 0.028 0.030 0.072 0.022 0.014 0.018
0.914 0.906 0.892 0.948 1.007 1.012 0.942 0.898 0.924
0.942 0.932 0.896 0.842 0.833 1.052 0.932 0.882 0.906
0.698 0.692 0.678 0.718 0.938 0.768 0.712 0.686 0.702
Table 2: Protocol–backend ablation against human ratings on eval1200 (n=1200). ↑ higher is better; ↓ lower is better. Bold rows: . best per protocol (lowest MAE). “+ OCR” in Sidecar rows denotes PaddleOCR-VL side features. resolved citations, missing caption–manuscript conflicts; ChartQA/SciFIBench probes discrete accuracy, not continuous [1,10] review quality. Our graded evidence-gated score aligns with annotation practice—hiding CC/CTX on partial evidence—and provides auditable rationales beyond MCQ accuracy. Backend behavior. Cross-protocol rankings reveal systematic heterogeneity. GPT-5.6-Sol is the most balanced, with near-zero bias under Agent. Qwen-VL-Max shows a dissociation between calibration (W-1 88.8%) and ranking (SRCC 0.416) under Direct; staging recovers ranking (SRCC 0.519) but leaves CC error intact (MAECC = 1.007), suggesting caption verification requires fundamentally better vision-language grounding than ordinal judgment. GLM4.6V benefits the most from protocol enrichment (∆MAE = 0.662 → 0.574), implying that pipeline design can compensate for weaker encoders. Doubao’s persistent negative bias (−0.120 to −0.097) suggests systematic harshness is baked into its training. Benchmark contributions. SciFigQual-Bench serves three diagnostic roles beyond leaderboard ranking. First, perdimension MAE localizes failure modes (legibility, caption fidelity, narrative alignment, misleading risk). Second,
the protocol ladder establishes judge architecture as a firstclass design choice; staging shifts MAE by 0.03–0.09 on matched backends, comparable to inter-VLM gaps under Direct. Third, persistent CC/CTX deficits define an open frontier: cross-modal verification over citing paragraphs that neither chart QA nor text QA communities have jointly addressed. SciFigQual-Bench is thus a stress test for tri-modal scientific literacy; success demands reviewer-like integration of visual, textual, and contextual evidence.
5
Conclusion
We introduce SciFigQual-Bench, a full-manuscript-context benchmark tailored for published CS figures, and SFQAgent, a staged protocol that synthesizes visual, caption, and textual evidence into auditable five-dimensional scores. On eval1200, SFQ-Agent (F3) achieves the lowest MAE (0.418) and the highest Within-1 rate (93.4%), consistently outperforming both Direct and Sidecar VLM judges. By anchoring each figure to its caption and citing paragraphs, our framework establishes a reproducible, evidence-grounded testbed that elevates AI-assisted figure inspection from isolated visual QA to context-aware scientific reasoning.
References Artifex Software, Inc. 2026. PyMuPDF Documentation. Software documentation. Accessed 2026-07-29. Chen, C.; Mo, J.; Hou, J.; Wu, H.; Liao, L.; Sun, W.; Yan, Q.; and Lin, W. 2024. TOPIQ: A Top-Down Approach From Semantics to Distortions for Image Quality Assessment. IEEE Transactions on Image Processing, 33: 2404–2418. Cleveland, W. S.; and McGill, R. 1984. Graphical Perception: Theory, Experimentation, and Application to the Development of Graphical Methods. Journal of the American Statistical Association, 79(387): 531–554. Cui, C.; Sun, T.; Liang, S.; Gao, T.; Zhang, Z.; Liu, J.; Wang, X.; Zhou, C.; Liu, H.; Lin, M.; Zhang, Y.; Zhang, Y.; Zheng, H.; Zhang, J.; Zhang, J.; Liu, Y.; Yu, D.; and Ma, Y. 2025. PaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Model. arXiv:2510.14528. Ding, J.; Tang, Z.; E, H.; Ji, M.; Liu, Y.; Tian, H.; Sun, H.; Sun, P.; Xu, Y.; Liu, Y.; Gao, H.; Xi, Z.; Jiang, R.; Zhao, P.; Li, R.; Li, Y.; Liu, J.; Yang, Z.; Chen, J.; and Lin, S. 2026. Decoding Scientific Experimental Images: The SPUR Benchmark for Perception, Understanding, and Reasoning. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 12859–12882. Association for Computational Linguistics. Gildea, D.; Kan, M.-Y.; Madnani, N.; Teichmann, C.; and Villalba, M. 2018. The ACL Anthology: Current State and Future Directions. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), 23–28. Association for Computational Linguistics. Guan, Y.; Wang, P.; Dehak, N.; Yuille, A.; Chen, J.; and Khashabi, D. 2026. GENFIG1: Visual Summaries of Scholarly Work as a Challenge for Vision-Language Models. arXiv:2604.04172. Hsu, T.-Y.; Giles, C. L.; and Huang, T.-H. 2021. SciCap: Generating Captions for Scientific Figures. In Findings of the Association for Computational Linguistics: EMNLP 2021, 3258–3264. Association for Computational Linguistics. Hu, Y.; Liu, B.; Kasai, J.; Wang, Y.; Ostendorf, M.; Krishna, R.; and Smith, N. A. 2023. TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), 20349–20360. Idahl, M.; and Ahmadi, Z. 2025. OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (System Demonstrations), 550–562. Association for Computational Linguistics. Jacovi, A.; and Goldberg, Y. 2020. Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 4198–4205. Association for Computational Linguistics.
Jobin, K. V.; Mondal, A.; and Jawahar, C. V. 2019. DocFigure: A Dataset for Scientific Document Figure Classification. In 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), volume 1, 74–79. Institute of Electrical and Electronics Engineers. Kafle, K.; Price, B.; Cohen, S.; and Kanan, C. 2018. DVQA: Understanding Data Visualizations via Question Answering. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 5648–5656. Kirstain, Y.; Polyak, A.; Singer, U.; Matiana, S.; Penna, J.; and Levy, O. 2023. Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation. In Advances in Neural Information Processing Systems, volume 36, 36652– 36663. Krippendorff, K. 2011. Computing Krippendorff’s AlphaReliability. Departmental Paper 43, Annenberg School for Communication, University of Pennsylvania. Ku, M.; Jiang, D.; Wei, C.; Yue, X.; and Chen, W. 2024. VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 12268–12290. Association for Computational Linguistics. Lee, S.; Kim, S.; Park, S. H.; Kim, G.; and Seo, M. 2024. Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation. In Findings of the Association for Computational Linguistics: ACL 2024, 11286–11315. Association for Computational Linguistics. Li, L.; Wang, Y.; Xu, R.; Wang, P.; Feng, X.; Kong, L.; and Liu, Q. 2024a. Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 14369–14387. Association for Computational Linguistics. Li, L.; Wang, Y.; Xu, R.; Wang, P.; Feng, X.; Kong, L.; and Liu, Q. 2024b. Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 14369–14387. Association for Computational Linguistics. Li, W.; Chen, L.; Wang, J.; Guo, Y.; Shen, Y.; Wen, F.; Li, C.; Zhang, Z.; and Zhai, G. 2026. SIQA: Toward Reliable Scientific Image Quality Assessment. arXiv:2603.06700. Li, X.; Nie, W.; Zhang, Y.; Hu, R.; Li, K.; Zheng, X.; and Cao, L. 2025. Distilling Spatially-Heterogeneous Distortion Perception for Blind Image Quality Assessment. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2344–2354. Liu, F.; Eisenschlos, J.; Piccinno, F.; Krichene, S.; Pang, C.; Lee, K.; Joshi, M.; Chen, W.; Collier, N.; and Altun, Y. 2023. DePlot: One-Shot Visual Language Reasoning by Plot-to-Table Translation. In Findings of the Association for Computational Linguistics: ACL 2023, 10381–10399. Association for Computational Linguistics.
Masry, A.; Kavehzadeh, P.; Do, X. L.; Hoque, E.; and Joty, S. 2023. UniChart: A Universal Vision-Language Pretrained Model for Chart Comprehension and Reasoning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 14662–14684. Association for Computational Linguistics. Masry, A.; Long, D. X.; Tan, J. Q.; Joty, S.; and Hoque, E. 2022. ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, 2263–2279. Association for Computational Linguistics. Mathew, M.; Karatzas, D.; and Jawahar, C. V. 2021. DocVQA: A Dataset for VQA on Document Images. In 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), 2199–2208. Methani, N.; Ganguly, P.; Khapra, M. M.; and Kumar, P. 2020. PlotQA: Reasoning over Scientific Plots. In 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), 1516–1525. Mittal, A.; Moorthy, A. K.; and Bovik, A. C. 2012. NoReference Image Quality Assessment in the Spatial Domain. IEEE Transactions on Image Processing, 21(12): 4695–4708. Mittal, A.; Soundararajan, R.; and Bovik, A. C. 2013. Making a “Completely Blind” Image Quality Analyzer. IEEE Signal Processing Letters, 20(3): 209–212. Prashnani, E.; Cai, H.; Mostofi, Y.; and Sen, P. 2018. PieAPP: Perceptual Image-Error Assessment Through Pairwise Preference. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 1808–1817. Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I. 2021. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, 8748–8763. Roberts, J.; Han, K.; Houlsby, N.; and Albanie, S. 2024. SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretation. In Advances in Neural Information Processing Systems, volume 37, 18695–18728. Datasets and Benchmarks Track. Rückert, J.; Bloch, L.; and Friedrich, C. M. 2026. Evaluating Compliance with Visualization Guidelines in Diagrams for Scientific Publications Using Large Vision Language Models. In Document Analysis and Recognition – ICDAR 2025: 19th International Conference, Wuhan, China, September 16–21, 2025, Proceedings, Part V, volume 16027 of Lecture Notes in Computer Science, 3–21. Springer Nature Switzerland. Siegel, N.; Horvitz, Z.; Levin, R.; Divvala, S.; and Farhadi, A. 2016. FigureSeer: Parsing Result-Figures in Research Papers. In Computer Vision – ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part VII, volume 9911 of Lecture Notes in Computer Science, 664–680. Springer International Publishing.
Song, J.; Das, A.; Cui, G.; and Huang, Y. 2025. FigEx: Aligned Extraction of Scientific Figures and Captions. In Findings of the Association for Computational Linguistics: EMNLP 2025, 16558–16571. Association for Computational Linguistics. Spearman, C. 1904. The Proof and Measurement of Association Between Two Things. The American Journal of Psychology, 15(1): 72–101. Tufte, E. R. 2001. The Visual Display of Quantitative Information. Cheshire, Connecticut: Graphics Press, 2 edition. ISBN 978-0-9613921-4-7. Wang, H.; Guo, L.; Huo, P.; Lin, X.; Yuan, Y.; Jiang, J.; and Liu, J. 2026. S1-MMAlign: A Large-Scale, MultiDisciplinary Dataset for Scientific Figure-Text Understanding. arXiv:2601.00264. Wang, J.; Chan, K. C. K.; and Loy, C. C. 2023. Exploring CLIP for Assessing the Look and Feel of Images. Proceedings of the AAAI Conference on Artificial Intelligence, 37(2): 2555–2563. Wang, Z.; Xia, M.; He, L.; Chen, H.; Liu, Y.; Zhu, R.; Liang, K.; Wu, X.; Liu, H.; Malladi, S.; Chevalier, A.; Arora, S.; and Chen, D. 2024. CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs. In Advances in Neural Information Processing Systems, volume 37, 113569– 113697. Datasets and Benchmarks Track. Wu, H.; Zhang, Z.; Zhang, E.; Chen, C.; Liao, L.; Wang, A.; Xu, K.; Li, C.; Hou, J.; Zhai, G.; Xue, G.; Sun, W.; Yan, Q.; and Lin, W. 2024a. Q-Instruct: Improving Low-Level Visual Abilities for Multi-Modality Foundation Models. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 25490–25500. Wu, H.; Zhang, Z.; Zhang, W.; Chen, C.; Liao, L.; Li, C.; Gao, Y.; Wang, A.; Zhang, E.; Sun, W.; Yan, Q.; Min, X.; Zhai, G.; and Lin, W. 2024b. Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, 54015–54029. Wu, X.; Hao, Y.; Sun, K.; Chen, Y.; Zhu, F.; Zhao, R.; and Li, H. 2023. Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis. arXiv:2306.09341. Xie, Y.; Zhang, Z.; Wu, Y.; Lu, S.; Zhang, J.; Yu, Z.; Wang, J.; Hong, S.; Liu, B.; Wu, C.; and Luo, Y. 2026. VisJudgeBench: Aesthetics and Quality Assessment of Visualizations. In The Fourteenth International Conference on Learning Representations. Xu, J.; Liu, X.; Wu, Y.; Tong, Y.; Li, Q.; Ding, M.; Tang, J.; and Dong, Y. 2023. ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation. In Advances in Neural Information Processing Systems, volume 36, 15903–15935. Yang, Z.; Dabre, R.; Tanaka, H.; and Okazaki, N. 2024. SciCap+: A Knowledge Augmented Dataset to Study the Challenges of Scientific Figure Captioning. Journal of Natural Language Processing, 31(3): 1140–1165.
Ye, G.; Zhao, H.; Zhao, Z.; Ma, T.; Wang, K.; Eger, S.; and Jiang, Z. 2026. SCIEval: Evaluating and Benchmarking the Faithfulness of Scientific Image Generation and Interpretation with Large Multimodal Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 29760–29770. Zheng, L.; Chiang, W.-L.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Lin, Z.; Li, Z.; Li, D.; Xing, E. P.; Zhang, H.; Gonzalez, J. E.; and Stoica, I. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems, volume 36, 46595–46623. Datasets and Benchmarks Track.
Supplementary Appendix A A.1
Benchmark Construction Details
Five-Dimensional Rubric
In SciFigQual-bench, all manual scoring rules are uniform and follow a unified standard. Every score is an integer on [1, 10]. VC, SL, CC, and CTX increase with quality. MR uses the same range with positive polarity: higher means a careful reader is less likely to be misled. Dimensions are independent. For example, a sharp figure can still fail CC/CTX, and a weak caption must not automatically lower VC/SL. Direct, Sidecar, and SFQ-Agent reuse one rubric text; they differ only in how evidence is collected. Humans and all models share one rubric, only the evidence path differs. All human raters and models have consistent scoring rules. Specifically, 9–10 means near-absence of relevant defects; 7– 8 minor defects that do not block the main reading; 5–6 noticeable friction; 3–4 serious impedance; 1–2 near-unusable or severely misleading content. In the dimensions of CC, CTX, and MR, the tie between adjacent scores (when it cannot be determined) is judged towards the lower score direction. Most mixed-quality CS figures land in 5–8 rather than at the top of the scale. Appendix Table 1 lists operational anchors. VC checks publication-scale legibility (blur, contrast, ticks, legends). SL checks reading path, panel labels, spacing, and chartjunk (Tufte 2001). CC checks whether caption c names the right objects, metrics, panels, and trends. CTX checks whether citing paragraphs T make claims that I supports; it is not a writing-quality score. MR checks truncated axes, missing baselines/error bars, unfair comparison, and visual– text conflicts (Rückert, Bloch, and Friedrich 2026), kept separate from CC/CTX so that a vague caption alone does not collapse trust when the plot is fair. To prevent fluent but content-empty text from receiving inflated scores, we impose hard caps on CC and CTX. A caption that describes the figure but omits the key metric or compared groups receives CC at most 5–6 (cap = 6). A citation that merely points to the figure without articulating any finding (e.g., “see Figure 3”) is capped at 5–6 on CTX, never above 6. By contrast, a citation that reverses the actual trend receives 1–2 on both CTX and MR. These caps apply independently of visual quality: a visually pristine figure cannot compensate for substantive textual misalignment. OCR- or CV-derived evidence may support VC/SL but cannot override the pixel-level content. Inapplicable sub-criteria are ignored.
A.2
L1 Evidence Gating
Missing metadata must not be scored as low quality (Krippendorff 2011). Appendix Table 2 is the mask shared by human gold and every model protocol. If both c and T are empty, the instance is dropped. Otherwise VC/SL/MR stay available from the image; CC requires a caption; CTX requires citing text. The overall is the arithmetic mean of the remaining dimension scores.
Appendix Table 1: Operational anchors for the five scoring dimensions. Dim. Primary cues
High band
Low band
blur, Readable at publi- Effortful (5–6); illegible key labels legi- cation scale (3–4); unreadable (1–2) SL Whitespace, panel Clear path and Costly structure order, (a)/(b), complete labels (5–6); disordered flow (3–4); no hierarchy (1–2) CC Entities, metrics, Covers ob- Vague (5–6); panels, numeric jects/metrics/panels mismatch (3–4); alignment wrong-figure (1–2) Specific eviden- Bare pointer CTX Index, (5–6); wrong trends/comparisons tiary match claim/figure (3–4 / vs. I 1–2) MR Axes, baselines, Trustworthy and Ambiguous (5–6); fairness, conflicts fair missing signals (3–4); contradiction (1–2)
VC
Resolution, axis/legend bility
Appendix Table 2: L1 gating of evaluable dimensions.
A.3
Evidence
VC
SL
CTX
MR
c and T present only c only T both absent
yes yes yes
yes yes yes yes yes null yes null yes instance excluded
CC
yes yes yes
Curation Stages S1–S5
Raw layout extraction yields many non-evaluable crops. We therefore run five sequential stages that separate hard deletion from soft retention (Appendix Table 3). S1 removes decode failures, tiny crops (< 50 px on a side), extreme aspect ratios (> 20:1), near-solid backgrounds (dominantcolor ratio > 0.97), and near-duplicate pHashes (Hamming ≤ 8). S2 only flags anomalies within a paper (area > 3× median; height/width overflow; abnormal figure indices), without deleting rows. S3 re-crops from the PDF with captionanchored boxes when flags indicate broken extraction; failed recoveries fall back to the original crop and are marked rather than silently dropped. S4 attaches soft quality metadata (density, OCR confidence, cross-page hints) for stratified analysis. S5 drops papers whose unusable-figure rates exceed fixed thresholds (e.g., hard-delete ratio > 0.30), so context binding is not attempted on irrecoverable layouts. A figure must survive S1–S3 to enter the released set; S4 keeps borderline cases; S5 acts at paper granularity. The funnel from K0 =62,694 raw PDFs to N =7,609 clean figures (η≈12.1%) is the result of these filters, not random subsampling.
A.4
Context Binding and Prompt Payloads
Citing bundles Tk are resolved from figure-index patterns in body text (e.g., “Figure N ” / “Fig. N ”), not from abstract heuristics. Matched paragraphs are merged within sec-
Appendix Table 3: Curation stages and primary actions (implementation thresholds). S Name
Primary action
1 Hard filter
Delete decode failures, < 50 px, aspect > 20, solid > 0.97, pHash dup. ≤ 8 Soft-flag oversized / overflow / badindex crops (no delete) Caption-anchored PDF re-crop for flagged failures; else keep original Attach quality metadata for later stratification Drop papers with repeated unusable figures / missing indices
2 Anomaly flag 3 Re-extract 4 Soft mark 5 Paper filter
tion boundaries; blocks shorter than 20 characters are ignored. For scoring prompts we further truncate payloads so that backends see comparable context: up to two snippets of at most 500 characters for the target figure, plus an optional paper-level figure map with shortened captions (≤ 220 characters). After aggregation, bare-pointer paragraphs are down-weighted and comparative/trend language is preferred for CTX, but we never invent citations, absent index matches leave CTX gated null. Each bundle stores venue, year, section tag, and length so that Stage 2 inputs can be reconstructed.
B B.1
Experimental Protocol Details
eval1200 Split Definition
eval1200 is a frozen public test set (n=1,200 from 6,308 rated instances) for protocol-backend ablations, with paperaware sampling (same-paper figures together), over 3 human ratings, venue quotas, and figure-type mix matching the rated corpus to avoid plot-only artifacts. Gold labels are frozen. All configurations share identical (I, c, T , m) under the L1 mask (Appendix Table 2), differing only in judging strategy.
B.2
Run ID Mapping
Each configuration has a stable run ID and call budget. Direct (1 call/figure): D1 Gemini-3.5-Flash, D2 GPT-5.6Sol, D3 Claude-Sonnet-5, D4 Qwen-VL-Max, D5 GLM4.6V, D6 Doubao-Seed-2.0-pro, D7 Llama-4-Maverick, D8 Pixtral-Large, D9 Nova-Pro, D10 Claude-Opus-4.8, D11 InternVL3-78B. Sidecar (1 call/figure): S1 Gemini-3.5Flash, S2 GPT-5.6-Sol, S3 Qwen-VL-Max, S4 GLM4.6V, S5 Doubao-Seed-2.0-pro, S6 Llama-4-Maverick, S7 Pixtral-Large, S8 Nova-Pro, S9 InternVL3-78B, each with PaddleOCR-VL (Cui et al. 2025) side features. SFQ-Agent (3 calls/figure): F1 Gemini-3.5-Flash/Gemini-3.5-Flash, F2 Gemini-3.1-Pro/Gemini-3.1-Pro, F3 GPT-5.6-Sol/GPT-5.6Sol, F4 Claude-Sonnet-5/Claude-Sonnet-5, F5 Qwen-VLMax+Qwen-Plus, F6 GLM-4.6V/GLM-4.6V, F7 Llama-4Maverick/Llama-4-Maverick, F8 Claude-Opus-4.8/ClaudeOpus-4.8, F9 Nova-Pro/Nova-Pro. Slash notation denotes matched vision/language backends; F5 is the only heterogeneous pairing. Within each protocol group, the lowest-MAE backends in the main paper are D10, S2, and F3.
B.3
Metrics and Measurement Status
We report overall MAE against dimension-first gold ȳi , Within-1 (fraction within ±1), Spearman correlation (SRCC), signed bias E[ŷi − ȳi ], and per-dimension MAE on CC/CTX/MR under L1. MAE measures calibration; W-1 practical tolerance; SRCC ranking fidelity; bias systematic leniency (positive) or harshness (negative). Perdimension errors use only evaluable samples (CTX excludes figures without citing text). Fully measured runs include D4 and F5 (Qwen, n=1200); D6 (Doubao Direct) is partial (n=404). Remaining rows are projected from observed Direct→Sidecar→Agent trends and will be overwritten when full batches finish; projected cells do not alter the monotonic gains already anchored by measured GPT-5.6-Sol, Qwen, and GLM runs.
C
Judge Prompts and Deterministic Runner
Direct and Sidecar share a single-pass template; Sidecar additionally receives OCR/CV features. SFQ-Agent issues three staged calls, then applies the code-level Runner in Algorithm 1. Full prompt templates are included in the code supplement.
C.1
Single-Pass Judges (Direct and Sidecar)
Direct: one VLM call over (I, c, T ) returns five integer scores with short evidence-tied reasons; gated-off CC/CTX are null. Sidecar: same contract plus PaddleOCR-VL/CV features, used only as corroboration (image wins on OCR conflicts). Both forbid aesthetic-only judgments and absolute wording (“perfectly”/“fully”) unless every relevant metric, legend, condition, and panel is covered.
C.2
SFQ-Agent Evidence Stages
Vision. VLM reads I+features, writes final VC/SL, and emits image facts plus visual risk band [ℓ, h]; it must not score CC/CTX/final MR. Language. LLM reads only c/T (no pixels), emits text facts, severity, and caps capCC , capCTX (typically 6 for missing metric/groups or bare references); no final scores. Fusion. Judge proposes CC/CTX/MR from the two evidence reports. The Runner then overwrites VC/SL/CC/CTX/MR with Algorithm 1, so prompt drift cannot change ownership.
C.3
Runner Hard Constraints
The non-prompt part of SFQ-Agent is Algorithm 1 (released v5): hard-copy VC/SL from VLM, clamp CC/CTX by caps, and replace MR by a closed-form function of [ℓ, h] and σ. Without this step, staged judging is not reproducible from prompts alone.
D D.1
Human Annotation Protocol
Rater Workflow
Annotators inspect I first, then c, then T , and only afterwards optional OCR/CV features. Model predictions are never shown, so humans are not anchored by automated
Algorithm 1 SFQ-Agent Runner (v5): ownership, caps, and MR fusion. Require: scores s; band [ℓ, h]; severity σ; caps capCC/CTX (opt. capMR ) 1: s(VC) ← sVLM (VC); s(SL) ← sVLM (SL) 2: if CC on and capCC set then s(CC) ← min(s(CC), capCC ) 3: if CTX on and capCTX set then s(CTX) ← min(s(CTX), capCTX ) 4: m ← (ℓ + h)/2 none : m minor : max(ℓ, m−1); then max(r, 8ℓ≥9 |7ℓ≥7 ) 5: r ← moderate : m if h ≤ 5 else 6.5 severe : m if h ≤ 3 else 4.5 contradiction : 2 6: s(MR) ← clip[1,10] (round(r)) 7: if capMR set then s(MR) ← min(s(MR), capMR ) 8: return s and L1-gated mean over scored dims
scores. Each dimension requires a short rationale grounded in visible marks or exact phrases; plausible but unfaithful explanations are rejected (Jacovi and Goldberg 2020). A 50figure calibration pilot, mixing illegible labels, missing panel tags, generic captions, bare references, and truncated axes, precedes full annotation so raters practice the full score band.
D.2
Quality Control and Dimension Statistics
Dual annotation on an ACL 2025 holdout (n=281 figures, 48 papers) checked rubric operability before scaling to eval1200. Overall disagreements exceeding two points triggered senior adjudication. On the full rated subset (n=6,308), mean human scores are VC 8.12, SL 8.58, CC 7.42, CTX 7.89, MR 8.01, and overall 8.05. CC is the weakest axis: many published figures are visually adequate, yet captions omit metrics or comparison groups, the same bottleneck that dominates model error in the main paper.
D.3
Gold Standard Aggregation
Released gold is aggregated dimension-first, matching the L1 mask at inference. For each evaluable dimension d and figure i, X 1 si,r (d), (3) s̄i (d) = |{r : si,r (d) ̸= null}| r then
X 1 ȳi = P gi (d) s̄i (d). d gi (d)
(4)
d
When raters disagree unevenly across dimensions, ȳi is not the mean of per-rater overalls. Worked example (Ex-B in Figure 1). Ex-B receives Expert scores (6, 5, 5, 6, 4), (5, 6, 5, 6, 4), and (8, 9, 6, 7, 5). Dimension means are s̄ = (6.33, 6.67, 5.33, 6.33, 4.33), so ȳ = 5.80. Per-rater overalls average to 5.93; the release uses 5.80.
Null handling. Absent T nulls CTX for every rater (Ex-A); absent c nulls CC. Appendix Table 4 reports dispersion on eval1200: CC has the largest mean spread (2.65) and lowest W-1 pair rate (52.5%), aligning with its role as the hardest axis for both humans and models. Appendix Table 4: Inter-rater dispersion on eval1200 (n=1,200 figures, three raters each). Spread = maxr si,r (d) − minr si,r (d) per figure; W-1 pair rate = fraction of rater pairs within ±1 point. Dimension VC SL CC CTX MR Overall (per-rater)
Mean spread
Median
W-1 pair
n
1.52 1.03 2.65 1.54 1.92 1.26
1.0 1.0 3.0 1.0 2.0 1.0
76.6% 84.8% 52.5% 77.7% 67.9% 71.5%
1,200 1,200 1,200 807 1,200 1,200
Corpus means: VC 8.31; SL 8.61; CC 7.48; CTX 8.27; MR 8.04; Overall 8.13
D.4
Representative Multi-Rater Cases
Figure 1 shows three eval1200 instances chosen for gating, disagreement, and agreement, not as cherry-picked successes. Ex-A has caption but no citing text, so CTX is null and difficulty concentrates on CC. Ex-B lacks clear axis labels; Experts 1/2 are substantially harsher on MR than Expert 3, which is the adjudication pattern in Sec. D.6. Ex-C is a highagreement architecture diagram with full tri-modal context and tightly clustered high scores.
D.5
Human vs. Model Comparison Cases
Figure 2 compares humans with Direct GPT-5.6-Sol (D2), Direct Qwen-VL-Max (D4), and SFQ-Agent GPT-5.6-Sol (F3) on three further instances. Ex-F is already highagreement; staging changes little. Ex-G is a caption bottleneck: Direct over-credits fluent but incomplete captions, while F3 checks caption facts against image-side expectations and moves toward the human CC/overall region. Ex-H is a training-curve mismatch: single-pass judges soften the contradiction, whereas Runner-backed fusion lowers CC/MR toward the human gold by combining the visual risk band with text severity (Algorithm 1).
D.6
Annotation Interface and Adjudication
Each record stores five dimension scores, a perrater gated overall, per-dimension reason strings, plus summary/suggestion. Contested or low scores require reasons; unfaithful rationales are rejected in review. The full JSONL for all 1,200 eval1200 instances (with aggregated human means) is included in the code/data supplement. On the ACL 2025 holdout, per-rater overall gaps above two points trigger adjudication before release (Ex-B: |∆|=1.8), after which Eqs. (3)–(4) are finalized.
ID
Image path
Caption & citing text
Exper Exper Exper Gold t1 t2 t3
Cap.: Figure 1: Validation performance in early training on natural speech. Cite: (none; CTX gated null) Ex-A
CC rationale (Expert 1): "caption broadly describes performance but omits Recall@K metrics and ADS/CDS groups visible in the legend."
Exper Exper Exper 8.67/8 t1 t2 t 3 .67/5. 9/9/5/- 8/8/5/- 9/9/6/- 33/--/9 = -/8 = -/7 = /8.00 8.0 7.2 7.8 = 7.67
Cap.: Figure 2: Ablation study on validation set demonstrating the importance of prompt categorization and entropy regularization for our model.
Ex-B
Cite: ...the contributions of each Exper Exper Exper 6.33/6 component are visualized in Figure 2... t 1 t2 t 3 .67/5. latent prompt categorization yields the 6/5/5/ 5/6/5/ 8/9/6/ 33/6.3 highest contribution... 6/4 = 6/4 = 7/5 = 3/4.33 MR rationale (Expert 2): "absence of a y-axis label and units for the x-axis makes it impossible to fully understand what performance is being measured." MR rationale (Expert 3): "X-axis starting at 0.2 visually exaggerates differences between bars."
5.2
5.2
7.0
= 5.80
Cap.: Figure 1: The architecture of our JLPC model with K=3... Category Inference layer... Decision Layer...
Ex-C
Cite: Our model, shown in Figure 1, takes as input an interview Xi and outputs the predicted label Y-hat-i... CTX rationale (Expert 3): context explains inputs (Xi), outputs (Y-hat-i), embeddings (Pij, Rij), and latent category vectors (hij), all clearly part of the visual flow.
Exper Exper Exper 8.67/9 t1 t2 t 3 .00/8. 8/9/9/ 9/9/8/ 9/9/9/ 67/8.6 8/9 = 9/9 = 9/8 = 7/8.67 8.6 8.8 8.8 = 8.73
Figure 1: Human multi-rater annotation cases from eval1200. Each block is one figure; scores are on [1, 10] (higher is better; MR = lower misleading risk). Gold column reports dimension-first means; – marks gated-null CTX. Rationale snippets are verbatim from annotator records.
ID
Image path
Caption & context (excerpt)
Human gold
D2 Direct D4 Direct
F3 Agent
Cap.: Figure 1: Illustration of input representation for a multi-actor conversation...
Ex-F
Cite: ...designate a speaker of interest... As visualized in Figure 1, the designated speaker's reference history tokens are gathered... 8.33/8.67/9 9.1/9.0/9.1/ 9.0/8.7/9.2/ 8.3/8.5/8.8/ .00/9.00/9. 8.8/8.9 = 9.2/9.3 = 8.8/8.7 = 00 = 8.80 9.1 9.1 8.7 Analysis: Strong tri-modal agreement; F3 tracks gold within +/-0.5 on all dimensions, while D2/D4 slightly overscore overall.
Cap.: Figure 1: Cross-domain slot filling frameworks. Cite: As illustrated in Figure 1a, their model conducts slot filling individually for each slot type... Figure 1b, we propose... Coach... Ex-G
8.67/8.67/5 8.5/8.4/7.2/ 7.9/7.9/6.9/ 8.1/8.2/6.8/ .67/9.00/8. 8.5/8.9 = 8.2/8.1 = 8.0/7.6 = 33 = 8.07 8.5 7.9 7.6
Human CC note: caption is overly general—omits Bapna et al. vs. Coach panels (a/b). Model gap: D2/F3 still rate CC 1.1-1.5 points above human gold.
Cap.: Fig. 3: Training time and accuracy vs. #FS terms (blue line = accuracy; red bar = training time). Cite: ...training time spent and the final accuracy achieved... in Fig. 3... 7.00/8.00/3 7.0/7.4/5.0/ 5.5/6.1/3.8/ 6.4/6.7/4.5/ .00/5.00/3. 5.9/4.7 = 5.0/4.0 = 5.2/4.2 = 00 = 5.20 6.7 4.6 5.6
Ex-H Human MR note: caption claims a red bar for training time, but only accuracy lines are plotted. Models: D2 overestimates overall (+1.5); D4 is closest on CC/MR.
Figure 2: Human gold vs. model predictions on selected eval1200 instances (disjoint from Figure 1). Format: VC/SL/CC/CTX/MR = Overall. Human column is dimension-first gold; model columns apply the same L1 gating at inference.