arXiv:2609.07652v1 [cs.SE] 7 Sep 2026
RefVerifier: Semi-Automated Reference Claim Verification for Scientific Manuscripts Stefania Mocan
Florian Angermeir
Mark Kreitz
Technical University of Munich Munich, Germany
fortiss Munich, Germany Blekinge Institute of Technology Karlskrona, Sweden
University of the Bundeswehr Munich Munich, Germany Blekinge Institute of Technoloy Karlskrona, Sweden
Abstract
1
As software engineering research submission counts surge, peer reviewers face severe time constraints, making systematic verification of citation-supported claims prohibitively expensive. Consequently, unsubstantiated claims and semantic drift can propagate undetected across scientific literature. Existing approaches such as fact-checking and retrieval-augmented generation tools operate on open-domain web data or evaluate claims in isolation without processing complete manuscripts. To address this gap, we present RefVerifier, a semi-automated, citation-bounded reference verification prototype designed to support in academic peer review. RefVerifier extracts citation-bearing sentences from manuscripts, checks bibliography metadata against scholarly databases, resolves references to full-text open-access PDFs, localizes relevant evidence passages, and generates verdicts with natural language explanations. Evaluating RefVerifier on public benchmarks shows claim detection at an F1 score of 0.990, open-access resolution of 57.6% of references, and evidence localization with a hit rate of 98% on abstracts and 68% on complete cited papers. In an end-to-end test with eight manuscripts, RefVerifier achieves a verdict accuracy of 71%. By automating document retrieval and evidence localization while preserving reviewer oversight, RefVerifier provides first indicators for the feasibility of semi-automated integrity checks in scholarly publishing.
Scientific progress relies on citations that establish an auditable evidence trail for asserted claims [17]. Citations attribute prior findings [19], prevent plagiarism [10], and serve as a cornerstone of peer review quality control [6, 16]. In practice, however, reviewers rarely verify whether cited sources actually support the accompanying assertions due to strict submission deadlines and heavy workloads [22]. As a result, inaccurate citations, overstatements, and unsubstantiated claims pass review and propagate through the literature. Greenberg demonstrated how an unsubstantiated claim regarding muscle damage propagated through a citation network of 242 papers, gaining unwarranted academic authority simply through repeated citation without underlying data [7]. Generative Artificial Intelligence (AI) tools exacerbate this risk by generating fluent text with hallucinated references or superficial citation placeholders [4, 27]. Existing verification tools offer limited support for peer review. Open-domain fact-checking systems (e.g., FEVER [23]) assess claim veracity against broad corpora rather than validating claims strictly against author-designated references. Traditional reference managers perform only surface-level metadata checking (e.g., DOI formatting, volume consistency) [9]. Meanwhile, specialized citation tools such as SemanticCite [8] evaluate isolated, manually supplied claims rather than parsing complete manuscripts. To bridge this research gap, we present RefVerifier, a semiautomated reference claim verification system for scientific manuscripts. RefVerifier restricts its evidence strictly to the references cited in the manuscript, executing a four-stage pipeline: (1) deterministic extraction and LLM enrichment of citation-bound claims, (2) automated reference metadata checking and multi-source openaccess paper retrieval, (3) passage-level evidence localization within full cited texts, and (4) verdict prediction with explanations. The reviewer maintains control over the final assessment, guided by an interactive split-pane interface. Contributions:
CCS Concepts • Information systems → Language models; • Computing methodologies → Information extraction.
Keywords Reference Claim Verification, Peer Review Integrity, Natural Language Processing, Large Language Models ACM Reference Format: Stefania Mocan, Florian Angermeir, and Mark Kreitz. 2026. RefVerifier: Semi-Automated Reference Claim Verification for Scientific Manuscripts. In Proceedings of the 1st International Workshop on Automated Techniques for Integrity and Quality in Software-Engineering Research (ATIQSER ’26), October 12–16, 2026, Munich, Germany. ACM, New York, NY, USA, 7 pages. https://doi.org/10.1145/3844135.3845866
This work is licensed under a Creative Commons Attribution 4.0 International License. ATIQSER ’26, Munich, Germany © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2999-7/2026/10 https://doi.org/10.1145/3844135.3845866
Introduction
• We formulate the task of manuscript-level, citation-bounded reference verification and identify the research gap in current peer-review automation tools. • We introduce the architecture and design of RefVerifier, a proof-of-concept tool to support reviewer in reference claim verification. • We build a small end-to-end testing dataset comprising eight manuscripts. • We conduct an empirical quantitative evaluation across established public benchmarks (CiteWorth, SciFact, SCitance,
ATIQSER ’26, October 12–16, 2026, Munich, Germany
Citation-Integrity, SciCiteVal) and a qualitative technology acceptance evaluation with 11 experienced reviewers.
2
RefVerifier addresses this research gap by delivering an end-toend, citation-bounded reference verification tool operating on full manuscripts.
Related Work
We present the foundational definitions, relevant datasets and the research gap.
2.1
Mocan, Angermeir, Kreitz
Foundational Definitions
3
Prototype Design
RefVerifier is implemented as a web application featuring a Python FastAPI backend and a React 19 single-page frontend. Figure 1 illustrates the core architecture across its four sequential stages.
We distinguish three core concepts in reference verification: • Claim: A verifiable proposition expressing a finding about a scientific entity or process [26]. • Citance: A sentence containing an inline citation that asserts a statement regarding the cited work [15]. The citance serves as our primary unit of verification. • Fact: A claim whose evidential support is sufficiently strong to justify treating it as true [18]. While open-domain fact-checking evaluates claims against external knowledge bases (e.g., Wikipedia or web crawls) to issue global truth verdicts [25], reference claim verification evaluates whether an author-designated source substantiates the asserted claim, regardless of external consensus [24]. Similarly, Retrieval-Augmented Generation (RAG) and Question Answering (QA) over scientific papers (e.g., QASPER [5], QASA [11]) optimize for informativeness rather than verifying claim-evidence alignment.
2.2
Benchmark Datasets
General-domain fact-checking benchmarks such as FEVER [23] and AVeriTeC [21] use open retrieval over large web or Wikipedia corpora. Abstract-level datasets (SciFact [26], SCitance [1]) evaluate whether scientific claims are supported, but limit verification to paper abstracts. At the full-paper level, Citation-Integrity [20] provides ground-truth evidence spans for biomedical citations, while SciCiteVal [12] constructs incorrect citation contexts through controlled distortions rather than using naturally occurring miscitations. SemanticCite [8] performs LLM-based citation checking against full reference documents but operates on supplied citation text rather than parsing complete manuscripts. In summary, existing benchmarks lack citation-bounded verification, rely on isolated abstracts, or assume claims are pre-extracted, establishing the gap that RefVerifier addresses. The full list and comparison of established datasets is provided in the online material [14].
2.3
Research Gap
Existing approaches suffer from four fundamental limitations: 1. Lack of citation-bounded verification: Open-domain systems search external corpora rather than restricting verification to authorcited sources. 2. Unextracted claims: Most tools assume prepackaged claims and cannot identify citation-bearing claims directly from complete manuscripts. 3. Fragmented workflows: Prior work addresses isolated sub-tasks (e.g., citation-worthiness detection via CiteWorth [28]) without offering an end-to-end workflow from manuscript parsing to verdict presentation. 4. Limited full-paper support: Datasets rely heavily on abstracts or isolated passages rather than complete multi-page documents.
Stage 1: Claim Identification Claim identification extracts citation-bearing sentences and enriches them with structural metadata through a hybrid deterministicLLM approach: Deterministic Detection: Inline citation markers (e.g., [12], (Smith et al., 2020)) are identified in extracted PDF text using PyMuPDF (fitz) or directly parsed from LaTeX citation keys. If the PDF carries no extractable text layer, Tesseract OCR is invoked as a fallback. LLM Enrichment: Detected citances are sent to the LLM (currently gpt-5.6-terra) in fixed-size batches. The model enriches each citance with: i) Claim Type: Categorized into seven rubrics based on Magnusson and Friedman [13] (Causal, Comparative, Predictive, Proportional, Statistical) and extended by two further types (Background, Naming/Attribution). ii) Hedge Level: Deterministic regex scoring flags uncertainty markers (e.g., "suggests", "may indicate"), relaxing verdict criteria so cautious assertions are not penalized. iii) Span Split: In a sentence with multiple citations, each citation is assigned only its corresponding verbatim span (e.g., X achieves 95% [1], while Y runs in 10ms [2]”, where the verbatim span for [1] is “X achieves 95%” and for [2] is “Y runs in 10ms”). Citations supporting the same proposition receive the same claim and verbatim span. iv) Citation Scope: The model records any lead-in sentences before the citation marker and continuation sentences after it that are supported by the same source. The citation span and its surrounding scope are verified together. v) Ambiguity Handling: Dangling pronouns or missing referents raise an ambiguity flag, triggering an LLM rewrite with explicit interpretation choices.
Stage 2: Metadata Check & Evidence Retrieval Before semantic verification, a meta-data checker validates reference existence. Bibliography entries are parsed into structured fields (authors, title, year, venue, DOI) and cross-checked against Semantic Scholar, Crossref, and OpenAlex. References are flagged as validated, unvalidatable (e.g., unpublished manuscripts), or suspicious (e.g., title/author contradictions indicative of hallucinated citations). Next, the retriever resolves each citation key to a full-text PDF via a multistep resolution chain: The resolver first checks a local cache and user uploads, falls back to direct ID lookups (arXiv, ACL), and sequentially queries open-access APIs like Semantic Scholar, Crossref, OpenAlex, Unpaywall, and Europe PMC before attempting a free-text search on Google Scholar. The resolver stops at the first valid PDF. Confirmed PDFs undergo a deterministic "right-paper" check matching embedded metadata and first-page titles against the reference.
RefVerifier : Semi-Automated Reference Claim Verification for Scientific Manuscripts
ATIQSER ’26, October 12–16, 2026, Munich, Germany
Manuscript under review
Cited Reference
Uploaded arXiv ID, LaTeX project or PDF
Auto-resolved or local PDF library
Stage 1: Claim Identification (Deterministic Marker Extraction + LLM Enrichment) Regex extracts citances from PDF/LaTeX; LLM enriches claim type, span, scope and hedging. Stage 2: Metadata Check & Evidence Retrieval (13-Step Resolution Chain) Cross-checks metadata against Crossref/OpenAlex/S2; fetches full-text PDFs via open-access APIs. Stage 3: Evidence Localization (Bi-Encoder + BM25 + Cross-Encoder / Merged LLM) Ranks passages in cited papers using semantic dense retrieval, BM25 lexical fusion, and cross-encoder reranking. Stage 4: Verdict Prediction (Confidence-Scored Verdict) Generates 4-class verdict (Supported, Partially Supported, Not Supported, NEI) with explanations and quotes.
Reviewer Web Interface (Dual-Pane Split Viewer) Manuscript Viewer Interactive PDF/HTML with highlighted citances and scope bounding boxes
Reference Sidebar
Cited Paper Viewer
Claims grouped by citation, verdict badges, confidence scores, evidence snippets
Side-by-side PDF/HTML viewer with auto-scroll and evidence passage highlighting
Figure 1: Overview of the RefVerifier four-stage verification pipeline and interactive reviewer web interface.
Stage 3: Evidence Localization Evidence localization identifies passages within the cited paper that substantiate the claim. RefVerifier implements two localizer modes: Two-Stage Retriever: The cited paper is divided into overlapping 3-sentence windows (5 sentences for causal/predictive claims). Ranking proceeds via: Semantic Ranking: Bi-encoder all-MiniLM-L6-v2 computes cosine similarity between claim and passage embeddings. Lexical Fusion: Okapi BM25 scores keyword overlap. Reciprocal Rank Fusion (RRF) merges dense and sparse rankings. Cross-Encoder Reranking: A cross-encoder (bge-reranker-v2-m3) reranks top candidates, scoring relevance confidence. Merged LLM Mode (Deployed Default): A single LLM call processes the full text of the cited paper, directly identifying supporting passages and generating the verdicts (see Stage 4) at once, bypassing separate retrieval models and effectively combining stage three and four into a single one.
• Not Enough Information (NEI): The tool could not confidently make a verdict based on the cited document. Prompts incorporate claim-type rubrics and hedge adjustments, returning a natural language explanation.
Web Interface The main page contains an upload pane where the user can choose between the three input sources: a manuscript PDF, an arXiv ID or URL, or a LaTeX project. Once the analysis starts, a progress bar tracks the processing stages. When all stages are complete, the user is forwarded to the frontend split view for reviewing the results. The left pane presents the manuscript under analysis. Clicking a highlighted citance in the manuscript pane auto-scrolls the reference sidebar to its claim card, displaying verdict badges and explanations. Clicking an evidence snippet opens the cited paper pane and auto-scrolls the cited-paper pane directly to the highlighted passage in the resolved PDF.
Stage 4: Verdict Prediction
4
The verdict engine evaluates localized evidence against the claim, returning a four-class verdict:
We evaluated RefVerifier quantitatively through benchmark analyses and qualitatively through a technology acceptance survey.
• Supported: Evidence directly confirms the claim assertion. • Partially Supported: Evidence partially substantiates the claim but reveals minor overstatements, missing nuances, or scope shifts. • Not Supported: Evidence directly refutes the claim or provides contradictory data.
4.1
Evaluation
Quantitative Evaluation
For the quantitative evaluation we employed five public datasets (CiteWorth [28], SciFact [26], SCitance [1], SciCiteVal [12], CitationIntegrity [20]) across the four pipeline stages. These datasets differ in task formulation, evidence scope, and label schemes, and
ATIQSER ’26, October 12–16, 2026, Munich, Germany
Mocan, Angermeir, Kreitz
therefore provide complementary evaluation settings. We additionally tested RefVerifier on a purpose-build end-to-end dataset of manuscripts. Figure 2 provides a high-level overview of the quantitative evaluation strategy. CiteWorth and arXiv PDFs
S1 Claim identification
23 papers in four domains (992 references)
S2 Reference retrieval
SciFact and Citation-Integrity
S3 Evidence localization
SciFact, SCitance, SciCiteVal, Citation-Integrity
S4 Verdict prediction
Eight manuscripts across S1–S4
Figure 2: Quantitative evaluation coverage. Purple boxes denote public benchmarks. Teal boxes denote the authors’ sampled-reference and end-to-end studies. Stage 1: Claim Identification. RefVerifier extracts claims at F1 0.990 on CiteWorth, at high precision and recall. We evaluated claim detection on 300 test paragraphs (1,703 sentences, 578 gold citances) from CiteWorth [28]. As shown in Table 1, deterministic regex detection achieves an F1 score of 0.990, with precision 0.980 and recall 1.000. This corresponds to 12 false positives across and no sentence-level false negatives. To evaluate the performance on PDFs, we choose ten arXiv published manuscripts and evaluated the extraction on the arXiv-rendered PDFs against the corresponding \cite commands in the LaTeX source. Detection recall reached 0.93. Table 1: Stage 1 claim detection performance on CiteWorth data. Evaluation Level Regex Detection Only Sentence-Level (End-to-End) Span-Level (End-to-End)
Precision Recall F1 Score 0.980 0.990 0.986
1.000 0.993 0.991
0.990 0.991 0.989
Stage 2: Reference Retrieval Availability. RefVerifier retrieves the full-text PDFs for 57.6% of references. We evaluated open-access PDF retrieval on 992 references cited across 23 randomly selected source papers from four research domains. RefVerifier resolves 57.6% (571/992) of these references to full-text PDFs. As shown in Table 2, resolution rates vary substantially by domain, with Medicine and Biology showing the lowest resolution rates. Adding Europe PMC and CORE integration raised biomedical availability from 21% to 35%. A title-matching precision audit confirmed that 93% of downloaded PDFs match the cited work. An open-access cross-check of the 421 unresolved references revealed that 231 (54.9%) have no free copy anywhere on the web (paywalled), 77 (18.3%) lack identifiers, and 113 (26.8%) represent open-access copies missed by the chain. Reachability is strongly predicted by publication recency (90% for 2020-2022 vs. 45% for pre-2015) and DOI/arXiv identifier presence (93% with ID vs. 45% without).
Table 2: Reference resolution availability across domains (992 references). Domain Cluster
References Reference Resolution
Computer Science / ML Physics Mathematics Biomedicine
275 119 140 458
86% 86% 52% 35%
Overall
992
57.6%
Stage 3: Evidence Localization. We evaluate evidence localization by measuring whether the annotated evidence appears within the top three returned passages (𝑘 = 3 throughout). We use subsets of SciFact (abstracts, 64 scored claims) and Citation-Integrity (full papers, 80 test citances, 65 with annotated spans), and compare several retrieval and reranking configurations together with the merged LLM mode. Localization performs better on SciFact abstracts, where the bi-encoder reaches a hit rate of 94%. Full-paper localization is more challenging. On Citation-Integrity, the bi-encoder retrieves the correct passage into its top-10 candidate pool for 75% of citances, but ranks it among the top three in only 52% of cases, indicating that ranking is a limitation. Cross-encoder reranking raises the top-3 hit rate to 69% (Table 3). The deployed Merged LLM Mode (gpt-5.6-terra) reaches a hit rate of 68% on full papers and 98% on abstracts (MRR 0.96) outperforming our previous approach using a bi-encoder and reranker. Merged mode selects evidence sentences directly from full paper text, eliminating multi-stage pipeline latency. Table 3: Evidence localization performance on CitationIntegrity full papers (𝑘 = 3 passages) with gpt-5.6-terra for merge LLM mode. Retrieval Configuration
Hit Rate MRR Recall@3
Bi-Encoder Cosine Baseline + Small Reranker (ms-marco) + Strong Reranker (bge-m3)
0.52 0.60 0.69
0.38 0.42 0.52
0.36 0.43 0.49
Merged LLM Mode
0.68
0.52
–
Stage 4: Verdict Prediction. We evaluate verdict prediction on ground-truth evidence, isolating the judgment step from retrieval errors. Because our rubric contains four classes while the benchmarks use three, Partially Supported is mapped to Supported under the lenient mapping and to Not Enough Information under the strict mapping. We report accuracy and Macro F1 for the different datasets in (Table 4). Across the four datasets, F1 ranges from 0.43 to 0.89. This range can be explained by the differences in dataset construction and label distributions. SciFact contains clean, humanannotated contradictions. SCitance’s score is inflated by explicit negation cues in its LLM-generated refutations. Citation-Integrity combines genuine contradictions with subtler errors such as weak substantiation and oversimplification, which do not map cleanly onto our tool’s verdict classes. SciCiteVal additionally contains constructed errors, provides only short excerpts, and has no Not Enough
RefVerifier : Semi-Automated Reference Claim Verification for Scientific Manuscripts
ATIQSER ’26, October 12–16, 2026, Munich, Germany
Information examples, which particularly lowers the Macro-F1 of our tool.
Table 6: End-to-end evaluation metrics across 8 manuscripts (145 confirmed citance-reference pairs) with gpt-5.6-terra. P=Precision, R=Recall.
Table 4: Verdict performance on four public datasets with gpt-5.6-terra. Mappings: Lenient (L) / Strict (S). Dataset
Acc. (L) Acc. (S) F1 (L)
SciFact Citation-Integrity SCitance SciCiteVal
0.91 0.72 0.78 0.66
0.65 0.32 0.60 0.39
0.89 0.62 0.80 0.43
During the development of RefVerifier, we employed small LLMs for cost-reasons. Those models are GPT-5.4-nano, GPT-5.4-mini. To provide a long-term comparable baseline we also tested the approach using an open-source model (Nemotron 3 Ultra 550B), as requested in literature [2, 3]. For the deployed and finally evaluated version we used gpt-5.6-terra. To understand how verdict prediction accuracy scales with different underlying architectures, we evaluated the same 100 Citation-Integrity test claims across the different models and changing reasoning efforts (Table 5) for two setups: First, when handed the relevant passages from the dataset, and second, when provided with the top three passages from our earlier stages only. For the small models, performance is rather similar to the open model, being on par with commercial ones. Nevertheless, gpt-5.6terra performed at any reasoning level better than the small models, but also at an increase in inference cost. Table 5: Verdict accuracy by model configuration on CitationIntegrity (100 test citances, lenient accuracy). We distinguish between (1) dataset passages provided to stage 4 and (2) the top 3 passages form ealier stages provided to stage 4. Model
Effort
(1) Dataset
(2) Top 3
gpt-5.4-nano gpt-5.4-nano gpt-5.4-mini Nemotron 3 Ultra (550B)
low high medium n/a
0.59 0.65 0.62 0.64
0.54 0.58 0.52 0.47
gpt-5.6-terra gpt-5.6-terra gpt-5.6-terra
low medium high
0.72 0.73 0.72
0.59 0.64 0.62
End-to-End Manuscript Verification. We evaluate RefVerifier on eight real-world scientific manuscripts comprising 153 manually annotated citance-reference rows (145 confirmed pairs). Table 6 lists the performance on the stages. Verdicts were blindly annotated, while citance text, reference mapping, and evidence passages were refined from the proposals of RefVerifier, with missing elements added during review when necessary. For evidence localization, the tool typically retrieves the relevant passages while also returning additional candidates that may still be relevant but were not selected by the annotator as the strongest evidence, resulting in lower precision than recall. Most verdict disagreements reflect a
Pipeline Stage
Evaluation Metric
Value
S1: Claim Detection S1: Claim Attribution S2: Reference Mapping S3: Evidence Localize S4: Verdict Prediction
P / R / F1 0.95 / 1.00 / 0.97 Attribution Accuracy 0.86 Mapping Accuracy 1.00 P/ Micro R / F1 0.53 / 0.91 / 0.67 4-Class Accuracy 0.71
cautious tendency toward Partially Supported where the annotator selected Supported (25 of 118 claims). Execution Runtime Feasibility: Across all live manuscript runs with 23-101 references (39 median) and 28-174 claims per paper (60.5 median)), the median end-to-end processing time was 634 seconds (11 minutes). Stage 2 reference retrieval dominated execution time, accounting for 35%-91% (median >65%) of total runtime due to external API restrictions and PDF downloads. LLM claim extraction and verdict generation completed in at most 6.3 minutes per paper.
4.2
Qualitative Evaluation
To assess qualitative acceptance, we conducted a technology acceptance survey (𝑁 = 11) evaluating Perceived Usefulness, Ease of Use, Output Correctness, and Reliance on a 7-point scale through 11 questions. Participants tested the online prototype on a manuscript and then reported their experience. Ten participants reviewed a manuscript inside their own field and one outside it. Six participants inspect cited sources often during reviews, three sometimes, and two rarely. Responses were overall positive. Participants rated RefVerifier as highly useful (median 6) and easy to learn (median 6). Perceived output correctness was also high. Claim identification received a median of 6 (one disagree rating). Finding the supporting evidence and verdict generation received both median ratings of 6, with no strongly agree ratings for either. At the same time, respondents remained cautious about relying on automated judgments. When asked whether they would accept a Support verdict without review, median agreement was relatively low (5, corresponding to 3, “Disagree”). Conversely, when asked whether they would re-check a Not Supported verdict, median agreement was high (7). A free-form question asked for improvement suggestions. In the following we provide a summary of the main improvement suggestions: • Claim identification. Extracted claims sometimes include unrelated preceding sentences. Respondents also noted missed implicit citances and lists interpreted item by item rather than collectively. • Retrieval. In some paywalled cases, only the publicly visible first pages were retrieved, leading to Not Supported rather than an inaccessible-source warning. • Verdict. Respondents frequently reported Partially Supported verdicts triggered by minor terminology differences. Some explanations also relied on manuscript context outside the
ATIQSER ’26, October 12–16, 2026, Munich, Germany
checked claim. One respondent would have marked about one third of the partial-support cases as Supported.
5
Discussion
Due to space restrictions, we only discuss the evaluation results and the barriers to real-world adoption. Evaluation Results. No available dataset evaluates the complete workflow targeted by RefVerifier, which motivated our stage-wise evaluation across complementary datasets. Consequently, the reported benchmark scores should be read as stage-specific indicators. These datasets are repurposed to evaluate specific components of our pipeline rather than reproduced under their original benchmark settings. Consequently, the reported stage scores should be interpreted as indicators of performance within our evaluation setup and are not directly comparable with published results on the respective datasets. Our preliminary eight-manuscript study complements these stage-wise evaluations with an end-to-end perspective, although a broader, jointly annotated benchmark is required for stronger generalization. Barriers to Real-World Adoption. The widespread adoption and usage in real review processes is currently challenging, if not impossible, mostly due to socio-economic and not technical reasons. First, access to full-text manuscripts is, among others, shaped by publication date and the research domain and related sharing practices. While open-access mandates and preprint culture are gradually reshaping how research is disseminated, this shift is a slow, long-term process whose pace varies substantially across disciplines (e.g., openness is far more established in computer science than in medicine or the humanities), and it does little to improve access to the vast body of literature published before such practices took hold. Second, copyright constrains reuse. Some full texts sit in openaccess databases (e.g., ACM’s Digital Library), but these remain a minority of academic publishing. Preprint servers often make paywalled work accessible, yet preprints may only be used cautiously, as they are not the final version, and copyright agreements often restrict their usage. Per-publisher agreements are legally cleanest but do not scale, especially without institutional backing. Third, institutional structures are required to operationalize an approach like ours at scale. To handle the financing of the LLM inference costs at scale. To help navigate the copyright issues discussed above, as publishers or consortia are far better positioned than individual researchers to negotiate processing rights. And to support data-privacy handling regarding unpublished intellectual property, ensuring manuscripts are not exposed to unwarranted retention or misuse by third-party LLM providers. Fourth, the peer-review community needs a cultural willingness to integrate and trust AI-assisted verification tools without over-relying on their automated judgments. This requires established guidance for how such tools should complement, rather than replace, human reviewers’ expertise.
Mocan, Angermeir, Kreitz
used for evaluation (e.g., Citation-Integrity, SciFact) have label definitions that do not map perfectly to our four-class verdict. For example, Citation-Integrity’s Refuted class groups direct contradiction alongside softer issues such as oversimplification and weak substantiation. Consequently, our system often predicts a cautious Partially Supported for these cases, which penalizes the exact match accuracy despite being practically useful for reviewers. Second, regarding external validity, our end-to-end evaluation on manuscripts is limited to a sample of 8 papers assessed by a single annotator. While this confirms the system’s viability, the lack of inter-annotator reliability and the modest sample size limit our ability to generalize the 0.71 accuracy across multiple research domains. Furthermore, the dataset predominantly features wellsupported claims, meaning our evaluation on genuine Not Supported citations relies heavily on the curated Citation-Integrity benchmark. Finally, the open-access resolution chain currently fails to retrieve roughly 42% of citations due to paywalls or lack of identifiers, bounding RefVerifier’s effectiveness by the open-access landscape of the respective field.
7
Conclusion
We presented RefVerifier, a semi-automated, citation-bounded reference claim verification tool for scientific manuscripts. RefVerifier automates citation extraction, multi-source open-access reference retrieval, full-text evidence localization, and verdict prediction. Quantitative evaluation demonstrates strong claim detection, effective reference retrieval open-access resolution), full-paper evidence localization on whole cited papers, and 71% verdict agreement on real-world manuscripts within a median runtime of 11 minutes, of which claim extraction and verdict generation take at most 6.3 minutes. Ongoing and Future Work: Ongoing work focuses on improving citation extraction across superscript and author-year formats and refining verdict rubrics for subtle overstatements. Future deployment depends less on further prototype functionality than on institutional partnerships. Rights-cleared access to full texts, privacy-preserving inference arrangements, sustainable cost allocation, and review-platform integration that keeps human reviewers in control. We therefore plan a live case study with an academic venue or institutional partner to evaluate usability, reviewer trust, and the practical impact of RefVerifier under these conditions.
Data Availability Code and data for the evaluation are available in the online material [14].
GenAI Usage During the preparation of this work, the authors used the models Opus 4.8 and Opus 5, Gemini 3.1 Pro and Gemini 3.6 Flash for language editing and code generation, taking full responsibility for the final content.
Acknowledgements 6
Threats to Validity
Our evaluation and prototype design are subject to several limitations. First, regarding construct validity, the public datasets
This work was funded by the KKS foundation through the SERT Research Profile project (research profile grant 2018/010) at Blekinge Institute of Technology.
RefVerifier : Semi-Automated Reference Claim Verification for Scientific Manuscripts
References [1] Carlos Alvarez, Maxwell Bennett, and Lucy Wang. 2024. Zero-shot Scientific Claim Verification Using LLMs and Citation Text. In Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024). Association for Computational Linguistics, Bangkok, Thailand, 269–276. https://aclanthology.org/2024. sdp-1.25/ [2] Florian Angermeir, Maximilian Amougou, Mark Kreitz, Andreas Bauer, Matthias Linhuber, Davide Fucci, Fabiola Moyón C., Daniel Mendez, and Tony Gorschek. 2025. Reflections on the reproducibility of commercial LLM performance in empirical software engineering studies. doi:10.48550/arXiv.2510.25506 [3] Sebastian Baltes, Florian Angermeir, Chetan Arora, Marvin Muñoz Barón, Chunyang Chen, Lukas Böhme, Fabio Calefato, Neil Ernst, Davide Falessi, Brian Fitzgerald, Davide Fucci, Junda He, Christoph Treude, Marcos Kalinowski, Stefano Lambiase, Daniel Russo, Mircea Lungu, Cristina Martinez Montes, Lutz Prechelt, Paul Ralph, Rijnard van Tonder, and Stefan Wagner. 2026. Guidelines for Empirical Studies in Software Engineering involving Large Language Models. Empirical Software Engineering (2026). arXiv:2508.15503 [cs.SE] https://arxiv.org/abs/2508.15503 [4] Nathan T. Camp, Jason A. Bengtson, and John C. Sandstrom. 2025. The citation catastrophe: Propagation of AI-generated counterfeit citations in scholarship. The Journal of Academic Librarianship 51, 4 (2025), 103065. doi:10.1016/j.acalib. 2025.103065 [5] Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Online, 4599–4610. doi:10.18653/v1/2021.naacl-main.365 [6] C. A. Divecha, M. S. Tullu, and S. Karande. 2023. The art of referencing: Well begun is half done! Journal of Postgraduate Medicine 69, 1 (Jan. 2023), 1–6. doi:10.4103/jpgm.jpgm_908_22 [7] Steven A. Greenberg. 2009. How citation distortions create unfounded authority: analysis of a citation network. BMJ 339 (2009), b2680. doi:10.1136/bmj.b2680 [8] Sebastian Haan. 2025. SemanticCite: Citation Verification with AI-Powered Full-Text Analysis and Evidence-Based Reasoning. arXiv:2511.16198 [cs.CL] https://arxiv.org/abs/2511.16198 [9] I. Heibi, Silvio Peroni, and Elia Rizzetto. 2025. Validating and monitoring bibliographic and citation data in OpenCitations collections. International Journal on Digital Libraries 26 (2025), 16. doi:10.1007/s00799-025-00425-9 [10] Gert Helgesson and Stefan Eriksson. 2015. Plagiarism in research. Medicine, Health Care and Philosophy 18, 1 (2015), 91–101. [11] Yoonjoo Lee, Kyungjae Lee, Sunghyun Park, Dasol Hwang, Jaehyeon Kim, HongIn Lee, and Moontae Lee. 2023. QASA: Advanced Question Answering on Scientific Articles. In Proceedings of the 40th International Conference on Machine Learning (ICML) (Proceedings of Machine Learning Research, Vol. 202). PMLR, 19036–19052. https://proceedings.mlr.press/v202/lee23n.html [12] Qinyue Liu, Yongxin Zhou, and Cyril Labbé. 2026. SciCiteVal: A Multi-Domain Dataset for Scientific Citation Verification. In Proceedings of LREC 2026. Mallorca, Spain. https://hal.science/hal-05576620v1 [13] Ian H. Magnusson and Scott E. Friedman. 2021. Extracting Fine-Grained Knowledge Graphs of Scientific Claims: Dataset and Transformer-Based Results. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP). arXiv:2109.10453 [cs.CL] [14] Stefania Mocan, Florian Angermeir, and Mark Kreitz. 2026. RefVerifier: ATIQSER ’26 Reproduction Package. https://github.com/stefaniamocan/refverifieratiqser26
ATIQSER ’26, October 12–16, 2026, Munich, Germany
[15] Preslav Nakov, Ariel S. Schwartz, and Marti A. Hearst. 2004. Citances: Citation Sentences for Semantic Analysis of Bioscience Text. In Workshop on Search and Discovery in Bioinformatics. https://api.semanticscholar.org/CorpusID:9981257 [16] Colin Neville. 2012. Referencing: principles, practice and problems. RGUHS Journal of Pharmaceutical Sciences 2, 2 (2012), 1–8. [17] Micah Ngatuvai, Cody Autrey, Mark McKenny, and Adel Elkbuli. 2021. Significance and implications of accurate and proper citations in clinical research studies. Annals of Medicine and Surgery 72 (2021), 102841. doi:10.1016/j.amsu.2021.102841 [18] Silas Boye Nissen, Tali Magidson, Kevin Gross, and Carl T Bergstrom. 2016. Research: Publication bias and the canonization of false facts. eLife 5 (dec 2016), e21451. doi:10.7554/eLife.21451 [19] Ani Oranda Panjaitan, Silphia Novelyn, and Frisca Angreni. 2024. The Impact of Proper Citation on Academic Integrity and Scholarship. Asian Journal of Advanced Research and Reports 18, 10 (2024), 192–201. doi:10.9734/ajarr/2024/ v18i10766 [20] Maria Janina Sarol, Shufan Ming, Shruthan Radhakrishna, Jodi Schneider, and Halil Kilicoglu. 2024. Assessing Citation Integrity in Biomedical Publications: Corpus Annotation and NLP Models. Bioinformatics 40, 7 (2024), btae420. doi:10. 1093/bioinformatics/btae420 [21] Michael Schlichtkrull, Zhijiang Guo, and Andreas Vlachos. 2023. AVERITEC: A Dataset for Real-World Claim Verification with Evidence from the Web. In Advances in Neural Information Processing Systems. Curran Associates Inc., New Orleans, LA, USA. [22] Benjamin K. Sovacool, Jonn Axsen, Laurence L. Delina, Hilary Schaffer Boudet, Varun Rai, Roman Sidortsov, Sefa Awaworyi Churchill, Kirsten E. H. Jenkins, and Ray Galvin. 2022. Towards codes of practice for navigating the academic peer review process. Energy Research & Social Science 89 (2022), 102675. doi:10.1016/j. erss.2022.102675 [23] James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a Large-scale Dataset for Fact Extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). Association for Computational Linguistics, New Orleans, Louisiana, 809–819. doi:10.18653/v1/N18-1074 [24] Juraj Vladika and Florian Matthes. 2023. Scientific Fact-Checking: A Survey of Resources and Approaches. In Findings of the Association for Computational Linguistics: ACL 2023. Association for Computational Linguistics, Toronto, Canada, 6215–6230. doi:10.18653/v1/2023.findings-acl.387 [25] Juraj Vladika and Florian Matthes. 2024. Comparing Knowledge Sources for Open-Domain Scientific Claim Verification. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, St. Julian’s, Malta, 2103–2114. doi:10.18653/v1/2024.eacl-long.128 [26] David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or Fiction: Verifying Scientific Claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Online, 7534–7550. doi:10.18653/v1/2020.emnlp-main.609 [27] William H. Walters and Esther Isabelle Wilder. 2023. Fabrication and Errors in the Bibliographic Citations Generated by ChatGPT. Scientific Reports 13, 1 (2023), 14045. doi:10.1038/s41598-023-41032-5 [28] Dustin Wright and Isabelle Augenstein. 2021. CiteWorth: Cite-Worthiness Detection for Improved Scientific Document Understanding. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. https://arxiv.org/ abs/2105.10912
Received 2026-08-16; accepted 2026-09-01