Preprint. Under review.
Document-as-Image Representations Fall Short for Scientific Retrieval Ghazal Khalighinejad, Raghuveer Thirukovalluru, Alexander H. Oh, Bhuwan Dhingra Department of Computer Science Duke University
arXiv:2604.18508v1 [cs.IR] 20 Apr 2026
Abstract Many recent document embedding models are trained on document-as-image representations, embedding rendered pages as images rather than the underlying source. Meanwhile, existing benchmarks for scientific document retrieval, such as ArXivQA and ViDoRe, treat documents as images of pages, implicitly favoring such representations. In this work, we argue that this paradigm is not well-suited for text-rich multimodal scientific documents, where critical evidence is distributed across structured sources, including text, tables, and figures. To study this setting, we introduce ArXivDoc, a new benchmark constructed from the underlying LaTeX sources of scientific papers. Unlike PDF or image-based representations, LaTeX provides direct access to structured elements (e.g., sections, tables, figures, equations), enabling controlled query construction grounded in specific evidence types. We systematically compare text-only, image-based, and multimodal representations across both single-vector and multi-vector retrieval models. Our results show that: (1) document-as-image representations are consistently suboptimal, especially as document length increases; (2) text-based representations are most effective, even for figure-based queries, by leveraging captions and surrounding context; and (3) interleaved text+image representations outperform document-as-image approaches without requiring specialized training.
Standard Document Retrieval Scientific Search Vision Encoder
Figure-based Query
LLM
Table-based Query Representations
Proj.
Document as Image
Text-based Query
Ours (ArXivDoc) Her e the r eson anc e ener gy $\var epsilon $ is fixe d by en ergy co nser va tion . $\var epsilon $ is plo tte d in Fig. ~2(b ) (blac k cur ve ) as a fu nct ion of t he m agnet ic fie ld $B$ t hat cont rols t he in ter actio n str engt h $g$ ( red [gr ay] cur ve) in t he expe rim ent . By in sert ing expr ession s (5) an d (6) in to (2 ) one o bta ins th e imp ort ant r esult \be gin{eq uat ion} J_{ 0,\var epsilon } = \lef t[ \var phi_{\ma thr m{QP }}^*( x) \,\fr ac{d\ch i_\var epsilon (x)} {dx} \ch i_\var epsilon (x)\,\fr ac{d\var phi_{\ma thr m{QP }}^*( x) } {dx} \rig ht]_{x= x_{ \ma thr m{ba r}} }, \ta g{7} \en d{equ atio n} with $\var phi_{\ma thr m{QP }}(x) $ being t he QP WF of th e at om es capin g from the tra p, def ined a s \be gin{eq uat ion} \var phi_{\ma thr m{QP }}(x) = \sqr t{2 } \int _{-\inf ty}^{ \inf ty} dx'\, \ph i_0 (x')\, \Ps i_{ \ma thr m{CM} }^*\!\lef t(\fr ac{x'+x}{2 }\rig ht)\, \Ps i_r ^*(x' - x). \ta g{8} \en d{equ atio n}
Text
Figures Figures
Latex Source
Vision Encoder or Captions
Scientific Search Figure-based Query
LLM
Table-based Query
Her e the r eson anc e ener gy $\var epsilon $ is fixe d by en ergy co nser va tion . $\var epsilon $ is plo tte d in Fig. ~2(b ) (blac k cur ve ) as a fu nct ion of t he m agnet ic fie ld $B$ t hat cont rols t he in ter actio n str engt h $g$ (r ed [gr ay] cur ve) in t he expe rim ent. By in sert ing expr ession s (5) an d (6) in to (2 ) one ob tains t he im por tant resu lt \be gin{eq uat ion} J_{ 0,\var epsilon } = \lef t[ \var phi_{\ma thr m{QP }}^*( x) \,\fr ac{d\ch i_\var epsilon (x )}{ dx} \ch i_\var epsilon (x)\,\fr ac{d\var phi_{\ma thr m{QP }}^*( x \en d{equ atio n}
Representations
Text
Text-based Query
Figure 1: Comparison of document representation paradigms for scientific retrieval. Standard document-as-image approaches process rendered pages through a vision encoder, which handles figure-based queries but struggles to accurately ground text- and table-based evidence. In contrast, ArXivDoc, leverages underlying LaTeX source files to process text and figures natively. This preserves fine-grained document structure, enabling robust retrieval across all multimodal query types.
1
Preprint. Under review.
1
Introduction
Scientific document retrieval requires locating evidence that may appear in text, equations, tables, or figures. For example, a query about a scaling law may depend on an equation and its surrounding discussion, while a query about model architecture may rely on a diagram, and one about experimental results may rely on a specific table row. Yet most existing retrieval systems represent documents either as plain text (e.g., Qwen3-Embedding (Zhang et al., 2025c), Llama-Embed (Babakhin et al., 2025)) or as images (e.g., ColPali (Faysse et al., 2025), Qwen-VL-Embedding (Li et al., 2026)). However, it is not clear what trade-offs are made when representing textual and tabular content as images, nor how these representations scale to longer scientific documents. This raises a fundamental question: which document representations are most appropriate for scientific retrieval, and under what conditions? Recent work has increasingly favored document-as-image representations for embedding, driven by the success of vision–language models (VLMs) that encode rendered page images directly into dense vectors (Yu et al., 2024; Ma et al., 2024a; Faysse et al., 2025; Macé et al., 2025; Günther et al., 2025; Zhang et al., 2025b). However, this design choice introduces a trade-off. Scientific papers are typically generated from structured markup languages such as LaTeX or XML, which explicitly encode document organization and content types. Representing them as images obscure this structure: boundaries between content types must be inferred visually, cross-references are no longer explicit, and distinctions between prose, equations, and figures are not machine-readable. Consequently, models must encode all content—including text, equations, and tables—as pixels rather than text tokens. As more text is packed into a page, it is unclear whether such image-based representations remain effective, especially in dense scientific documents where relevant evidence often appears in prose, equations, or tables rather than figures. Empirical evidence supports this concern: PixelWorld shows that pixel-based inputs degrade more on text-centric reasoning tasks, and recent work finds that VLM performance on scientific documents varies with layout and rendering templates, suggesting sensitivity to surface-level presentation choices (Lyu et al., 2025; Cheng et al., 2025). More broadly, there is an inherent trade-off between textual and visual document representations. Text-based representations align well with language model pretraining and preserve explicit semantic content such as terminology, equations, and logical structure, while visual representations capture layout and multimodal cues but require models to infer content boundaries and relationships implicitly from appearance (Faysse et al., 2025; Wei et al., 2025; Lyu et al., 2025; Jiang et al., 2024). Much of the recent shift toward visual representations has been driven by document collections where accurate text extraction or structural markup is unavailable (Macé et al., 2025; Ma et al., 2024b; Cho et al., 2024). As a result, these approaches rely on OCR tools (Smith, 2007) to recover text, which are prone to errors. In contrast, scientific papers provide clean LaTeX (for those on ArXiv) or XML sources that preserve both content and structure, offering a unique testbed for systematically comparing textual and visual document representations under controlled conditions. For papers without available sources, dedicated tools can recover structured LaTeX from PDFs; for example, Mathpix1 provides high-quality parsing into LaTeX. In this work, we introduce ArXivDoc, a benchmark for analyzing scientific document retrieval across representations. Using raw LaTeX sources, we construct a corpus of 8, 210 documents containing 144, 653 pages, and generate 547 targeted, evidence-grounded queries that are manually verified. We further build multiple document representations—text-only, figure-only, text + VLM captions, document-as-image, and interleaved text–image—enabling controlled comparison within a single framework. Each query is explicitly grounded in text, tables, or figures, allowing fine-grained analysis of how retrieval performance varies with evidence type. Our main findings are as follows: 1. Document-as-image is consistently suboptimal, even for figure-based queries where visual representations might be expected to excel. 2. Text + VLM captions achieves the strongest overall performance, demonstrating that augmenting text with vision–language model descriptions of figures and tables is more effective than replacing text with images. 1 https://mathpix.com
2
Preprint. Under review.
3. Interleaved text–image representations outperform document-as-image. We reuse the same embedding models that were originally trained on page-level document images, and apply them to interleaved text–image inputs without any additional training. Despite this mismatch, they still outperform document-as-image representations, suggesting that combining modalities is more robust than relying on rendered pages alone. 4. As more text is added to a page, document-as-image representations degrade faster than text-based representations. This makes text-based representations more suitable for long scientific documents. 5. Single-vector models outperform late interaction ones despite much smaller index sizes. Single-vector models (e.g., Qwen-Embedding, Qwen-VL-Embedding), which encode each document into a single embedding, consistently achieve better retrieval performance than late interaction models (e.g., ColQwen), which represent documents as multiple vectors. Despite this added complexity, late interaction methods require up to ∼40× larger index sizes while underperforming, indicating a less favorable efficiency–performance trade-off. add other retrieval datasets. compare text/page Benchmark
# Docs
# Pages
# Queries
# Tokens/Page
Target Unit
Modality
Source Availability
Open-Domain
Scientific
QASper ArXivQA MRMR ViDoRe V1 ViDoRe V2 NL-DIR MMDocIR
1,585 100k – – 66 – 313
– – 26,223 – 3,266 41,000 –
5,049 100k 1,435 5,000 3,000 205k 1,658
– – 421 – 488 – 700
Document Figure Webpage Page/Figure Page Page Page
Text Image Text+Image Image Image Image Text+Image
✓ ✗ ✓ ✗ ✗ ✗ ✗
✗ ✗ ✓ ✗ ✓ ✗ ✗
✓ ✓ ✓ ✗ ✓ ✗ ✓
ArXivDoc (Ours)
8,210
144,653
547
958
Document
Text+Image
✓
✓
✓
Table 1: Comparison of document-level multimodal and scientific retrieval benchmarks. Target Unit indicates the expected retrieval granularity (e.g., retrieving a specific page vs. the entire document). Modality indicates the default representation of the corpus. ArXivDoc is the only large-scale, opendomain scientific retrieval benchmark providing raw LaTeX source access alongside rendered images.
2
Related Work
In this section, we position ArXivDoc within existing document retrieval and understanding benchmarks, highlighting the key gaps that remain. Table 1 summarizes these differences. ArXivDoc is an open-domain scientific document retrieval task: given a query, the model must retrieve the single relevant paper from a corpus of more than 8,000 arXiv documents. Queries are context-independent and grounded in specific evidence (text, tables, or figures). This differs from benchmarks such as ArXivQA (Li et al., 2024), which focus on retrieving from a pool of figures rather than complete multimodal documents. Several document benchmarks do not operate in a true retrieval setting. For instance, QASPER (Dasigi et al., 2021) assumes the document is given and evaluates question answering for queries within a paper, while MMLongBench-Doc (Ma et al., 2024b), MMDocIR (Dong et al., 2025), and UniDocBench (Peng et al., 2026) focus on restricted or non-open-domain queries where the relevant context is already provided. As a result, queries do not require identifying the correct document from a large corpus, and models are not challenged to distinguish among many candidates—sidestepping the central difficulty of scientific retrieval. Other benchmarks evaluate retrieval at the page level using document images. ViDoRe V1 and V2 (Faysse et al., 2025; Macé et al., 2025) retrieve relevant pages from visually rich documents, and NL-DIR (Guo et al., 2025) extends this paradigm to larger-scale image-based retrieval. In these settings, documents are decomposed into pages rather than treated as unified documents. While effective for visually salient content, this formulation is less suitable for scientific papers, where evidence is often distributed across text, equations, tables, and figures spanning multiple pages. Additionally, while OCR can be applied to extract text or isolate elements such as figures and tables, it requires additional preprocessing (Choi et al., 2025; Han et al., 2025) that is often error-prone and computationally expensive, introducing noise and variability in the extracted content. As a result, it 3
Preprint. Under review.
Step 1: Generate query from paper
Generated Queries Generation
Step 2: Make the query self-contained
Step 3: Does BM25 fail to retrieve the source?
Decontext. Queries
Decontextualize
Verified Queries Verification
Example: In what way can strong short-range repulsion make the escape rate of a paired two-particle state match that of a noninteracting antisymmetric pair...?
How can strong shortrange repulsive interactions cause the escape (tunneling) rate of a paired two-particle state to equal that of a noninteracting antisymmetric state despite different in-trap spatial densities...?
Step 4: Make more natural
Is the query naturallooking?
Naturalized Queries
No
Is it non-ambiguous and answerable by a specific paper?
Naturalize How can strong shortrange repulsion make the tunneling escape rate of The question is a paired two-particle contextindependent and state match that of a uses interpretable noninteracting physics concepts. antisymmetric twoparticle state, and what boundary-coupling approximation at the trap edge explains this despite different in-trap densities?
Yes Rewrite it to make it more natural.
Final Query How can strong short-range repulsion make the tunneling escape rate of a paired two-particle state match that of a noninteracting antisymmetric twoparticle state?
Figure 2: Dataset construction pipeline. Query counts across stages: LLM verification reduces queries from 761→525 (text), 650→159 (figure), and 2648→280 (table); manual verification further reduces them to 229 (text), 100 (figure), and 218 (table).
becomes difficult to systematically evaluate how different document representations (e.g., text vs. image) impact retrieval performance. MRMR (Zhang et al., 2025a) is closer in spirit to our setting in that it studies open-domain retrieval over multimodal documents with access to structured sources. However, the underlying document distribution differs substantially: MRMR operates over webpages, which are typically less information-dense and less structurally explicit than scientific papers, where critical evidence is often embedded in tightly coupled text, equations, tables, and figures. More importantly, the task objectives diverge. ArXivDoc is designed to reflect a realistic scientific search scenario, where a natural, context-independent query must be used to identify the single correct paper from a large corpus of over 8,000 documents. In contrast, MRMR emphasizes reasoning-intensive retrieval, where queries are often highly specific and tied to localized webpage content, frequently requiring deep multimodal interpretation (e.g., understanding an image) to resolve. ArXivDoc addresses these gaps. It combines: (1) open-domain retrieval, (2) scientific documents, and (3) document-level retrieval. In addition, it provides access to the underlying document sources (e.g., LaTeX when available) alongside rendered pages, which allows us to compare text-based and page-based representations directly. Finally, queries are grounded in specific evidence types (text, tables, figures), enabling controlled evaluation across modalities.
3
Dataset
We introduce ArXivDoc, an open-domain scientific document retrieval benchmark constructed from raw LaTeX sources. The LaTeX source is used to compute document representations and to generate queries, without relying on OCR-extracted PDF text. This allows document content and structure to be preserved consistently across all representations derived from the same underlying source. The benchmark contains 8,210 full-length scientific papers, spanning substantially more pages than prior multimodal retrieval benchmarks (Table 1). This scale supports evaluation over large document collections in which many papers share similar topics, notation, and experimental structure. Example documents and queries are provided in Appendix C.
Page distribution Range
Count
%
1–5 pages 6–10 pages 11–20 pages 21–40 pages 41+ pages
730 2,170 3,152 1,701 455
8.9% 26.4% 38.4% 20.7% 5.5%
Token distribution Range
Count
%
<1k 1k–5k 5k–10k 10k–20k 20k+
20 300 2,429 3,317 2,144
0.2% 3.7% 29.6% 40.4% 26.1%
Queries are generated, filtered, and verified through a multi-stage Table 2: Page and token distribupipeline using gpt-5.2, and are then manually rewritten, edited, tions 4
Preprint. Under review.
Dataset
Example query
ArXivQA ViDoRe V1 MMDocIR ArXivDoc
What does the black sphere with an arrow represent in these diagrams? What process is being depicted in the figure? What variable is being altered to create the different patterns in each subplot? Why do UV-based measurements of the low-redshift star formation rate density often come out higher than other estimates?
Table 3: Example queries from existing benchmarks and ArXivDoc. Existing benchmarks often include short or context-dependent queries tied to local visual content, whereas ArXivDoc emphasizes decontextualized queries designed to retrieve a single target document from a large corpus.
and validated by human annotators. In total, the dataset contains 547 queries: 100 figure-based, 218 table-based, and 229 text-based queries. Design Trade-offs in Query Construction. Retrieval benchmark design involves a fundamental trade-off between three competing objectives: (i) naturalness of queries, (ii) decontextualization (self-contained queries that do not rely on implicit document references), and (iii) unambiguous ground truth. In practice, these objectives are often in tension. Natural queries tend to be short and underspecified, relying on implicit context and permitting multiple valid answers, which increases ambiguity and leads to false negatives. This issue is evident in existing datasets such as ArXivQA (Li et al., 2024), MMDocIR (Dong et al., 2025), and ViDoRe V1 (Faysse et al., 2025), where queries are frequently ambiguous and admit multiple valid matches, making them less suitable for document retrieval (e.g., “What is plotted along the x axis?” from ViDoRe can correspond to many documents). We also identified multiple false negatives in ViDoRe V2: despite its framing as blind contextual querying, some queries remain close-domain or admit multiple plausible matches. Other benchmarks avoid this problem by enforcing decontextualization during dataset construction; for example, MultimodalQA (Talmor et al.; Cho et al., 2024) uses formal languages to generate crossmodal queries. However, such design choices can come at the expense of naturalness. ArXivDoc instead prioritizes decontextualization and unambiguous ground truth while keeping queries as natural as possible, enabling controlled evaluation of open-domain scientific document retrieval over a large corpus, where the goal is to identify the correct document among many topically similar papers. In this setting, minimizing ambiguity is essential for reliable evaluation. Table 3 shows examples of queries from different datasets. 3.1
Query Generation and Filtering
We construct open-domain retrieval queries from three distinct evidence types present in scientific documents: text, tables, and figures. Each evidence type defines a separate pool of candidate queries, and the same multi-step generation and filtering pipeline is applied independently to each pool. We construct text-based queries from LaTeX (.tex) content, figure-based queries from figures in a LaTeX source, and table-based queries from \begin{table} environments. For each query type, we apply a multi-step pipeline: synthetic query generation, decontextualization, difficulty-based filtering, verification, and naturalization (see Figure 2). After this process, expert human annotators review, edit, and filter the resulting queries to ensure quality and validity. Prompt templates used at each stage, and annotation instruction are provided in Appendix B. We note that queries in ArXivDoc are intentionally grounded in localized evidence (a specific text span, table, or figure) rather than requiring multi-hop reasoning across multiple document components. This design choice reflects our focus on document retrieval rather than document-level reasoning: the primary challenge is identifying the correct document from a large corpus, not aggregating evidence within a document. Generation For each evidence source (text, table, or figure), we prompt gpt-5.2 to generate a single query targeting the underlying scientific content. The prompt enforces that the query (i) 5
Preprint. Under review.
requires expert-level reasoning (e.g., about implications, trends, limitations, or constraints), (ii) avoids direct restatement and minimizes lexical overlap through abstraction and paraphrasing, (iii) is answerable from the document without relying on keyword or phrase matching or referencing document-specific elements (e.g., sections, figures, or experiment names), and (iv) consists of exactly one realistic, concise sentence. Full prompt templates are provided in Appendix B. Decontextualization. We observe that many generated queries are context-dependent. For example, the query “Can the top polyhedron be obtained from the cube by a shear?” relies on figure-specific references. After decontextualization, it becomes “Can an oblique parallelepiped be obtained from a cube by an affine shear?”, which removes these references and introduces the required geometric terminology. To make these queries compatible with open-domain retrieval, we rewrite each synthetic query, using gpt-5.2, into a context-independent form that removes explicit references to figures, tables, or document-local structure. However, this step often produces queries that are only superficially decontextualized. Many rewritten queries remain underspecified, as they do not introduce sufficient scientific context to stand on their own. For example: Original: If the variable on the x-axis represents time, what can be inferred about the rate of change of the parameter over time? Rewritten: If the independent variable represents time, what can be inferred about the rate of change of the parameter over time? Although gpt-5.2 is supposed to return null when it cannot generate a valid rewrite, it frequently outputs very slight paraphrases like the one above, which are still unclear without the original context. This makes a final verification step necessary. Difficulty-Based Filtering. We remove trivially easy queries using an automated difficulty filter that leverages retrieval behavior. For each query, we run BM25 (Lù, 2024) over a chunked document corpus and remove the query if its gold document ranks within the top five results, as this suggests the answer can be found through shallow lexical overlap. This step eliminates roughly 40% of all queries. Final Verification. All remaining queries undergo a final verification step using gpt-5.2 to ensure that they constitute valid open-domain retrieval queries. This verification checks that queries are interpretable without document context. After verification, the query counts are reduced from 761 to 525 (text), 650 to 159 (figure), and 2648 to 280 (table). Human Annotation Each query is manually evaluated along three dimensions: naturalness, ambiguity, and document answerability. Queries that are unclear, underspecified, or unsupported by evidence in the gold document are rewritten or removed. In total, we involve three annotators: two PhD students and one Master’s student. A Master’s student reviews all queries using the following criteria: queries must be clear and plausible (naturalness), uniquely identify a target document (ambiguity), and be directly supported by evidence in the gold document (document answerability). Further annotation details are provided in Appendix 8.
4
Experiments
4.1
Problem Formulation
We study open-domain scientific document retrieval. Let D = {d1 , . . . , d N } denote a corpus of scientific documents. Each document di is represented as a collection of embedding units ei , where ei = {ei1 , . . . , eiMi }, and eij denotes the j-th embedding unit of document di . Each unit corresponds to a document component such as a text chunk, figure, or page, depending on the chosen representation and model. Let q denote a context-independent natural language query targeting a specific piece of scientific evidence (text, table, or figure). 6
Preprint. Under review.
Given a query q, a retrieval system computes a similarity score between q and each embedding unit eij ∈ ei using a scoring function s(q, eij ). The document-level score for di is then defined as S(q, di ) = maxeij ∈ei s(q, eij ), and documents are ranked according to S(q, di ). 4.2
Experimental Setup
We use our dataset, ArXivDoc, to assess how different document representations support opendomain retrieval from scientific papers. Starting from the underlying LaTeX source, we construct multiple representations of the documents and compare their retrieval performance. We also report the corresponding index sizes. Retrieval performance is measured using normalized discounted cumulative gain at rank 10 (nDCG@10), which evaluates whether the relevant document is ranked near the top of the retrieval list. Representations and Models. We consider three classes of document representations: text-only, image-only, and text + image. Within each class, we evaluate multiple representations. Text-only. (i) Text (LATEX), which indexes raw LATEX source text. We first flatten the source to a single file to account for projects split across multiple .tex inputs (e.g., via \input or \include). We then apply lightweight normalization to remove non-semantic markup: comments and common formatting commands (e.g., \cite, \ref, \label, \footnote, and styling macros such as \emph, \textbf), while preserving scientific content such as plain text, math, and structure. (ii) Text + VLM Captions, which augments document text with figure descriptions generated by a vision–language model, appended to the end of the document. Image-only. (iii) LATEX Figures, which indexes rendered figures extracted from the LATEX sources while ignoring document text. We collect all figure assets (e.g., .png, .jpg, .pdf, .eps) and convert them into a unified format prior to embedding. This representation isolates visual content, but may miss critical information when the evidence required to answer a query resides in the text. Text + Image. (iv) Document-as-Image, which indexes full document pages rendered as images. (v) Interleaved Text + Images, which jointly indexes text and figures while preserving their original order. We parse the LATEX source to extract textual spans and figure references, resolve each reference to its rendered image, and construct an interleaved sequence reflecting the document’s narrative flow. Retrieval units are formed by segmenting text into chunks and associating each chunk with nearby figures, producing multimodal units with one or two images. Across these representations, we evaluate several embedding models, depending on modality compatibility: Qwen3-Embedding-8B (Zhang et al., 2025c) as a text-only embedder, and Qwen3-VLEmbedding-8B (Li et al., 2026), ColQwen2 v1 (Faysse et al., 2025), OpenCLIP ViT-G/14 (Cherti et al., 2023; Radford et al., 2021), and Ops-MM-Embedding v1 (Lin et al.) as image embedders. For each representation–model pair, we report results using the best-performing configuration; full hyperparameter sweeps are reported separately. ColQwen is a late-interaction model based on the ColBERT framework (Khattab & Zaharia, 2020; Santhanam et al., 2022), built on top of the Qwen2-VL (Wang et al., 2024a), which encodes queries and documents into sets of token-level embeddings and computes document relevance via maxsimilarity aggregation across embedding units, resulting in substantially larger indices than singlevector models. In contrast, Qwen, OpenCLIP, and Ops-MM-Embedding produce a single embedding per input unit and rely on standard vector similarity for retrieval. ColQwen was originally introduced for page-level visual document retrieval and is not trained on text documents. Nevertheless, we apply ColQwen to text chunks by treating each chunk as an embedding unit and using the model’s language encoder. This allows us to evaluate a late-interaction retrieval model on purely textual representations and to compare its behavior directly with single-vector text embedding models under identical document inputs. Surprisingly, we find that ColQwen is effective on textual inputs, which motivates our experiments with interleaved text–image representations; as we show, these outperform document-as-image representations when encoded using the same model. We further observe that Qwen-VL-Embedding—despite not being trained for text embedding, similar to ColQwen—exhibits the same trend, achieving better performance on interleaved text + image representations than on document-as-image inputs. 7
Preprint. Under review.
Input
Model
Index (GB)
Text
Table
Figure
Avg.
Text Only
OpenCLIP Ops-MM-Embedding ColQwen Qwen3-Embedding
1.28 2.24 40.54 2.56
0.18 0.75 0.77 0.87
0.12 0.52 0.56 0.60
0.48 0.75 0.80 0.76
0.21 0.66 0.69 0.74
Text + VLM Captions
OpenCLIP Ops-MM-Embedding ColQwen Qwen3-Embedding
1.65 2.88 52.10 1.54
0.19 0.74 0.74 0.87
0.11 0.50 0.54 0.60
0.50 0.75 0.90 0.80
0.21 0.65 0.69 0.75
Figures Only
OpenCLIP Ops-MM-Embedding ColQwen Qwen3-VL-Embedding
0.49 0.86 35.43 1.05
0.03 0.26 0.18 0.22
0.07 0.25 0.18 0.25
0.60 0.80 0.87 0.83
0.15 0.36 0.30 0.35
Doc-as-Image
OpenCLIP Ops-MM-Embedding ColQwen Qwen3-VL-Embedding
0.60 1.04 70.56 1.30
0.02 0.70 0.73 0.78
0.05 0.51 0.52 0.55
0.41 0.75 0.84 0.78
0.10 0.63 0.67 0.69
Interleaved (Text + Image)
ColQwen Qwen3-VL-Embedding
49.59 2.98
0.78 0.85
0.56 0.57
0.82 0.75
0.70 0.72
Table 4: Retrieval performance (NDCG@10) across different document representations and models. Index (GB) reflects storage cost.
Index Size. Index size is defined as the total storage required for all embedding vectors, excluding model parameters. For text-based representations, index size is controlled by varying the chunk size used to segment documents prior to embedding. Smaller chunks increase the number of units and the total index size, while larger chunks reduce storage at the cost of coarser representations. This mechanism is used for all text indexings. Note that varying chunk size does not substantially change the index size for ColQwen, since text is encoded at the token level and stored as a set of embeddings regardless of chunk boundaries. For vision-based representations, index size is controlled via the max_pixels parameter, which caps the total number of input pixels processed per image. Images are resized to approximately preserve aspect ratio while satisfying this budget. The number of visual tokens scales with the ′ ′ ′ ′ effective image resolution and can be approximated as Tvis ∝ HPW 2 , where H W ≤ max_pixels is the resized image resolution and P is the vision encoder’s patch size. Reducing max_pixels therefore decreases the number of visual tokens and the resulting index size, trading visual detail for storage efficiency. We tune the max_pixels parameter for each model–representation pair and report the best-performing configuration. 4.3
Main Results
Table 4 reports retrieval performance measured by nDCG@10, using the best configuration for each representation–model pair. Results are reported separately for text, table, and figure queries. Main Results. (1) Document-as-image representations are consistently suboptimal. Across all query types (text, table, and figure), document-as-image representations underperform compared to alternatives. Even for figure-based queries, the best-performing model is ColQwen with text + VLM captions, a purely text-based representation. Moreover, the interleaved text + image representation also outperforms document-as-image, indicating that preserving structure and modality alignment is more effective than treating the document as a flat image. (2) Text alone is surprisingly competitive for figure-based queries. Even without access to figures or VLM-generated captions, text-only models achieve strong performance on figure-based queries. For example, Qwen (text-only) is within 0.02 of Qwen-VL (doc-as-image) on figure queries. This suggests that scientific documents often describe and interpret figures in the surrounding text. 8
Preprint. Under review.
(3) Interleaved representations outperform document-as-image despite no dedicated training. Even though none of the embedding models are explicitly trained for interleaved text+image inputs, this representation still outperforms document-as-image. This highlights the importance of preserving the document’s native structure and aligning text with corresponding figures, rather than collapsing the entire document into a single image representation. (4) Single-vector representations outperform multi-vector ones despite smaller index sizes. Across both text and multimodal settings, single-vector models outperform late-interaction models while requiring substantially smaller index sizes. For text inputs, the best-performing configuration of Qwen-Embedding outperforms ColQwen by 0.05. Similarly, for multimodal/image-based inputs, Qwen-VL-Embedding outperforms ColQwen by 0.02. These results suggest that the added complexity and storage cost of multi-vector representations do not translate into improved retrieval performance in this setting. 4.4 4.4.1
Analysis Scaling Behavior
We study how retrieval performance changes as the amount of context increases. Starting from the flattened LaTeX source, we sample a base window of 500 tokens and progressively expand it to larger contexts (1000, 4000, and 8000 tokens), while keeping the query fixed across all variants. For each window, we convert the text into a minimally executable LaTeX snippet and render it to a PDF. Then, we extract a single page image. This enables a controlled comparison between text-only and document-as-image representations under matched content. For retrieval, we use a fixed 100-token query sampled from the original 500-token window, from 130 documents. Figure 3, shows that document-asimage representations degrade more sharply as context grows, in contrast to text-based repre- Figure 3: Retrieval performance (NDCG@10) as sentations. a function of context length (token length) across document representations. The max-pixels param4.4.2 Why Do Text Representations eters are tuned (see Appendix A). Work for Figure-Based Queries? To understand why text representations perform well on figure-based queries, we analyze the highest-scoring retrieved text chunk from the correct document for each query. For each case, we check whether the top chunk (i) lies near the figure, (ii) explicitly references it, or (iii) contains caption-level information. A chunk is considered near if it lies within a small window (±2 chunks). To detect figure references, we use regex patterns such as Fig., \ref{fig}, and variants of Figure X. To assess caption-level information, we extract informative words from the caption (after removing stopwords) and measure their overlap with the chunk text. As a baseline, we perform the same analysis on top chunks retrieved from incorrect documents. Metric Correct Doc Incorrect Doc Results are summarized in Table 5. CorrectCount 63 61 document retrievals frequently exhibit all three Near Figure (%) 41.3 23.0 signals, while baseline chunks show lower rates References Figure (%) 60.3 42.6 across these indicators. These results suggest Contains Caption Info (%) 68.3 24.6 that text retrieval does not require access to the figure itself. Instead, it relies on surrounding Table 5: Analysis of the highest-scoring retrieved textual descriptions—references, explanations, text chunk for figure-based queries. and caption content—that encode the figure’s information in text form. 9
Preprint. Under review.
4.4.3
Comparison with ViDoRe Benchmark
To assess the generalizability of our findings, we evaluate text-only and document-as-image retrieval representations on the ViDoRe benchmark. Since ViDoRe provides only document images, we extract text using PaddleOCR (Cui et al., 2025) to obtain text-only representations. We then compare text-only representation, against document-as-image, across the subsets. As shown in Table 6, the results on ViDoRe differ from those on ArxivDoc, with document-asText Doc as Image Subset #Tokens image representations generally performing betQwen ColQwen Qwen VL ter than text-only representations. We attribute Biomedical 108 0.64 0.60 0.68 this discrepancy to several differences between 0.50 0.53 0.51 Economics 704 the two benchmarks. First, ArxivDoc consists ESG 582 0.50 0.56 0.65 ESG HL 583 0.61 0.60 0.68 of scientific documents that are inherently textdense, with an average of 947 tokens per page, Table 6: nDCG@10 on ViDoRe dataset. structured prose, and technical vocabulary. In contrast, ViDoRe comprises lecture slides, corporate ESG reports, and economic reports that are more visually designed, with fewer tokens per page (108–704) and greater reliance on layout, charts, and infographics to convey information. This makes ViDoRe documents better suited for visual representations, whereas the retrieval signal in scientific documents resides predominantly in the text. Second, ViDoRe’s page-level evaluation introduces noise due to the combination of shorter, less specific queries (21 tokens on average, compared to 34 in ArxivDoc) and page-level granularity. Because these queries contain less domain-specific terminology, multiple pages from the same document often include relevant information. We sampled 100 queries where the Qwen text embedder did not retrieve the ground truth in the top 5 and used an LLM judge (GPT-5.2) to assess the retrieved documents by asking whether the query can be answered given each document. Across the retrieved documents, the judge identified 22% of queries containing at least one missed but answerable result, indicating false negatives, an issue also highlighted by recent works (Wang et al., 2024b; Wasserman et al., 2025). In contrast, ArxivDoc operates at the document level with more specific queries that map cleanly to a single ground truth document, and in our evaluation of 100 queries we observe no false negatives. These findings indicate that the ViDoRe results should be interpreted with caution. These findings highlight a key difference between our setting and ViDoRe: document-as-image representations may be advantageous when queries are broad or when multiple pages contain overlapping information, whereas text-based representations benefit from more specific queries and clear document-level grounding. Notably, this distinction is also consistent with findings from the MRMR (Zhang et al., 2025a) study, which similarly reports stronger performance from Qwen3-based text embeddings relative to document-as-image approaches on their document-level dataset.
5
Conclusion and Future Work.
We present ArXivDoc, a benchmark for scientific document retrieval built from raw LaTeX sources. Using LaTeX allows direct access to structured elements such as sections, tables, figures, and equations, enabling controlled analysis of how different representations behave across text, tables, and figures. Our results show that document-as-image representations are not always well-suited for scientific documents, particularly when evidence is embedded in dense textual and structured content. An important observation is that interleaved text+image representations outperform document-asimage approaches even without explicit training for such inputs, suggesting that training embedding models on interleaved scientific documents is a promising direction. More broadly, this points to a gap between how scientific documents are structured and how current retrieval models are trained. Moreover, extending ArXivDoc to support queries that require combining evidence across multiple parts of a document—or across documents—can further advance retrieval in scientific settings.
Ethics Statement We do not believe there are significant ethical issues associated with this research. 10
Preprint. Under review.
References Yauhen Babakhin, Radek Osmulski, Ronay Ak, Gabriel Moreira, Mengyao Xu, Benedikt Schifferer, Bo Liu, and Even Oldridge. Llama-embed-nemotron-8b: A universal text embedding model for multilingual and cross-lingual tasks, 2025. URL https://arxiv.org/abs/2511.07025. Jiale Cheng, Yusen Liu, Xinyu Zhang, Yulin Fei, Wenyi Hong, Ruiliang Lyu, Weihan Wang, Zhe Su, Xiaotao Gu, Xiao Liu, et al. Glyph: Scaling context windows via visual-text compression. arXiv preprint arXiv:2510.17800, 2025. Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2818–2829, 2023. Jaemin Cho, Debanjan Mahata, Ozan İrsoy, Yujie He, and Mohit Bansal. M3docrag: Multi-modal retrieval is what you need for multi-page multi-document understanding. 2024. Yejin Choi, Jaewoo Park, Janghan Yoon, Saejin Kim, Jaehyun Jeon, and Youngjae Yu. Zero-shot multimodal document retrieval via cross-modal question generation. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 26079–26094, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-3326. doi: 10.18653/v1/2025.emnlp-main.1324. URL https://aclanthology.org/2025. emnlp-main.1324/. Cheng Cui, Ting Sun, Suyin Liang, Tingquan Gao, Zelun Zhang, Jiaxuan Liu, Xueqing Wang, Changda Zhou, Hongen Liu, Manhui Lin, et al. Paddleocr-vl: Boosting multilingual document parsing via a 0.9 b ultra-compact vision-language model. arXiv preprint arXiv:2510.14528, 2025. Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. A dataset of information-seeking questions and answers anchored in research papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4599–4610, 2021. Kuicai Dong, Yujing Chang, Derrick Goh Xin Deik, Dexun Li, Ruiming Tang, and Yong Liu. MMDocIR: Benchmarking multimodal retrieval for long documents. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 30959–30993, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-3326. doi: 10.18653/v1/2025.emnlp-main.1576. URL https://aclanthology.org/2025. emnlp-main.1576/. Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, CELINE HUDELOT, and Pierre Colombo. Colpali: Efficient document retrieval with vision language models. In The Thirteenth International Conference on Learning Representations, 2025. Michael Günther, Saba Sturua, Mohammad Kalim Akram, Isabelle Mohr, Andrei Ungureanu, Bo Wang, Sedigheh Eslami, Scott Martens, Maximilian Werk, Nan Wang, and Han Xiao. jinaembeddings-v4: Universal embeddings for multimodal multilingual retrieval. In David Ifeoluwa Adelani, Catherine Arnett, Duygu Ataman, Tyler A. Chang, Hila Gonen, Rahul Raja, Fabian Schmidt, David Stap, and Jiayi Wang (eds.), Proceedings of the 5th Workshop on Multilingual Representation Learning (MRL 2025), pp. 531–550, Suzhuo, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-345-6. doi: 10.18653/v1/2025.mrl-main.36. URL https://aclanthology.org/2025.mrl-main.36/. Hao Guo, Xugong Qin, Jun Jie Ou Yang, Peng Zhang, Gangyan Zeng, Yubo Li, and Hailun Lin. Towards natural language-based document image retrieval: new dataset and benchmark. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 29722–29732, 2025. Siwei Han, Peng Xia, Ruiyi Zhang, Tong Sun, Yun Li, Hongtu Zhu, and Huaxiu Yao. Mdocagent: A multi-modal multi-agent framework for document understanding. arXiv preprint arXiv:2503.13964, 2025. 11
Preprint. Under review.
Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. E5-v: Universal embeddings with multimodal large language models. arXiv preprint arXiv:2407.12580, 2024. Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pp. 39–48, 2020. Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. Multimodal ArXiv: A dataset for improving scientific comprehension of large vision-language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14369–14387, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/ 2024.acl-long.775. URL https://aclanthology.org/2024.acl-long.775/. Mingxin Li, Yanzhao Zhang, Dingkun Long, Keqin Chen, Sibo Song, Shuai Bai, Zhibo Yang, Pengjun Xie, An Yang, Dayiheng Liu, Jingren Zhou, and Junyang Lin. Qwen3-vl-embedding and qwen3-vl-reranker: A unified framework for state-of-the-art multimodal retrieval and ranking, 2026. URL https://arxiv.org/abs/2601.04720. Sheng-Chieh Lin, Chankyu Lee, Mohammad Shoeybi, Jimmy Lin, Bryan Catanzaro, and Wei Ping. Mm-embed: Universal multimodal retrieval with multimodal llms. In The Thirteenth International Conference on Learning Representations. Zhiheng Lyu, Xueguang Ma, and Wenhu Chen. Pixelworld: Towards perceiving everything as pixels. Transactions on Machine Learning Research, 2025. Xing Han Lù. Bm25s: Orders of magnitude faster lexical search via eager sparse scoring, 2024. URL https://arxiv.org/abs/2407.03618. Xueguang Ma, Sheng-Chieh Lin, Minghan Li, Wenhu Chen, and Jimmy Lin. Unifying multimodal retrieval via document screenshot embedding. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 6492–6505, Miami, Florida, USA, November 2024a. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.373. URL https: //aclanthology.org/2024.emnlp-main.373/. Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, et al. Mmlongbench-doc: Benchmarking long-context document understanding with visualizations. Advances in Neural Information Processing Systems, 37:95963–96010, 2024b. Quentin Macé, António Loison, and Manuel Faysse. Vidore benchmark v2: Raising the bar for visual retrieval. arXiv preprint arXiv:2505.17166, 2025. Xiangyu Peng, Can Qin, Zeyuan Chen, Ran Xu, Caiming Xiong, and Chien-Sheng Wu. Unidoc-bench: A unified benchmark for document-centric multimodal rag, 2026. URL https://arxiv.org/ abs/2510.03663. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748–8763. PmLR, 2021. Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. ColBERTv2: Effective and efficient retrieval via lightweight late interaction. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3715–3734, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.naacl-main.272. URL https: //aclanthology.org/2022.naacl-main.272/. R. Smith. An overview of the tesseract ocr engine. In Ninth International Conference on Document Analysis and Recognition (ICDAR 2007), volume 2, pp. 629–633, 2007. doi: 10.1109/ICDAR. 2007.4376991. 12
Preprint. Under review.
Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi, and Jonathan Berant. Multimodalqa: complex question answering over text, tables and images. In International Conference on Learning Representations. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024a. Shiqi Wang, Yeqin Zhang, and Cam-Tu Nguyen. Mitigating the impact of false negative in dense retrieval with contrastive confidence regularization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 19171–19179, 2024b. Navve Wasserman, Roi Pony, Oshri Naparstek, Adi Raz Goldfarb, Eli Schwartz, Udi Barzelay, and Leonid Karlinsky. Real-mm-rag: A real-world multi-modal retrieval benchmark. arXiv preprint arXiv:2502.12342, 2025. Haoran Wei, Yaofeng Sun, and Yukun Li. Deepseek-ocr: Contexts optical compression, 2025. URL https://arxiv.org/abs/2510.18234. Shi Yu, Chaoyue Tang, Bokai Xu, Junbo Cui, Junhao Ran, Yukun Yan, Zhenghao Liu, Shuo Wang, Xu Han, Zhiyuan Liu, et al. Visrag: Vision-based retrieval-augmented generation on multi-modality documents. arXiv preprint arXiv:2410.10594, 2024. Siyue Zhang, Yuan Gao, Xiao Zhou, Yilun Zhao, Tingyu Song, Arman Cohan, Anh Tuan Luu, and Chen Zhao. Mrmr: A realistic and expert-level multidisciplinary benchmark for reasoning-intensive multimodal retrieval, 2025a. URL https://arxiv.org/abs/2510.09510. Xin Zhang, Yanzhao Zhang, Wen Xie, Mingxin Li, Ziqi Dai, Dingkun Long, Pengjun Xie, Meishan Zhang, Wenjie Li, and Min Zhang. Bridging modalities: Improving universal multimodal retrieval by multimodal large language models. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9274–9285, 2025b. doi: 10.1109/CVPR52734.2025.00866. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176, 2025c.
A
Hyperparameters
Table 7 reports retrieval accuracy (nDCG@10) under varying storage budgets, grouped by document representation. Storage is controlled through representation-specific hyperparameters, including text chunk size, visual token budgets (max_pixels), and their combinations for interleaved representations.
13
Preprint. Under review.
Representation
Model
Index Size (GB)
Text
Table
Figure
2.56 1.20 0.34
0.87 0.85 0.83
0.60 0.54 0.47
0.76 0.76 0.64
40.54 40.54
0.77 0.77
0.56 0.57
0.80 0.80
22.09 35.43 65.99
0.17 0.18 0.17
0.16 0.18 0.16
0.86 0.87 0.86
47.23 94.79
0.71 0.73
0.51 0.52
0.83 0.84
Qwen Text
chunk=512 chunk=1024 chunk=4096 ColQwen chunk=1024 chunk=4096 ColQwen
LATEX Figures
max_pixels = 600K max_pixels = 1M max_pixels = 2M ColQwen
Document-as-Image
max_pixels = 1M max_pixels = 2M
Table 7: Retrieval accuracy (NDCG@10) under varying storage budgets, grouped by document representation.
14
Preprint. Under review.
B
Query Generation
The query generation pipeline consists of four stages, three of which involve prompting gpt-5-mini. In this section, we report the system prompts used at each stage. B.1
Synthetic Query Generation Prompt (Text) PROMPT You are given extracted text from a scientific research paper. Your task is to generate a single, high-quality synthetic query that would meaningfully test a document retrieval system. Instructions: 1. The query must require expert-level reasoning over implications, trends, limitations, or constraints discussed in the document, and must not be a direct restatement of any sentence from the input. 2. The query must minimize lexical overlap with the input text by avoiding distinctive phrases or terminology, relying instead on abstraction and paraphrasing rather than keyword matching. 3. The query must be answerable from the document but not trivially retrievable via keyword or phrase matching, and must not reference sections, figures, experiments, or document-specific wording. 4. The query must ask exactly one focused question, without combining multiple sub-questions or enumerating parameters or conditions. 5. The query must be realistic and concise, phrased as a single sentence that a knowledgeable researcher would plausibly ask, without verbose framing or artificial difficulty. 6. If no query satisfying these criteria can be generated, return null. Required Output Format: { "query": "<generated question or null>" }
Here is the document content: {paper_text}
Figure 4: Prompt used for generating synthetic, open-domain retrieval queries from scientific text.
15
Preprint. Under review.
B.2
Query Decontextualization Prompt PROMPT You are a scientific question rewriter. You are given an original question that references a specific portion of a research paper. Your task is to rewrite it into a context-independent, open-domain scientific query that targets the same underlying concept, without relying on document-local or visual references. Do not refer to any figure, plot, panel, image, document, or use deictic expressions such as this, that, above, or below. Requirements: 1. Preserve the core scientific intent, variables, and conditions present in the original question. 2. Replace visual or deictic phrasing with concept-level wording (e.g., remove references such as “based on the graph” and ask directly about the relationship or effect). 3. If symbols (e.g., f spec ) appear without definition, retain them exactly as written and do not invent meanings. A minimal parenthetical alias may be included only if it appears in the input. 4. Remove all references to figures, plots, tables, panels, or document-local indices. 5. Ensure the rewritten query can be answered by a knowledgeable reader without access to the original document or image. 6. Retain units, ranges, and experimental or observational conditions if present. 7. Avoid unresolved pronouns or placeholders (e.g., “the parameter”, “the system”) unless the domain makes them unambiguous. 8. If the original question contains multiple sub-questions, keep only one and discard the rest. 9. The final query must be a single, concise sentence with no superfluous framing or background. Required Output Format: { "query": "<single rewritten question or null>", "reasoning": "<one-sentence rationale>" }
If a valid context-independent query cannot be produced, set "query" to null and briefly explain why in "reasoning".
Figure 5: Prompt used for decontextualizing document-dependent scientific questions into opendomain queries.
16
Preprint. Under review.
B.3
Query Verification Prompt PROMPT You are a validator that checks whether a decontextualized question is well-formed for open-domain retrieval. Judge only from the provided JSON fields. Do not assume access to the original figure, table, or paper. What “valid decontextualized question” means: A question is valid if and only if all of the following criteria are satisfied: 1. Context-independent: The question contains no references to local context such as “this figure,” “the table above,” “these results,” or any indexical phrasing that requires the original document or image. 2. Answerable in principle: A knowledgeable person or external source could answer the question without access to the original paper or figure. The domain and variables must be sufficiently specified. Crucially, the question must not rely on parameters, symbols, or notations that are defined arbitrarily or only within the source paper (e.g., a tuning parameter with no standard meaning in the field). 3. Intent preserved: The question targets the same underlying information need as the original question, but generalized beyond the local figure or document context. 4. Clarity and unambiguous entities: Any entities, variables, or notations must be interpretable by an expert in the relevant field without requiring the specific paper. Unresolved pronouns or placeholders (e.g., “the parameter,” “the system”) are not allowed unless they are standard and unambiguous within the domain. Guiding Principle for Ambiguity: Requiring background domain knowledge is acceptable and expected for real search queries. However, ambiguity arising from terms that are defined only within the source document or that depend on the original figure context is not acceptable. Common failure modes (label them if present): • underspecified_parameter (especially if defined arbitrarily in the source paper) • still_context_bound • domain_missing_or_vague • ambiguity_pronouns_placeholders • unanswerable_generic Required Output Format (JSON only): { "is_valid": boolean, "score": integer, "decision_rationale": string, "confidence": integer }
Figure 6: Prompt used to verify whether a generated question is a valid, context-independent query suitable for open-domain retrieval.
17
Preprint. Under review.
C
Example Queries arXiv ID: 1007.4239 How can an optical system rapidly and cheaply switch between four orbital angular momentum (helical-phase) mode indices while keeping polarization independent? arXiv ID: 1011.0302 In the dense phase of the O(n) loop model, how does changing the topology of the graph representation determine if a local perturbation is RG-relevant or RG-irrelevant according to Coulomb gas scaling dimensions? arXiv ID: 1806.03783 How does increasing the normalized accretion rate lead to a turnover in the fraction of IR luminosity reprocessed by circumnuclear dust? arXiv ID: 1205.2806 Which ion-neutral atom pair best enables quantum-threshold scattering while being least sensitive to stray static electric fields, i.e., with a high threshold energy but low static-field strength needed for excess micromotion to reach that energy? arXiv ID: 1304.2695 How do the implicit midpoint and trapezoidal (Crank-Nicolson) rules differ in using a single midpoint evaluation versus averaging endpoint evaluations? arXiv ID: 1411.3004 Which subtype of pulsating star shows the strongest positive correlation between variability amplitude and pulsation period across stars spanning a wide range of distances? arXiv ID: 2006.00262 What is the sequence of processing steps that transforms source and target monolingual corpora into mapped cross-lingual word embeddings (CLWEs)?
Figure 7: Representative examples of decontextualized, evidence-grounded queries in TeXODQ. Each query targets a specific piece of scientific evidence.
18
Preprint. Under review.
C.1
Human Annotation Protocol
ANNOTATION GUIDELINES You are evaluating scientific retrieval queries generated from research documents. Queries may be questions or short search-style phrases. Each query must be assessed and, if necessary, revised based on three criteria: naturalness, ambiguity, and document answerability. Criteria: 1. Naturalness: The query must be clear, understandable, and plausible as a realistic search query. A query is considered unnatural if it (i) combines multiple questions (e.g., “What is X, and why Y?”), (ii) contains excessive domain-specific terminology (approximately 8 or more specialized terms), (iii) is overly verbose (e.g., more than three clauses), or (iv) is difficult to interpret. Such queries should be rewritten to improve clarity while preserving the original scientific intent. 2. Ambiguity: The query must be specific enough to identify a single target document. A query is considered ambiguous if it is overly broad, underspecified, admits multiple interpretations, or could be answered by many documents. Ambiguous queries should be rewritten to resolve underspecification and better target the intended document, and if that’s not a possibility, they should be removed. 3. Document Answerability: The query must be directly supported by evidence in the document. The annotator must verify that a specific passage, parsed table, or rendered figure from the LATEX source contains the information needed to answer the query. Queries whose premise is unsupported or contradicted by the document, or for which no identifiable evidence exists, must be removed. Workflow: 1. Perform initial evidence screening using retrieval tools (e.g., NotebookLM or Gemini Flash 2.5) to surface candidate passages, tables, or figures and identify potential issues with naturalness, ambiguity, or answerability. 2. Assign a coarse quality score (1-10) to guide assessment of ambiguity and retrieval specificity; this score is used for calibration and not thresholded directly. 3. Manually inspect the retrieved evidence against the original LATEX source, including text, parsed tables, and rendered figures, to confirm correctness. 4. Rewrite queries to improve clarity and specificity while preserving intent, or discard queries that cannot be made valid.
Figure 8: Human annotation protocol for evaluating and refining scientific retrieval queries.
19