Evaluating Structured Information Extraction with Open Models in a High Risk Public Sector Application

Evaluating Structured Information Extraction with Open Models in a High Risk Public Sector Application Elias Schubert1 and Felix Bießmann1,2[0000−0002−3422−1026]

arXiv:2608.18289v1 [cs.AI] 18 Aug 2026

1

Berlin University of Applied Sciences, Luxemburger Str. 10, 13353 Berlin, Germany {s92538,felix.biessmann}@bht-berlin.de https://www.bht-berlin.de 2 Einstein Center Digital Future, Wilhelmstraße 67, 10117 Berlin, Germany https://www.digital-future.berlin

Abstract. The extraction of structured information from unstructured documents represents a critical component of digital transformations in all sectors. While proprietary solutions dominate commercial applications, a rapidly growing ecosystem of open-source Optical Character Recognition (OCR) engines, Large Language Models (LLMs), and Vision-Language Models (VLMs) offers accessible alternatives. However, systematic evaluations on realistic, multi-step extraction pipelines remain scarce. Responsible usage of such extraction tools require comprehensive evaluations on realistic tasks, especially as these solutions will be key components of applications in the public sector that the EU AI act categorizes as high risk. To address this gap we present a comprehensive benchmark assessing the end-to-end performance of open-source systems on a complex real-world document processing task classified as high risk: Student applications for an international study program. We conduct a comprehensive empirical evaluation with state-of-the-art OCR engines, LLMs and VLMs. Our results reveal that while VLMs generally outperform OCR+LLM pipelines, even state-of-the-art open-source models struggle to handle such tasks reliably in zero-shot settings. Only 4 of 35 configurations achieved F1 scores above 0.5, with the best OCR+LLM pipeline matching top VLM performance, though most OCR+LLM combinations performed substantially worse. Roughly 75% of all configurations scored below 0.25. Model scale influences performance, yet the relationship is non-linear: substantially larger models do not guarantee proportionally better results. Input quality, particularly the structural preservation of OCR output, emerges as a critical factor independent of downstream model capability. These findings provide practical guidance for practitioners selecting models for document processing workflows while highlighting fundamental limitations in current open-source approaches to structured information extraction. Keywords: Optical Character Recognition · Large Language Model · Visual Large Language Model · Benchmark · Open-Source.

2

1

Schubert and Bießmann

Introduction

The ability to extract structured information from unstructured or semi-structured sources has become an increasingly critical capability in modern data-driven workflows. Digital transformations across all sectors require these workflows to reliably integrate information from scanned documents or digitally edited forms. The rapid advancement of Optical Character Recognition (OCR), Large Language Models (LLMs), and Vision-Language Models (VLMs) has fundamentally transformed this landscape, making such extraction tasks more accurate, scalable, and accessible than ever before. Next to proprietary solutions, often marketed as all-encompassing general-purpose models, a rich ecosystem of freeto-use, open-source OCR engines, LLMs, and VLMs has emerged. These openly available models present substantial opportunities for processing large volumes of unstructured data, such as the ubiquitous PDF format, without incurring the costs or restrictions of commercial systems. Leveraging this potential requires more and systematic empirical evaluations of how freely available, off-the-shelf models perform on realistic extraction tasks. This is even more important given that most of this technology has been developing faster than corresponding regulation. While many benchmarks and evaluations focus on tasks that are either simple or not classified as high-risk according to the EU AI Act [6], using the technology in high risk settings such as health care or workflows in the public sector requires extensive empirical evaluations. Even though these kinds of evaluations are more on the applied side of research we argue that such evaluations are contributions that can and should be made by researchers to build trust in open source technology in the public sector. To address this challenge, this work makes two main contributions. For one, we devise an evaluation protocol for extraction performance of a high risk use case in the public sector, student applications in an international study program. This task is representative for many other complex tasks: it requires processing of heterogeneous PDF documents containing a mixture of structured and unstructured data, extracting targeted information, and returning it in a well-defined structured format. This workflow, illustrated in Figure 2 is representative of operational use cases encountered across numerous domains. Second we use this evaluation protocol for a benchmark assessing the capabilities of open OCR systems combined with LLMs, as well as standalone VLMs. The results are discussed with respect to the extraction performance as well as other factors relevant for practical usage, such as computational efficiency.

2

Related Work

While OCR engines, LLMs, and VLMs have each been evaluated extensively in single-step scenarios, benchmarks assessing their individual or combined performance on complex, multi-step extraction tasks in a zero-shot setting remain sparse. Such tasks, which involve converting documents into machine-readable text, extracting specified information, and returning it in a strictly defined structure, closely reflect how these models are likely to be deployed in practice and

Evaluating Open Structured Information Extraction

3

constitute precisely the end-to-end pipeline evaluated in this work. Existing benchmarks tend to isolate individual components of this pipeline. With respect to OCR, Khan et al. [18] evaluated several engines on the CBC Reports Dataset, measuring accuracy, processing time and error rates. PaddleOCR and EasyOCR emerged as the strongest performers, a finding that directly informed the selection of OCR models used in this work. On the side of information extraction from unstructured and semi-structured documents, Ntinopoulos et al. [21] benchmarked mostly LLMs and table transformers on electronic health records in a zero-shot setting, finding that proprietary models consistently achieved strong results with F1 scores above 0.9, outperforming several open-source models. They also found that table transformers achieved higher extraction performance than LLMs when only layout structure is considered, but LLMs surpass table transformers once cell content becomes relevant. Similarly, Gao et al. [7] examined fine-grained extraction across multiple information types, observing that encoder-decoder architectures generalize better to unseen information types, while decoder-only architectures handle unseen task forms more effectively. Notably, model scale did not consistently predict performance, a theme that recurs across the literature. The interplay between OCR and multimodal LLMs for table extraction was investigated by Nunes et al. [22], who benchmarked OCR systems and MLLMs on extracting tables from images using a subset of non-complex tables from PubTables-1M. Although their setup does not include a full end-to-end extraction pipeline, it closely aligns with the benchmark presented here and provides valuable insights. Similar to this work, they enforce structured outputs by prompting models with a predefined Pydantic schema. Evaluating both proprietary and open models, they report strong performance across categories, with several models achieving F1 scores between 85 and 95, depending on the evaluation aspect. Regarding structured output generation, Tenckhoff et al. [28] benchmarked 22 open-source models across five prompting strategies on a synthetically generated dataset, evaluating their ability to extract structured data and produce valid JSON from natural-language text, with GPT-4o included as a proprietary reference. The results reinforce that prompting strategy and model architecture are at least as influential as model scale, and that leading open-source models such as Gemma3-27B can rival proprietary systems. Finally, Roberts et al. [26] assessed VLMs on their ability to extract structure directly from images, using an automatic evaluation approach based on rendering model outputs and comparing them to the original. Closed-weight models, in particular GPT-4o, significantly outperformed open-weight alternatives, and VLMs struggle to pick up visual nuances and were shown to be sensitive to prompt formulation, suggesting that zero-shot standardized prompting may be suboptimal for such tasks. Taken together, these works highlight meaningful progress across the individual components relevant to document extraction pipelines, while also underscoring the absence of benchmarks that evaluate free and open models holistically on

4

Schubert and Bießmann

the complete three-step problem of document ingestion, information extraction and structured output generation that this paper directly addresses.

3

Methods

In the following we first describe the study program application data and the extraction task, followed by a description of the extraction workflow and evaluation metrics. 3.1

Data

The dataset used in this benchmark consists of 100 academic transcripts submitted as part of applications to a German Data Science Master’s programme, provided as PDF documents. Of these, 94 are written entirely in English, three are bilingual in English and Turkish, two are in German, and one combines English and Russian. The vast majority of applicants originate from Asian countries, accounting for over 90% of the dataset. Each document typically contains structured data in the form of tables as well as unstructured elements such as free text, images and institutional sigils. As the documents contain personal information, the dataset cannot be published in its current form and must be anonymized prior to any release. An example document is shown in Figure 1. All documents were standardized to A4 format and additionally converted to JPG images to accommodate models requiring image input. Each document was then manually parsed to extract all courses deemed relevant to the extraction task, yielding a ground truth against which model outputs are evaluated. The extraction follows a ruleset designed to handle the heterogeneity and formatting inconsistencies typical of international academic documents: – Extract only Computer Science or Mathematics courses. – In multi-language documents, prefer the English version. – In case of multiple grade formats, prefer letter grades. – Missing values are filled with N/A. – Without a clear designation of total credits, select the first value in a row. – Formatting inconsistencies such as erroneous spacing or incorrect characters are preserved as-is in the ground truth. Table 1, which shows the ground truth belonging to the document shown in Figure 1, illustrates the application of these rules. With multiple credit columns given and none being explicitly designated, the first credit value for each course is selected. For example, "OPERATING SYSTEMS" appears with both theory (3.0) and practical (1.0) credits in the original document, but only 3.0 is extracted as it is the first credit value associated with that course. The extracted ground truth comprises four columns: academic field, course name, grade, and awarded credits, totaling 952 unique rows across all documents. The number of extracted rows per document ranges from 0 to 44, with no duplicate entries within any single document.

Evaluating Open Structured Information Extraction

5

Fig. 1. Anonymized example document.

Table 1. Manually extracted Ground Truth for the Document in Figure 1 Academic Field Course Name Grade Awarded Credits Mathematics MATHEMATICS FOR COMPUTER B 4.00 THEORY ENGINEERING & INFORMATION TECHNOLOGY Computer Science OPERATING SYSTEMS B 3.00 Computer Science DESIGN AND ANALYSIS OF ALGO- C 3.00 RITHMS Computer Science APPLICATION DEVELOPMENT B 2.00 TOOLS Computer Science PROGRAMMING FOR SCIENTIFIC B+ 3.00 THEORY COMPUTING Computer Science NO SQL DATABASE SYSTEMS B+ 1.00

6

3.2

Schubert and Bießmann

Extraction

We compare the performance of OCR engines combined with LLMs against standalone VLMs in a three-step pipeline: digitalizing and preparing document content, extracting task-relevant information as specified in the prompt, and returning it as a structured output. All LLMs and VLMs are run using the Ollama platform, which provides consistent base settings across all models as well as an integrated format parameter that enforces structured output via a Pydantic schema.

Fig. 2. Workflow of extraction and evaluation pipeline

The OCR engines used in this benchmark were selected based on two criteria: popularity among practitioners, measured by GitHub star count, and performance reported in recent benchmarking literature [9] [20]. All models are used with pretrained weights without any further fine-tuning. The following engines were evaluated: Pytesseract [27], PP-OCRv5 (PaddleOCR) [5], PP-StructureV3 (PaddleOCR) [5], MinerU [30], EasyOCR [11], and docTR [15]. Table 2 provides an overview of each model’s input, output and language configuration. Table 3 details the architecture components for engines where explicit configuration was specified, with selections based on the highest-performing options as indicated in each engine’s official documentation and benchmarks.

Evaluating Open Structured Information Extraction

7

Table 2. OCR model configurations and output formats. Model docTR

Input PDF

Language Config Output Format auto-detect Unstructured; line breaks between text blocks EasyOCR Image (JPG) English Unstructured; tabs between text blocks (manually added) MinerU PDF English Structured; HTML tags for tables, line breaks between text rows PPOCRv5 Image (JPG) English Unstructured; tabs between text blocks (manually added) PPStructureV3 PDF English Structured; HTML tags for tables, line breaks between text blocks Pytesseract Image (JPG) English Structured; line breaks between text rows

Table 3. OCR model architecture components (specified models only). Model Orient. Class. Doc Unwarping Text Detection Textline Orient. Text Recognition docTR — — db_resnet50 — master EasyOCR — — — — — MinerU — — — — — PPOCRv5 PP-LCNet_x1_0_doc_ori UVDoc OCRv5_server_det PP-LCNet_x1_0_textline_ori PP-OCRv5_server_rec PPStructureV3 PP-LCNet_x1_0_doc_ori UVDoc OCRv5_server_det PP-LCNet_x1_0_textline_ori PP-OCRv5_server_rec Pytesseract — — — — —

For the extraction of relevant structured information from the OCR output, a single LLM was deliberately chosen to reduce variables and place the focus on the effect of model scale. Qwen3 [31], a modern model family released in 2025, was selected as it offers a wide range of parameter counts within a single architecture. Five parameter sizes were evaluated, as detailed in Table 4. Table 4. Qwen3 model architecture parameters [31]. Context lengths as specified in Ollama [23]. Q/KV heads refer to query and key-value attention heads [29]. Model Parameters Context Length Layers Heads (Q / KV) Qwen3 0.6B 40K 28 16 / 8 Qwen3 4B 256K 36 32 / 8 Qwen3 14B 40K 40 40 / 8 Qwen3 32B 40K 64 64 / 8 Qwen3 235B 256K 94 64 / 4

VLM based Extraction VLMs were selected following the similar criteria as the OCR engines, drawing from both the most used models from Ollama model zoo and recent literature [14]. An effort was made to keep parameter

8

Schubert and Bießmann

counts comparable across models at approximately 30 billion parameters, though not all models offered variants at this scale. The following models were evaluated: LLaVA (7B) [17], Ministral-3 (14B) [16] , Gemma3 (27B) [8], Qwen2.5VL (32B) [2], and Qwen3VL (32B) [3]. As with the LLM based pipeline, structured outputs were generated using the same Pydantic schema via the Ollama format API. Table 5 provides a detailed overview of each model. Table 5. Vision-language model overview. Model Year Architecture Parameters Context Vision Encoder Gemma 3 2025 Decoder-only 27B 128K SigLIP LLaVA 2024 Encoder-decoder 7B 32K CLIP Ministral 3 2025 Decoder-only 14B 256K ViT Qwen2.5-VL 2025 Encoder-decoder 32B 125K ViT Qwen3-VL 2025 Encoder-decoder 32B 256K SigLIP-2

Prompting Strategy and Experimental Design The prompt design incorporates several evidence-based strategies from recent literature: meaning-typed prompting via Pydantic schemas rather than rigid JSON definitions for reliable structured output [10], a verification phase where the model reviews its own output [25], delimiters to separate schema definitions and text sections [4], and concise, direct instructions [13]. Structured output is enforced via the following Pydantic schema: class Course ( BaseModel ): academic_field : Literal [ " Computer ␣ Science " , " Mathematics " ] course_name : str grade : float | str | None = None awarded_credits : float | None = None class Courses ( BaseModel ): courses : List [ Course ]

The prompt follows a structured format: input data and goal definition, schema presentation with illustrative examples, output rules, context provision, and a verification section with output format specifications. Temperature is set to 0 for all models, following best practices for deterministic structured generation as recommended in the Ollama documentation [24]. All experiments were executed on a compute cluster using a single NVIDIA H200 GPU with 141 GB VRAM, 32 GB system memory, and 8 CPU cores. System memory and CPU utilization remained well below capacity throughout all experiments. For the Qwen3:235b model, an NVIDIA B200 GPU with 192 GB VRAM was briefly utilized to accelerate inference.

Evaluating Open Structured Information Extraction

3.3

9

Evaluation

Model outputs were first validated and normalized to ensure comparability with ground truth. Empty grade and credit cells columns were filled with N/A to match the ground truth format. When models failed to return data in the required structure due to recursive reasoning patterns, additional text generation, inclusion of extra columns, or omission of required fields, an empty dataframe with the correct schema was substituted, effectively treating non-compliant outputs as complete extraction failures. Both the extracted data and ground truth underwent identical normalization procedures: (1) index columns were dropped, (2) leading and trailing whitespace was stripped, (3) academic field and course name values were converted to lowercase, and (4) numerical values were cast to floats, while any alphabetic characters in grade or credit columns were also converted to lowercase. Evaluation Metrics For our evaluation we focus on exact string matches after lowercasing. Both the ground truth and extracted data were tokenized by converting each row, consisting of academic field, course name, grade, and awarded credits, into a single unique token as is illustrated in Figure 3. Given that ground truth contains no duplicate rows, both datasets were represented as sets of row tokens, enabling efficient set-based comparison. From these sets, a combined set of all unique tokens was constructed using set intersection.

Fig. 3. Tokenization visualized

Following standard practice in information or text extraction benchmarks [1, 12, 19] we computed Precision, Recall, and F1 scores. To determine True Positives (TP), False Positives (FP), and False Negatives (FN) for these metrics, we adopt a set-theoretic approach based on Jaccard similarity [1]: – True Positives (TP): Tokens present in both extracted output and ground truth – False Positives (FP): Tokens in extracted output but absent from ground truth – False Negatives (FN): Tokens in ground truth but absent from extracted output In the subsequent analysis, F1 Score serves as the primary metric for model evaluation and comparison, as it provides a balanced measure of both extraction accuracy and completeness, making it particularly suitable for assessing

10

Schubert and Bießmann

performance on imbalanced extraction tasks where both false positives and false negatives carry significant cost. 3.4

Results

In Table 6 we show the overall results for each OCR+LLM and VLM model and Figure 4 shows the best F1 score per model, comparing the top-performing OCR+LLM combination for each OCR engine against all VLMs. VLMs outperform OCR+LLM approaches In the OCR+LLM pipeline, the bestperforming LLM was Qwen3:235B in all cases. The results reveal a clear performance gap between OCR engines, largely attributable to the quality and structure of their output. MinerU stands out as the strongest OCR engine, achieving an F1 score of 0.509, placing it on par with the best-performing VLM, Qwen2.5VL (0.509). This can be attributed to MinerU’s output format, which preserves the spatial layout of the page by grouping text that appears on the same horizontal line into a single row, and representing tabular content using HTML tags to explicitly encode cell boundaries. Other OCR engines, by contrast, insert line breaks after each individual text block regardless of its position on the page, and largely fail to represent table structure semantically. This directly demonstrates the critical importance of input quality to downstream LLM performance: the structure of the OCR output is at least as consequential as the capability of the LLM processing it. On the VLM side, results are polarized. Qwen2.5-VL (0.509) and Ministral-3 (0.508) achieve the highest scores overall, while LLaVA (0.000) and Gemma3 (0.015) perform at or near zero, most likely due to malformed outputs that were treated as complete extraction failures under the evaluation protocol. Notably, Qwen2.5-VL marginally outperforms its successor Qwen3-VL (0.487), suggesting that a newer model version does not necessarily translate to improved task-specific performance. Overall, results across both pipelines are poor: the mean F1 score across all VLMs and the best OCR+LLM combination per engine is 0.282, with the highest observed score capped at 0.509. Parameter Size and Extraction Performance Figure 5 illustrates how F1 score varies across Qwen3 parameter sizes for each OCR engine. A consistent pattern emerges: the 0.6B model performs substantially worse than all larger variants across every OCR engine, frequently producing near-zero scores. This is likely not solely a function of model capacity, but also a known instability of the Qwen3:0.6B model, which has been documented to enter a repetitive generation loop, producing incoherent output that fails to conform to the required Pydantic schema. Beyond the 0.6B outlier, larger models generally yield modest but consistent improvements, with the 235B variant typically achieving the highest score per engine. However, this trend is not universal: for PPOCRv5 and docTR, the 14B model scores marginally higher than the 32B model, reinforcing the finding from related work that model scale and task performance do not maintain a strictly linear relationship.

Evaluating Open Structured Information Extraction

11

Table 6. Extraction performance metrics for OCR+LLM pipeline and VLM models. OCR Engine LLM Model Precision Recall F1 OCR + LLM Pipeline docTR Qwen3 0.6B 0.014 0.022 0.016 docTR Qwen3 4B 0.199 0.217 0.201 docTR Qwen3 14B 0.202 0.238 0.215 docTR Qwen3 32B 0.204 0.222 0.209 docTR Qwen3 235B 0.301 0.329 0.311 EasyOCR Qwen3 0.6B 0.007 0.009 0.008 EasyOCR Qwen3 4B 0.079 0.088 0.080 EasyOCR Qwen3 14B 0.079 0.111 0.088 EasyOCR Qwen3 32B 0.103 0.125 0.108 EasyOCR Qwen3 235B 0.104 0.126 0.112 MinerU Qwen3 0.6B 0.078 0.081 0.075 MinerU Qwen3 4B 0.415 0.472 0.427 MinerU Qwen3 14B 0.455 0.499 0.465 MinerU Qwen3 32B 0.471 0.551 0.501 MinerU Qwen3 235B 0.489 0.542 0.509 PPOCRv5 Qwen3 0.6B 0.002 0.002 0.002 PPOCRv5 Qwen3 4B 0.160 0.199 0.170 PPOCRv5 Qwen3 14B 0.207 0.262 0.222 PPOCRv5 Qwen3 32B 0.163 0.184 0.169 PPOCRv5 Qwen3 235B 0.312 0.345 0.324 PPStructureV3 Qwen3 0.6B 0.022 0.020 0.020 PPStructureV3 Qwen3 4B 0.132 0.146 0.129 PPStructureV3 Qwen3 14B 0.123 0.133 0.124 PPStructureV3 Qwen3 32B 0.157 0.163 0.157 PPStructureV3 Qwen3 235B 0.176 0.191 0.179 Pytesseract Qwen3 0.6B 0.005 0.005 0.005 Pytesseract Qwen3 4B 0.091 0.095 0.092 Pytesseract Qwen3 14B 0.086 0.100 0.091 Pytesseract Qwen3 32B 0.090 0.096 0.092 Pytesseract Qwen3 235B 0.145 0.162 0.151 VLM-Only Pipeline VLM Model Precision Recall F1 Gemma3 27B 0.021 0.018 0.015 LLaVA 7B 0.000 0.000 0.000 Ministral-3 14B 0.480 0.565 0.508 Qwen2.5-VL 32B 0.473 0.574 0.509 Qwen3-VL 32B 0.441 0.573 0.487

12

Schubert and Bießmann

Fig. 4. Best F1 score per model: top OCR+LLM combination (always Qwen3:235B) versus all VLMs.

Fig. 5. F1 scores across Qwen3 parameter sizes for each OCR engine.

Simplified Extraction with Fewer Attributes To investigate the sensitivity of the performance with reduced task complexity, we simplified the task and neglected the grade and awarded credits attributes. Figure 6 presents the improvement in F1 score (∆F1) when the grade column, the awarded credits column, or both are removed from both ground truth and model output prior to evaluation. Positive deltas indicate improved scores under the reduced column setting. Across all model configurations, the largest improvements are observed when both columns

Evaluating Open Structured Information Extraction

13

are removed, retaining only academic field and course name. Removing grades consistently yields a larger improvement than removing awarded credits, though the magnitude varies considerably across configurations, ranging from below 0.01 to above 0.1. This suggests that grade extraction poses a greater challenge, likely due to the diversity of grading formats across transcripts, including both letter and numerical grades, which introduces an additional layer of ambiguity even when explicit handling rules are provided in the prompt. These patterns hold consistently across both VLM and OCR+LLM configurations. Finally, as observed in Figure 5, the 0.6B Qwen3 configurations show minimal improvement even when columns are removed. This further supports the interpretation that the near-zero baseline scores for these configurations are driven primarily by output invalidity due to the model’s repetitive generation behavior, rather than extraction difficulty.

Fig. 6. F1 improvement when removing grade, awarded credits, or both.

14

4

Schubert and Bießmann

Conclusion

This benchmark evaluated the end-to-end capability of open-source OCR engines, LLMs, and VLMs to extract structured information from heterogeneous real-world documents in a zero-shot setting, using all models in their pretrained state with best available configurations. The results reveal that none of the tested approaches can be considered a reliable out-of-the-box solution for complex extraction tasks of this kind, with roughly 75% of all configurations scoring below an F1 of 0.25. A central finding is that the quality of LLM output is strongly dependent on the quality of the context provided. OCR engines that preserve the spatial and structural layout of the source document, as MinerU does via row-wise text grouping and HTML-encoded tables, yield dramatically better downstream extraction results than engines that produce unstructured, block-fragmented output. This underlines that input preparation is at least as important as model capability in such pipelines. VLMs generally outperform OCR+LLM pipelines in this task, as they bypass the OCR step entirely and operate directly on the document image. However, their performance is far from reliable, with several models producing near-zero scores due to malformed outputs. It is also worth noting that a newer model is not always a better one: Qwen2.5-VL outperformed its successor Qwen3-VL on this task, and similarly, larger LLM parameter counts do not guarantee proportionally better results, though very small models such as Qwen3:0.6B are clearly insufficient for tasks of this complexity. These findings apply specifically to the zero-shot, off-the-shelf setting evaluated here. The performance levels observed are not necessarily an inherent ceiling: improved preprocessing and data quality feeding into the LLM or VLM, more targeted prompting strategies, and refined schema definitions are all promising directions that could meaningfully improve results. These do however require additional effort and experimentation, and suggest that deploying such pipelines reliably in practice demands more than simply selecting a capable model off the shelf. Improved data preparation, more targeted prompting strategies, and refined schema definitions are all promising directions that could meaningfully improve performance without requiring fine-tuning, and warrant further investigation. Disclosure of Interests. The authors have no competing interests to declare that are relevant to the content of this article.

References 1. Adhikari, N.S., Agarwal, S.: A comparative study of PDF parsing tools across diverse document categories (2025), https://arxiv.org/abs/2410.09871 2. Bai, S., et al.: Qwen2.5-VL technical report (2025), https://arxiv.org/abs/2502.13923

Evaluating Open Structured Information Extraction

15

3. Bai, S., et al.: Qwen3-VL technical report (2025), https://arxiv.org/abs/2511.21631 4. Chen, L.C., Weng, H.T., Pardeshi, M.S., Chen, C.M., Sheu, R.K., Pai, K.C.: Evaluation of prompt engineering on the performance of a large language model in document information extraction. Electronics 14(11), 2145 (2025). https://doi.org/10.3390/electronics14112145 5. Cui, C., et al: PaddleOCR 3.0 technical report (2025), https://arxiv.org/abs/2507.05595 6. European Parliament and Council: Regulation (eu) 2024/1689 of the european parliament (July 2024), https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng, oJ L 12.7.2024, p. 1–226 7. Gao, J., Zhao, H., Zhang, Y., Wang, W., Yu, C., Xu, R.: Benchmarking large language models with augmented instructions for fine-grained information extraction (2023), https://arxiv.org/abs/2310.05092 8. Gemma Team, Kamath, A., et al.: Gemma 3 technical report (2025), https://arxiv.org/abs/2503.19786 9. Heakl, A., Sohail, A., Ranjan, M., Hossam, R., Ahmad, G.S., El-Geish, M., Maher, O., Shen, Z., Khan, F., Khan, S.: KITAB-Bench: A comprehensive multi-domain benchmark for arabic OCR and document understanding (2025), https://arxiv.org/abs/2502.14949 10. Irugalbandara, C.: Meaning typed prompting: A technique for efficient, reliable structured output generation (2024), https://arxiv.org/abs/2410.18146 11. JaidedAI: EasyOCR: Ready-to-use OCR with 80+ supported languages. https://github.com/JaidedAI/EasyOCR (2020), https://github.com/JaidedAI/EasyOCR 12. Kim, D., Hong, Y.: Figure text extraction in biomedical literature. PloS one 6(1), e15338 (2011). https://doi.org/10.1371/journal.pone.0015338 13. Li, Y.: A practical survey on zero-shot prompt design for in-context learning. In: Proceedings of the Conference Recent Advances in Natural Language Processing Large Language Models for Natural Language Processing. pp. 641–647. INCOMA Ltd. (2023). https://doi.org/10.26615/978-954-452-092-2_069 14. Li, Z., Wu, X., Du, H., Liu, F., Nghiem, H., Shi, G.: A survey of state of the art large vision language models: Alignment, benchmark, evaluations and challenges (2025), https://arxiv.org/abs/2501.02189 15. Liao, H., RoyChowdhury, A., Li, W., Bansal, A., Zhang, Y., Tu, Z., Satzoda, R.K., Manmatha, R., Mahadevan, V.: DocTr: Document transformer for structured information extraction in documents (2023), https://arxiv.org/abs/2307.07929 16. Liu, A.H.e.a.: Ministral 3 (2026), https://arxiv.org/abs/2601.08584 17. Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning (2023), https://arxiv.org/abs/2304.08485 18. Naqi Hadi, M.A., Gul, M., Khan, M., Alwakid, G.N., Zaman Jhanjhi, N.: Benchmarking performance analysis of optical character recognition techniques. In: 2024 26th International Multi-Topic Conference (INMIC). pp. 1–6 (2024). https://doi.org/10.1109/INMIC64792.2024.11004392 19. Nasar, Z., Jaffry, S., Malik, M.: Information extraction from scientific articles: a survey. Scientometrics 117, 1931–1990 (2018). https://doi.org/10.1007/s11192018-2921-5 20. Nazeem, M., R, A., S, N., R. R, R.: Open-source OCR libraries: A comprehensive study for low resource language. In: Proceedings of the 21st International Conference on Natural Language Processing (ICON). pp. 416–421. NLP Association of India (NLPAI) (2024), https://aclanthology.org/2024.icon-1.48/

16

Schubert and Bießmann

21. Ntinopoulos, V., Rodriguez Cetina Biefer, H., Tudorache, I., Papadopoulos, N., Odavic, D., Risteski, P., Haeussler, A., Dzemali, O.: Large language models for data extraction from unstructured and semi-structured electronic health records: a multiple model performance evaluation. BMJ Health & Care Informatics 32(1), e101139 (2025). https://doi.org/10.1136/bmjhci-2024-101139 22. Nunes, G., Rolla, V., Pereira, D., Alves, V., Carreiro, A., Baptista, M.: Benchmarking table extraction: Multimodal LLMs vs traditional OCR. In: Proceedings of the 1st Joint Workshop on Large Language Models and Structure Modeling (XLLM 2025). pp. 8–15. Association for Computational Linguistics, Vienna, Austria (2025). https://doi.org/10.18653/v1/2025.xllm-1.2 23. Ollama: Qwen3 — ollama library. https://ollama.com/library/qwen3 (2025), https://ollama.com/library/qwen3, accessed: 2025-03-03 24. Ollama: Structured outputs — ollama documentation. https://docs.ollama.com/capabilities/structured-outputs (2025), https://docs.ollama.com/capabilities/structured-outputs, accessed: 2025-0303 25. Polak, M.P., Morgan, D.: Extracting accurate materials data from research papers with conversational language models and prompt engineering. Nature Communications 15(1), 1569 (2024). https://doi.org/10.1038/s41467-024-45914-8 26. Roberts, J.S., Lee, T., Wong, C.H., Yasunaga, M., Mai, Y., Liang, P.: Image2Struct: Benchmarking structure extraction for vision-language models. In: Advances in Neural Information Processing Systems. vol. 37, pp. 115058–115097. Curran Associates, Inc. (2024). https://doi.org/10.52202/079017-3653 27. Smith, R.: An overview of the tesseract OCR engine. In: Ninth International Conference on Document Analysis and Recognition (ICDAR). vol. 2, pp. 629–633. IEEE (2007). https://doi.org/10.1109/ICDAR.2007.4376991 28. Tenckhoff, S., Koddenbrock, M., Rodner, E.: LLMStructBench: Benchmarking large language model structured data extraction (2026), https://arxiv.org/abs/2602.14743 29. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need (2023), https://arxiv.org/abs/1706.03762 30. Wang, B., et al.: MinerU: An open-source solution for precise document content extraction (2024), https://arxiv.org/abs/2409.18839 31. Yang, A., et al.: Qwen3 technical report (2025), https://arxiv.org/abs/2505.09388

Related documents

• Integrating Open and Closed Information Extraction: Challenges and First Steps#464577
• OpenAgenet / OAN White Paper: Open Infrastructure for Trusted Agent Interconnection#800057
• Secret Scanner Agent: Extracting Secrets and Access Context from Unstructured Documents#361416
• The Model Proposes, the Code Disposes: A Pre-Registered Ablation of a Verifier-and-Acceptance Stage in an LLM-Orchestrated Offensive-Security Agent#1109923
• The Model Proposes, the Code Disposes: A Pre-Registered Ablation of a Verifier-and-Acceptance Stage in an LLM-Orchestrated Offensive-Security Agent#919275