Wyvern: An Agentic Framework for Generating Grounded Multimodal Reports

Wyvern: An Agentic Framework for Generating Grounded Multimodal Reports Beatrice Alessandra Motetti1,2 , Emilien Guandalino2 , Daniele Jahier Pagliari1 , Alessio Burrello1 , Lorenz K. Müller2 , Konstantin Berestizshevsky2 , Lukas Cavigelli2 1 Politecnico di Torino, Italy 2 Computing Systems Lab, Huawei Research, Switzerland Correspondence: beatrice.motetti@polito.it

arXiv:2608.14446v1 [cs.AI] 14 Aug 2026

User's input topic

Abstract In the current artificial intelligence-driven innovation era, the pace of knowledge growth is accelerating, and is hard to keep up with. While generative models are increasingly used to synthesize content, they often lack in information grounding. To address these peculiarities of our time, we propose Wyvern, a multi-agent framework for the automated generation of grounded, multimodal technical reports. Wyvern allows for the generation of multimodal outputs, integrating images, tables, and text with supporting references in a unified report. Additionally, a particular focus is placed on the grounding of the content, with the implementation of a claims auto-revision stage. We conduct a human evaluation study to assess the quality of our proposed framework. The results show that the figures’ informativeness is perceived as superior to that of a recent baseline in 87% of cases. Furthermore, Wyvern’s reports are rated as more useful than those produced by three alternative methods in 63% to 100% of instances. We also carry out automatic evaluations showing that Wyvern gains up to 2.3× in citation recall and 1.6× in citation precision with respect to the baselines.

1

yvern Web search

Grounded multimodal report

Report generation • Include figures and overview table References

Claims grounding

[1] [2] [3]

• Revise claims wrt cited references

Figure 1: Overview of the main functionalities of Wyvern

cal human judgment, they can ease the burden to efficiently retrieve and manage information. Several frameworks have been proposed recently to automate knowledge retrieval and its organization into machine-generated textual documents (Shao et al., 2024; Wang et al., 2024; Jiang et al., 2024). Furthermore, as multimodality allows for more engaging content (Fu et al., 2022), recent works have also considered the inclusion of relevant images in the output reports to increase their informativeness (Yang et al., 2025, 2026). A key requirement for these frameworks is the ability to ground generated statements on retrieved evidence, thereby increasing user trust. Traceability between claims and sources is fundamental, as it enables to explore with more depth the aspects of interest. Building on these observations, we propose Wyvern, a multi-agent framework for the automatic generation of multimodal technical reports on userselected topics, with a particular focus on content groundedness. Our Wyvern framework capitalizes on a structured workflow of specialized Large Language Model (LLM)-based agents, each focused on a narrow task, such as refining the quality of previously acquired information. Organizing these agents into a coordinated framework significantly enhances reliability, modularity, and overall quality of the produced content, as recent works have demonstrated (Huang et al., 2024; He et al., 2025).

Introduction

The volume of newly published content, spanning from scientific literature to general webpages, is rapidly growing, particularly in fields such as artificial intelligence (Bornmann et al., 2021). Keeping up with the vast quantity of heterogeneous domainspecific information has become extremely critical and complex simultaneously, especially for researchers (Park et al., 2023; Jones, 2009). In this context, automated synthesis and summarization tools can act as first-pass filters that retrieve, aggregate and condense large and relevant information into technical reports, helping researchers and endusers to stay informed and keep up in fast-evolving fields. While these tools cannot substitute criti1

Images Claims insertion verification STORM (Shao et al., 2024) ✗ ✗ WebThinker (Li et al., 2025) ✗ ✗ WikiAutoGen (Yang et al., 2025) ✓ ✗ Wyvern (ours) ✓ ✓

As illustrated in Figure 1, Wyvern comprises various modules that search and process relevant information from the web, produce the multimodal report, and verify whether the textual statements are supported by the previously retrieved evidence. We evaluate Wyvern through a human evaluation study, comparing its generated reports with those produced by STORM (Shao et al., 2024), WebThinker (Li et al., 2025) and WikiAutoGen (Yang et al., 2025). The results show that Wyvern produces reports whose figures and factuality are more positively evaluated by the end-users. We also conduct an automatic evaluation, and examine the predictive power of different evaluation means with respect to human judgment. Our key contributions can be summarized as follows:

Method

Table 1: Overview of expository text generators. Wyvern integrates multimodality and explicit claims verification and revision into a unified framework.

outline definition and a subsequent section expansion phase (Shao et al., 2024; Jiang et al., 2024; Yang et al., 2025; Wang et al., 2024; Yang et al., 2026); others employ a more dynamic planning strategy that builds upon recursive task decomposition principles (Li et al., 2025; Xiong et al., 2025). Specifically, STORM (Shao et al., 2024) and CoSTORM (Jiang et al., 2024) employ ensembles of multi-perspective agents with a question-andanswer approach for information retrieval and outline planning tasks, generating Wikipedia-like articles. AutoSurvey (Wang et al., 2024) tackles the problem of creating literature surveys with a fourstage framework comprising information retrieval and outline definition, and subsections drafting, integration, and evaluation. WebThinker (Li et al., 2025) integrates the capabilities of performing web searches and report writing and editing within the reasoning of Large Reasoning Models. Another emerging line of works tackles the generation of multimodal reports, incorporating retrieved images within the textual body (Yang et al., 2025, 2026). In WikiAutoGen (Yang et al., 2025) an agent proposes candidate positions and descriptions of images, then retrieved from the web. Multimodal DeepResearcher (Yang et al., 2026) uses instead a structured textual representation of charts to allow for the direct generation of visualizations, by using a LLM to produce JavaScript code. We report in Table 1 an overview of the features of the main works.

• We propose Wyvern, a multi-agent framework for the generation of grounded, multimodal technical reports, retrieving the most relevant information from the web. As key novelties: – we design a module for the retrieval, selection, and positioning of the most informative images within the textual report; – we build a claims auto-revision stage to verify the grounding of the statements on the collected evidence. • We assess the quality of Wyvern-generated reports through a human evaluation study and we compare these results with automatic evaluations, to analyze the predictive performance of the latter for human judgments. Code and data are available at https://github .com/huawei-csl/wyvern.

2

Related Work

2.1

Automated long-form expository writing

The generation of long text by LLMs is a complex task, due to their limited context length and the need to maintain semantic consistency (Wang et al., 2024; Yang et al., 2025). To overcome the limitations of static parametric knowledge, a common approach is to integrate Retrieval-Augmented Generation (RAG) strategies (Lewis et al., 2020), often combined with web search engines (Shao et al., 2024; Li et al., 2025), to perform an up-to-date and comprehensive information retrieval. Many recent works on long-form expository writing adopt a top-down approach, with an initial

2.2

Grounding

Measuring and improving the factuality of text generated by LLMs is an active area of research, with two main directions (Jacovi et al., 2025): one defines factuality as text entailment with respect to cited sources; the other verifies claims against external ground-truth answers. Additionally, various definitions of claim unit exist, with some works identifying it at a sentence level (Gao et al., 2023; Shao et al., 2024; Balepur et al., 2023), and others 2

and description to generate n search queries. For each search query, it retrieves the top-k1 URLs from a search engine. The type of webpage content determines the selection of the most suitable parser, which is used to extract the URL’s textual content into Markdown format. To assure content quality and processability, only documents whose length falls within a predefined token range are retained. This removes both empty pages and documents that are too large for the LLM’s context length. To refine the set of documents to be included in the knowledge database, Wyvern employs an ensemble of independent agents that analyze the retrieved information. Agent 2 assigns a score on a 10-point scale to assess the completeness of the extracted Markdown content for each webpage, in parallel. Documents with a completeness score below a predefined threshold λcomp are excluded from the references base, thereby preserving content quality by removing the documents for which the parser failed to extract meaningful text. Agent 3 evaluates, for each webpage, the need for an additional search step to retrieve additional relevant material. Similarly to agent 2 , it assigns a score on a 10point scale to each document, estimating whether additional informative content can be fetched with a new search. If the score exceeds a set threshold λsearch , agent 4 generates a candidate search query, which is used to fetch the most relevant URL. This URL is then considered for inclusion in the reference base, following the parsing and completeness evaluation procedure carried out by agent 2 . To broaden the coverage of relevant aspects, Wyvern analyzes the embedded links within the parsed documents. This is done in a two-stage process. First, the embedded hyperlinks are extracted from each collected resource by agent 5 , and the relative links are resolved to their absolute versions. Afterwards, for each document, agent 6 is provided with the textual content, the list of extracted hyperlinks, and the user-provided topic of interest, and is tasked with assigning a relevance score to each link. Links scoring below a set threshold λlink are considered out of scope, while the top-k2 by score are processed, following the same content extraction process as the initial documents. This iterative exploration process enables Wyvern to dynamically enrich the collected knowledge base. To minimize redundancy in the reference base, a deduplication step is carried out to remove the most similar documents (e.g., earlier drafts or alternative versions of the same paper or webpage). Specifi-

at a finer granularity by decomposing sentences into atomic facts (Min et al., 2023; Li et al., 2024). FActSCORE (Min et al., 2023) decomposes sentences into atomic facts, to overcome the challenge of assigning a binary entailment label to sentences with multiple pieces of information. SelfChecker (Li et al., 2024) first decomposes the text into claims, and then generates queries to retrieve relevant content to verify them. Chain-ofVerification (Dhuliawala et al., 2024) mitigates hallucinations in baseline LLMs responses by generating a list of verification questions for the key claims, and then revising the responses via self-validation using the model’s parametric knowledge. Gao et al. (2023) measure groundedness with respect to the cited sources by computing the citation recall and precision over sentence-level statements. Differently, we define the citation quality metrics on atomic claims derived from the full-sentences, avoiding cases of partial support.

3

Proposed framework

Wyvern comprises distinct modules that operate sequentially on individual macro-tasks (see Figure 2). Each of the modules handles a specific phase of the report generation by relying on ensembles of agents. In particular, Wyvern integrates: (i) a search module, that is tasked with the retrieval and processing of relevant information; (ii) a report generation module, that writes the report and integrates it with the most informative figures from the retrieved documents, and with an overview table; (iii) a grounding module, which examines all the statements of the report with the aim of improving its factuality and limiting the hallucinations and the unsupported statements. In the following, we refer to each agent involved in our framework by the subscript number next to its icon in Figure 2. 3.1

Search phase

The goal of the search phase is to acquire information from the web and construct a knowledge database consisting in a set of references R and their corresponding content summaries RS . The search begins with the user providing Wyvern with a topic title, optionally specifying aspects of interest or non-interest and a list of relevant URLs. Afterwards a LLM generates a refined topic title and a detailed description of it, and the URLs are directly incorporated into the initial references base R. Wyvern’s agent 1 then leverages the topic title 3

• Topic title • Topic description [optional]

Search Initial retrieval Additional search

Report generation

Search queries generation

Document completeness eval.

2

Additional search eval. & search query generation

4

Embedded links extraction & scoring

5

Summarization

7

Relevance filtering m

Agent m

n

3

Outline

Two-step outline generation

9

Sections

Parallel section expansion

11

Images

Description generation 12 Figures selection & placement Insertion 14

6

8

Refinement

10

13

Table building 15 Placement 16 Insertion 17

Overview table

Deduplication step

Filtering

1

Retrieval via search engine & parsing

Section-by-section refinement

Agent n (with Chain-of-Thought reasoning LLM as base model)

Final report

Report draft

References

18

•

Grounding Text decomposition into atomic claims

Grounding check

Two-stage grounding assessment wrt references

19

20

Claims revision

Revision/removal of the unsupported claims

21

Headings revision

Headings refinement wrt the new text version

22

Auxiliary tool

Figure 2: Overview of the key components of Wyvern, our proposed framework. The search module (left) retrieves relevant documents from the web. The report generation module (middle) handles of the writing of the report’s text and of the figures insertion. The grounding module (right) verifies if the claims are supported by the cited references, and possibly revises them. Figure 1 1 Figure

3.2.1

Document 1 Document 2

12

Description generation

Figure FigureN11 • Caption • Text snippets • Title • Summary

12

Description generation

…

3.2

• Caption • Text snippets • Title • Summary …

cally, the Jaccard similarity index between all pairs of documents is computed. For each pair exceeding a similarity threshold λsim , only the longer document is kept in the final reference base R. Once the reference base is finalized, agent 7 summarizes each document di ∈ R, thus creating the set of reference summaries RS that will be used for report generation. Eventually, agent 8 , based on a Chain-of-Thought reasoning LLM, conducts a final filtering step to detect semantic outliers in RS , i.e. documents that are off-topic or partially misaligned with the original topic description. The agent considers all the summaries of the references simultaneously, to infer the underlying domain, overcoming the limitation of relying only on the LLM’s static knowledge about the input topic. In this way, documents which are not aligned to the topic can be flagged as outliers and excluded from both RS and R, allowing for more focused content.

Figure FigureN12 • Caption • Text snippets • Title • Summary

12

Description generation

13

＞ Selection of the most relevant figures ＞ Assignment of each figure to the most compatible section

For each section: 14

＞ Image positioning within the section ＞ Text revision for seamless imagetext integration

Figure 3: Structure of the figures selection and insertion routine. The pipeline comprises the generation of the descriptions of all the images extracted from the sources, the selection of the most relevant figures, and their integration in the report.

with full access to the references summaries set RS . For each first-level heading, the agent writes the section and its associated subsections, following the outline O. The agent is thus responsible for selecting the most salient content and placing citations to the supporting references. Eventually, the concatenation of all the expanded sections is performed to obtain the draft of the report.

Report generation Outline creation and sections expansion

Similarly to prior works (Shao et al., 2024; Wang et al., 2024), the generation of the textual content of the report follows a two-stage process, comprising outline creation and section expansion. The adopted approach is inspired by the explorationexploitation trade-off. The set of summarized references RS is used by agent 9 to generate No draft outlines {O1 , ..., ONo } to maximize topic coverage. Agent 10 then merges the generated outline versions to produce a final outline O including the most informative aspects of the explored options. Once the outline O is produced, agent 11 generates the text of each section s ∈ O simultaneously,

3.2.2 Image insertion Once the textual content of the report is generated, the workflow shifts to selecting, placing, and integrating figures within the report draft. The pipeline is illustrated in Figure 3. The pool of candidate images, denoted as F , comprises all the figures extracted from source documents. Agent 12 is tasked with generating a description df for each candidate figure f ∈ F . Only textual elements are used for the elaboration of the detailed depiction. Specifically, the agent retrieves the reference document title, its summary rs ∈ RS , 4

the caption of the figure within the full-text document r ∈ R, and any in-text excerpt referring to the figure identifier extracted from the caption. Subsequently, reasoning agent 13 evaluates all the figures’ descriptions together, in conjunction with the report outline O, and selects the most relevant and informative images. If the length of the concatenation of the descriptions exceeds the context length of the LLM, this process is conducted hierarchically. The agent provides as output the set F̂ of relevant figures, and assigns each of them to the most pertinent section of the report. Then, reasoning agent 14 carries out the placement of the figures, independently for each section. Given the set of figures F̂s ⊆ F̂ assigned to section s, the agent has to assess their individual relevance for the specific section text, and images deemed as potentially out of scope are discarded. The key difference with respect to the selection performed a priori by agent 13 is that the reasoning agent has access to the section text, thus has more elements to filter out images which do not convey useful information for the textual flow. Eventually, agent 14 completes the images insertion by generating their captions and revising the section’s text to integrate the figures seamlessly within the narrative. 3.2.3

Figure 4: Overview of the grounding verification pipeline. The grounding of the claims is evaluated with their associated references, and then possibly revised.

section, independently, to identify potential gaps or inconsistencies. When needed, it retrieves and inserts supplementary information from the references set RS to improve the depth and clarity of the section under consideration. When new content is inserted, the agent checks and possibly revises the citations. Additionally, it edits the text to ensure that the tone of the report remains neutral. 3.3

Once the report has been generated, an ensemble of agents performs a final groundedness enhancement step. The objective of this phase is to review all the statements in the report and revise or remove them if they are not supported by the cited references. This is achieved by first decomposing the text into atomic claims and then evaluating each of them independently, as illustrated in Figure 4. More in detail, the text is segmented into a set T of distinct text units, with paragraph-level granularity. For each text unit ti ∈ T , the cited references constitute the references set RSi ⊆ RS for such textual element, which forms the basis for the subsequent grounding verification and content correction. Each text unit ti is then further decomposed by agent 19 into a set Ci of atomic textual claims, that all share the same reference set RSi . Following the decomposition, reasoning agent 20 evaluates the grounding of each atomic claim cj ∈ Ci with respect to the support reference set, independently. First, it checks the claim against the set RSi , comprising the summaries of the associated references. If the claim is flagged as potentially not supported, the agent performs a second, more detailed entailment check with respect to the full-text of the references Ri , and the final Natural Language Inference (NLI) outcome is based on the second evaluation round. Each claim is also accompanied by a brief rationale that explains the agent’s judgment. Once all the atomic claims have been evaluated, agent 21 carries out the revision routine. The agent operates on each text unit independently, but takes

Overview table insertion

To further enrich the report’s content, Wyvern inserts an overview table summarizing the key concepts. First, reasoning agent 15 analyzes the references set RS , and extracts shared and informative aspects to differentiate among the relevant works. Based on this analysis, it builds a comparative table, along with a detailed explanation of its contents. Subsequently, agent 16 determines the most appropriate placement of the table within the outline O, and outputs the name of the section in which the table should be integrated. Finally, agent 17 performs the actual table insertion. It receives the target section’s text, the table and the accompanying explanation generated by agent 15 . Its tasks are identifying the optimal insertion point within the specific section’s text, generating a caption for the table, and editing the text of the section to ensure a meaningful content integration. 3.2.4

Grounding

Refinement

Following the writing of the text and the inclusion of visual content, agent 18 performs a final review to enhance the completeness and coherence of the report. In particular, it analyzes the content of each 5

into account the broader context of the section to which the text unit belongs, in order to avoid redundancy or inconsistency during the refinement of the statements. Specifically, the agent considers the entire text of ti ’s section, the set of unsupported unsupported claims Ci ⊆ Ci to be modified within ti along with their accompanying grounding explanations, and the summaries of the references RSi . The agent then localizes each claim within the given text unit, and either revises it, in case it is only imprecise, or completely removes it. Finally, agent 22 operates on all the first-level sections independently, revising headings and subheadings to ensure consistency between sections’ titles and content after the claims amendment.

4

Experiments

4.1

Experimental setup

producibility reasons, as proprietary models can be continuously updated over time, hindering the possibility of fair comparisons without a fixed snapshot of the models’ weights. For WebThinker, we employ the Serper retriever and the same open-source models as in (Li et al., 2025), i.e. WebThinkerQwQ-32B, a fine-tuned version of QwQ-32B, as reasoning model, and Qwen2.5-32B-Instruct as assistant model. For WikiAutoGen, we use the Serper retriever and replace proprietary models with opensource alternatives, namely Pixtral Large (Mistral AI, 2024), DeepSeek-R1 and DeepSeek-V3. For all the baselines, we use as input for the report generation the concatenation of the topic title and the detailed description of the aspects of interest.

4.3

We implement our framework with LangChain. For each input topic, we generate n = 30 search queries. We use the Serper1 search API for the retrieval of the top-k1 relevant resources, with k1 = 3. For the embedded links extraction, we set k2 = 40. We set the threshold values λcomp and λsearch to 5, λlink to 7, and λsim to 0.9. For the parsing of the PDF-based documents we use Docling (Auer et al., 2024), which allows also the extraction of figures, while we use the Playwright2 Python library and Trafilatura (Barbaresi, 2021) for the other types of sources. We parse Wikipedia pages with the MediaWiki Action API3 and the html2text4 library. We employ DeepSeek-R1 (DeepSeek-AI, 2025) to build all reasoning agents, and DeepSeekV3 (DeepSeek-AI, 2024) for all the other agents, both with temperature equal to 0. We employ as evaluator models DeepSeek-V3 and Qwen332B (Qwen Team, 2025), with temperature of 1 and 0.6 respectively. 4.2

Human evaluation

To evaluate the quality of the reports, we conduct a user study involving 27 volunteers from diverse corners of computing systems research, including senior researchers, engineers and Ph.D. students. To have evaluations of high technical quality, we ask each participant to select a topic of interest. We then provide each reviewer with two reports, one generated using Wyvern and one using a baseline method. We obtain nine distinct comparisons of Wyvern with respect to each baseline. We do not mention the used methodologies and we randomize the order between the two reports to ensure the absence of positional bias in the aggregated results. We structure the questionnaire for the human evaluation in two main parts, i.e. the relative assessment and the absolute grading. The relative assessment consists of a pairwise evaluation of two reports on various rubrics, concerning the structure, relevance, coverage, content presentation, figures and tables informativeness, engagement, and usefulness of the report. The main objective of the relative evaluation is to directly compare our method with the baselines. Absolute grading involves a more detailed evaluation on the same criteria, but applied to a single report, whose generation method is unknown to the participant. In particular, each participant assigns a score on a 5-point Likert scale (Likert, 1932) to every statement in the questionnaire. This section of the evaluation enables a more precise assessment of the quality of the report generated with our proposed methodology.

Baselines

We compare our method with STORM (Shao et al., 2024), WebThinker (Li et al., 2025) and WikiAutoGen (Yang et al., 2025). For STORM, we use the Serper retriever with the default parameters as implemented by (Shao et al., 2024), and we replace the proprietary models with the open-source DeepSeek-V3. This substitution was made for re1

https://serper.dev/ https://playwright.dev/ 3 https://www.mediawiki.org/wiki/API:Main_page 4 https://pypi.org/project/html2text/ 2

6

4.4

Preference rate for our method (%)

Automatic evaluation method

4.4.1

0

Grounding

vs STORM 100.00 100.00 100.00 92.86

PN −1 i=0

f (ci , Ri ) N

uc Str

4.4.2

(3)

Report quality

We employ the evaluation rubrics proposed by Shao et al. (2024), concerning the interest level of the report, its coherence and organization, relevance and focus, and broad coverage. We frame the evaluation as pairwise relative assessments, as in the human evaluation study. Namely, we provide as input to the evaluator models two reports, one generated by Wyvern, and the other by a baseline. We consider both order permutations, and we experimentally assess their effect on automated evaluation outcomes.

5

Results

5.1

Human evaluation

5.1.1

N/A 100.00 100.00 100.00 100.00 N/A

66.67 75.00 37.50 62.50

t t s s y e e n e tur nten erag tatio igure Tabl tualit men lnes e c fu F Co Cov sen Fa ngag Use e r P E

the reports, extracted from 23 completed questionnaires (out of the 27 distributed). Each cell contains the preference percentage of our method with respect to a considered baseline on a specific rubric. We cluster the various criteria that we ask the participants to evaluate into macro-areas, to provide a high-level overview. Additional details are given in the Appendices. Wyvern generates reports that surpass the ones produced by STORM on all the considered criteria. Additionally, it outperforms in 86.96% of cases WikiAutoGen on the figures quality (informativeness, positioning, captioning). WebThinker achieves a higher preference rate (62.50%) on the questions concerning the structure and the engagement level created by the report. While this highlights directions of improvement, Wyvern manages to produce reports that are qualitatively better on all the other considered aspects. In particular, users rate Wyvern’s reports as more useful than STORM’s, WebThinker’s and WikiAutoGen’s ones in 100.00%, 62.50% and 87.50% of cases, respectively.

PN −1 P|Ri |−1

f (ci , Ri ) ∧ g(ci , rk ) PN −1 i=0 |Ri |

100

(1)

where [·] denotes the Iverson bracket, which evaluates to 1 if the enclosed logical condition is true and 0 otherwise. The overall citation precision, considering all the claims and the associated citations across all paragraphs in a document can be computed as: k=0

80

Figure 5: Relative grading of Wyvern with respect to selected baselines in the human evaluation study. Each cell reports the preference percentage for Wyvern over the baseline on a rubric. STORM and WebThinker cannot be compared on the Figures criteria, as they produce text-only reports.

g(ci ,rk ) = [[f (ci ,rk ) = 1]∨[f (ci ,Ri \{rk }) = 0]] = 1 (2)

i=0

60

vs WikiAutoGen 75.00 75.00 87.50 68.75 86.96 86.36 87.50 75.00 87.50

where f (ci , Ri ) is the binary output of the NLI model for claim i, that assumes value 1 if the concatenation of the references Ri supports the claim ci , and 0 otherwise. The citation precision measures the relevance of the citations in supporting the claims of the report. A citation rk ∈ Ri is considered relevant if:

CP =

40

vs WebThinker 37.50 68.75 75.00 62.50

To measure the verifiability of the statements in the report we employ the citation recall and the citation precision, as in previous works (Gao et al., 2023; Shao et al., 2024; Wang et al., 2024). The citation recall CR measures the number of claims in the report that are supported by the cited references, i.e. given N claims, where each claim ci is associated to a set of cited references Ri : CR =

20

5.1.2

Absolute grading

To investigate the perceived quality of our framework, we include in the human evaluation study a finer-grained absolute scoring task of the reports generated by Wyvern. Figure 6 shows the results of this absolute grading, on the same rubrics considered in Figure 5. Wyvern exhibits the highest scores on the rubrics concerning factuality and the overview table, with an absolute score of 4.22 and 4.25 out of 5, respectively. The engagement rubric receives the lowest score overall, equal to 3.65. However, it is also visible how the standard deviation of this criterion is the highest, highlighting the subjectivity of the question.

Pairwise relative comparisons

We report in Figure 5 the results of the evaluation study on the pairwise comparison between 7

3.87

4.07

4.13

4.20

4.25

4.16

4.22

4.00

3.65

Win ratio [human] (%)

Score

5 4 3 2 1

t t e n e res Table uality men lness tio ten ag tur t uc Con over enta Figu e c r fu g t a e s S F C ga Us Pre En

Figure 6: Results of the absolute grading of Wyvern’s reports from the human evaluation study

Value (%)

Recall

100

75.45

70.32 52.29

31.45

Ours

STORM

47.91

Automatic evaluation

5.2.1

Grounding metrics

60 40 20 0

5.2.2

0 20 40 60 80 100

0 20 40 60 80 100

RUBRIC Interest Level Coherence & Organization Relevance & Focus Broad Coverage

Before (%) 60.92 ± 6.10

After (%) 73.79 ± 5.44

Gain 1.21×

84.34 ± 4.24

94.64 ± 1.55

1.12×

67.15 ± 6.17

75.45 ± 7.54

1.12×

Pairwise relative comparisons

We perform an automatic evaluation, framed in a relative assessment scheme, adopting the four rubrics presented in (Shao et al., 2024), that cover the aspects of interest level, coherence and organization, relevance and focus, and broad coverage. We consider as evaluator models DeepSeek-V3 and Qwen3-32B. The goal is to compare the results obtained with the automatic evaluation with the ones of the human evaluation study, to assess whether the first can be used as proxy for the latter, being more scalable and cost- and time-effective. Figure 8 reports the comparison between the results obtained over all the reports with different evaluator models, on the four criteria of judgment, and the human evaluation scores of the respective semantically equivalent question. The automatic means of evaluation (whose scores are reported on the x-axis) achieve a good correlation degree with respect to the human-assigned scores (on the y-axis) for STORM and WikiAutoGen. However, they fail to capture the scores of WebThinker, especially on the Broad Coverage and Relevance & Focus criteria. This highlights how LLM-based automatic evaluation of reports on qualitative rubrics still lacks behind in capturing the trends of the human-assigned evaluations, in particular when a clear preference gap is not always present in the latter (as it emerges in Figure 5). We extend our analysis by performing experiments to assess whether the evaluator models exhibit ordering bias, i.e. if the order in which the reports are passed as input to the model influences the final score. We consider two evaluator models, DeepSeek-V3 and Qwen3-32B. Ordering bias appears in 36.11% and 30.56% of the comparisons for DeepSeek-V3 and Qwen3-32B, respectively. This

We report in Figure 7 the citation recall and precision computed over the set of generated reports for each considered baseline. We were unable to compute such metrics for WebThinker, as it does not include citations in the report’s text. Wyvern produces reports with higher grounding metrics than STORM and WikiAutoGen. In particular, it reaches 73.79% of citation recall (+21.50 and +42.34 percentage points with respect to STORM and WikiAutoGen), and 94.64% when considering only the claims with citations, thus excluding paragraphs without citations from the analysis. We further analyze the impact of the claims revision module on the grounding metrics computed over all the distributed reports in Table 2. Both citation recall and precision show a significant improvement after the application of this step. In particular, the citation recall increases from 60.92% to 73.79%, highlighting the positive effect of the claims revision stage on Wyvern’s final output. Metric Cit. recall Cit. recall w/o uncited claims Cit. precision

80

vs STORM vs WebThinker vs WikiAutoGen

WikiAutoGen

Figure 7: Citation quality metrics computed on all the reports generated by the various methods (27 documents in total for ours, 9 for STORM and WikiAutoGen). The error bar shows the standard deviation.

5.2

100

VS B ASELINE

Figure 8: Comparison of human (y-axis) and automatic (x-axis) evaluation results on the four rubrics with two different evaluator models. The diagonal line represents a 1:1 correspondence between human and automatic evaluation.

Precision 60.87

55.17

Qwen3-32B

Win ratio [auto] (%) Win ratio [auto] (%)

94.64 73.79

50 0

Recall w/o uncited claims

DeepSeek-V3

Table 2: Impact of the claims revision module on the citations quality metrics

8

demonstrates that there is still an ample margin of improvement for achieving robustness in this type of automated evaluation.

eralization capability of the framework to multilingual settings should be analysed in future works.

6

Wyvern builds upon content retrieved from the web, which may inherently contain systematic biases as well as scientific inaccuracies. Furthermore, we rely on existing open-source LLMs that might not have been fully aligned with the ethical values of the reader. Currently, our framework does not incorporate an explicit mechanism for filtering or correcting biased or erroneous information. However, Wyvern does not amplify these issues beyond their presence in the original sources.

Ethical considerations

Conclusions

We presented Wyvern, a multi-agent framework that enables the generation of grounded, multimodal technical reports on given input topics. We designed different modules, composed of ensembles of LLM-based agents, to handle the various stages of the report generation, including figures inclusion. Furthermore, with the integration of a final stage of text decomposition into atomic claims, verified on the collected evidence, Wyvern allows for an improvement in citation recall up to 2.3× with respect to the other considered methods, namely STORM, WebThinker, and WikiAutoGen. We conducted a human evaluation study to assess the quality of the reports generated by Wyvern, showing that their usefulness is perceived as higher in 63% to 100% of the instances, depending on the reference baseline. We additionally showed how the automatic means of evaluation lack accuracy and robustness in estimating human preference.

Acknowledgments We thank the evaluators who participated in the human evaluation study for providing their truthful judgment of the generated reports quality. This publication is part of the project PNRRNGEU which has received funding from the MUR – DM 118/2023.

References

Limitations

Christoph Auer, Maksym Lysak, Ahmed Nassar, Michele Dolfi, Nikolaos Livathinos, Panos Vagenas, Cesar Berrospi Ramis, Matteo Omenetti, Fabian Lindlbauer, Kasper Dinkla, Lokesh Mishra, Yusik Kim, Shubham Gupta, Rafael Teixeira de Lima, Valery Weber, Lucas Morin, Ingmar Meijer, Viktor Kuropiatnyk, and Peter W. J. Staar. 2024. Docling technical report. Preprint, arXiv:2408.09869.

Wyvern relies on auxiliary search APIs to collect relevant documents from the web. Although our proposed framework includes specific sources filtering and selection pipelines, the generated multimodal report ultimately depends on the quality of the webpages retrieved by the search engine. To achieve broad coverage on the user-selected topic, Wyvern generates multiple, distinct search queries. However, this approach and the reliance on the search APIs do not always ensure a comprehensive and optimal coverage of the available information. Additionally, given the inherent characteristics of search APIs, reproducibility is not fully guaranteed, as search engines may return different results according to the inquiry time, location, and the always evolving indexing policies. We evaluated Wyvern’s performance exclusively on topics formulated in English language. Although a translator module could be inserted at the beginning of the pipeline to translate a topic expressed in any language to English (e.g., with an additional LLM-based agent), the generation of search queries in English language for the information retrieval penalizes (but not necessarily excludes) resources in other languages. The gen-

Nishant Balepur, Jie Huang, and Kevin Chang. 2023. Expository text generation: Imitate, retrieve, paraphrase. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 11896–11919, Singapore. Association for Computational Linguistics. Adrien Barbaresi. 2021. Trafilatura: A web scraping library and command-line tool for text discovery and extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations, pages 122–131, Online. Association for Computational Linguistics. Lutz Bornmann, Robin Haunschild, and Rüdiger Mutz. 2021. Growth rates of modern science: a latent piecewise growth curve approach to model publication numbers from established and new literature databases. Humanities and Social Sciences Communications, 8(1):224. DeepSeek-AI. 2024. Deepseek-v3 technical report. Preprint, arXiv:2412.19437.

9

DeepSeek-AI. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948.

Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024b. Prometheus 2: An open source language model specialized in evaluating other language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4334–4353, Miami, Florida, USA. Association for Computational Linguistics.

Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. 2024. Chain-of-verification reduces hallucination in large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 3563–3578, Bangkok, Thailand. Association for Computational Linguistics.

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledgeintensive nlp tasks. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 9459–9474. Curran Associates, Inc.

Tsu-Jui Fu, William Yang Wang, Daniel McDuff, and Yale Song. 2022. DOC2PPT: Automatic Presentation Slides Generation from Scientific Documents. Proceedings of the AAAI Conference on Artificial Intelligence, 36(1):634–642. Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023. Enabling Large Language Models to Generate Text with Citations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6465–6488, Singapore. Association for Computational Linguistics.

Miaoran Li, Baolin Peng, Michel Galley, Jianfeng Gao, and Zhu Zhang. 2024. Self-checker: Plug-and-play modules for fact-checking with large language models. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 163–181, Mexico City, Mexico. Association for Computational Linguistics.

Junda He, Christoph Treude, and David Lo. 2025. Llmbased multi-agent systems for software engineering: Literature review, vision, and the road ahead. ACM Trans. Softw. Eng. Methodol., 34(5).

Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yutao Zhu, Yongkang Wu, Ji-Rong Wen, and Zhicheng Dou. 2025. Webthinker: Empowering large reasoning models with deep research capability. In Advances in Neural Information Processing Systems (NeurIPS).

Dong Huang, Jie M. Zhang, Michael Luck, Qingwen Bu, Yuhao Qing, and Heming Cui. 2024. Agentcoder: Multi-agent-based code generation with iterative testing and optimisation. Preprint, arXiv:2312.13010.

Rensis Likert. 1932. A technique for the measurement of attitudes. Archives of Psychology, 140:1–55.

Alon Jacovi, Andrew Wang, Chris Alberti, Connie Tao, Jon Lipovetz, Kate Olszewska, Lukas Haas, Michelle Liu, Nate Keating, Adam Bloniarz, Carl Saroufim, Corey Fry, Dror Marcus, Doron Kukliansky, Gaurav Singh Tomar, James Swirhun, Jinwei Xing, Lily Wang, Madhu Gurumurthy, and 7 others. 2025. The facts grounding leaderboard: Benchmarking llms’ ability to ground responses to long-form input. Preprint, arXiv:2501.03200.

Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12076–12100, Singapore. Association for Computational Linguistics.

Yucheng Jiang, Yijia Shao, Dekun Ma, Sina Semnani, and Monica Lam. 2024. Into the unknown unknowns: Engaged human learning through participation in language model agent conversations. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9917–9955, Miami, Florida, USA. Association for Computational Linguistics.

Mistral AI. 2024. Pixtral large. https://mistral.ai /news/pixtral-large. Accessed: 2025-07-31.

Benjamin F. Jones. 2009. The Burden of Knowledge and the “Death of the Renaissance Man”: Is Innovation Getting Harder? The Review of Economic Studies, 76(1):283–317.

Michael Park, Erin Leahey, and Russell J. Funk. 2023. Papers and patents are becoming less disruptive over time. Nature, 613(7942):138–144.

Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. 2024a. Prometheus: Inducing finegrained evaluation capability in language models. In The Twelfth International Conference on Learning Representations.

Qwen Team. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388. Yijia Shao, Yucheng Jiang, Theodore Kanell, Peter Xu, Omar Khattab, and Monica Lam. 2024. Assisting in writing Wikipedia-like articles from scratch with large language models. In Proceedings of the 2024

10

presence of any positional bias in the participants’ answers, we randomize the order and naming of the reports. In Table 3, the list of questions and the associated answers from the human evaluation study are given. Wyvern is systematically preferred over STORM on all the considered rubrics. WebThinker obtains a higher preference rate only on the structure, i.e. rubric (1), and on the engagement level of the report, namely rubric (15), where Wyvern reaches a 37.50% favor rate. Regarding the structure of the report, the main criticalities that emerged concern the redundancy of the content over the sections and the overall length of the report. Wyvern and WebThinker achieve the same preference rate on rubric (6), which refers to the clarity of the explanations, and on rubric (4), related to the presentation of benefits and limitations about the topic. Wyvern excels on the topic coverage, i.e. rubric (5), where it obtains a 100.00% preference rate; and on the amount of technical details and factuality of the content (rubrics (3) and (15)), where it reaches 75.00% of win percentage. When compared to WikiAutoGen, the only baseline that supports the inclusion of figures within the produced reports, Wyvern achieves preference rates of 85.71%, 87.50% and 87.50% for rubrics (8), (9), and (10), which concern the quality of the positioning, the quality of the captions, and the informativeness degree of the figures, respectively. On all the other rubrics, Wyvern obtains a preference rate spanning from 62.50% to 87.50%.

Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6252–6278, Mexico City, Mexico. Association for Computational Linguistics. Yidong Wang, Qi Guo, Wenjin Yao, Hongbo Zhang, Xin Zhang, Zhen Wu, Meishan Zhang, Xinyu Dai, Min zhang, Qingsong Wen, Wei Ye, Shikun Zhang, and Yue Zhang. 2024. Autosurvey: Large language models can automatically write surveys. In The Thirtyeighth Annual Conference on Neural Information Processing Systems. Ruibin Xiong, Yimeng Chen, Dmitrii Khizbullin, Mingchen Zhuge, and Jürgen Schmidhuber. 2025. Beyond outlining: Heterogeneous recursive planning for adaptive long-form writing with language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 24678–24714, Suzhou, China. Association for Computational Linguistics. Zhaorui Yang, Bo Pan, Han Wang, Yiyao Wang, Xingyu Liu, Luoxuan Weng, Yingchaojie Feng, Haozhe Feng, Minfeng Zhu, Bo Zhang, and Wei Chen. 2026. Multimodal deepresearcher: Generating text-chart interleaved reports from scratch with agentic framework. In Proceedings of the AAAI Conference on Artificial Intelligence. Zhongyu Yang, Jun Chen, Dannong Xu, Junjie Fei, Xiaoqian Shen, Liangbing Zhao, Chun-Mei Feng, and Mohamed Elhoseiny. 2025. Wikiautogen: Towards multi-modal wikipedia-style article generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 15532– 15541.

A

Human evaluation

A.2

We report in this section the complete, unaggregated list of questions and results of the relative assessment and absolute grading of the human evaluation study presented in the main text of the paper. All evaluators consented to the use of aggregated anonymous statistics collected from their responses, and of portions of the generated reports, for publication purposes. A.1

Absolute grading

We report in Table 4 the complete list of questions and results of the absolute grading section of the human evaluation study, before the clustering applied to obtain Figure 6 of the paper. The considered rubrics are the same used for the relative assessment. As discussed in the paper, the absolute grading is used to have a more detailed overview of the quality of Wyvern’s reports over distinct specific rubrics. The methodology of the report for which this evaluation was asked was unknown to the participants: only a generic report index (“Report A" or “Report B") was given as indication of the target, hiding that it was corresponding to the one generated by Wyvern. For a performance comparison between Wyvern and the other baselines, refer to the relative assessment results. From Table 4 it is possible to observe that the highest marks are obtained on rubrics (8) and (11),

Relative assessment

For the relative assessment in the human evaluation study, we provide each participant with two reports, named “Report A" and “Report B", without specifying the generation method. Each person receives one report generated by Wyvern, and one generated by another baseline. We divide the 27 participants into three groups of cardinality equal to 9, to have a uniform amount of comparisons over all the baselines. Furthermore, to avoid the 11

Question 1 2 3 4 5 6 7 8

9

10

11

12

13 14 15 16

Which report has the most logical structure? Which report better provides relevant information with respect to the topic? Which report presents the best amount of technical details about the topic? Which report better presents both benefits/advantages and challenges/limitations related to the topic? Which report has a broader coverage of multiple aspects of the topic? Which report has clearer explanations? Which report has the most neutral tone? Which report has the most logical figures placement (i.e., figures are assigned to a coherent section)? If not applicable (e.g., there are no figures), please select N/A. Which report has the most appropriate figures descriptions and captions? If not applicable (e.g., there are no figures), please select N/A. Which report has the most informative figures? If not applicable (e.g., there are no figures), please select N/A. Which report has the most logical overview tables placement (i.e., the overview table is assigned to a coherent section)? If not applicable (e.g., there is no overview table), please select N/A. Which report has the most appropriate overview tables descriptions and captions? If not applicable (e.g., there are no overview tables), please select N/A. Which report has the most informative overview tables? If not applicable (e.g., there are no overview tables), please select N/A. Which report has a better factuality to the best of your knowledge? Which report is more engaging and thought-provoking? Which report do you find the most useful?

vs STORM 100.00%

Preference rate for Wyvern vs WebThinker vs WikiAutoGen 37.50% 75.00%

100.00%

62.50%

75.00%

100.00%

75.00%

75.00%

100.00%

50.00%

87.50%

100.00%

100.00%

87.50%

100.00% 85.71%

50.00% 75.00%

75.00% 62.50%

-

-

85.71%

-

-

87.50%

-

-

87.50%

100.00%

62.50%

85.71%

100.00%

75.00%

85.71%

100.00%

62.50%

87.50%

100.00%

75.00%

87.50%

100.00%

37.50%

75.00%

100.00%

62.50%

87.50%

Table 3: Results of the relative assessment section of the human evaluation study. Each percentage value represents the preference rate in the pairwise comparison between Wyvern and one of the baselines on a specific rubric. Comparisons cannot be made on criteria concerning figures when considering STORM and WebThinker as the other method, as explained in the paper.

12

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Question The structure of the report is logical. The report provides relevant information with respect to the topic. The report presents the right amount of technical details about the topic. The report presents both benefits/advantages and challenges/limitations related to the topic. The report has a broad coverage of multiple aspects of the topic. The explanations of the report are: The tone of the report is: The position of the figures within the report is logical (i.e., they are assigned to a coherent section). If not applicable (e.g., there are no figures), please select N/A. The caption and the description of the figure’s content in the text are appropriate. If not applicable (e.g., there are no figures), please select N/A. The figures are informative. If not applicable (e.g., there are no figures), please select N/A. The position of the overview table within the report is logical (i.e., it is assigned to a coherent section). If not applicable (e.g., there are no overview tables), please select N/A. The caption and the description of the overview table’s content in the text are appropriate. If not applicable (e.g., there are no overview tables), please select N/A. The overview table is informative. If not applicable (e.g., there are no overview tables), please select N/A. To the best of your knowledge, the factuality of the report is: How engaging and thought-provoking is the report? How useful do you find the generated report?

Score 3.87 ± 0.61 4.26 ± 0.53 3.87 ± 0.85 3.91 ± 0.78 4.35 ± 0.56 3.91 ± 0.88 4.48 ± 0.65 4.52 ± 0.58 3.87 ± 1.23 4.09 ± 0.72 4.50 ± 0.58 4.14 ± 0.69 4.10 ± 0.75 4.22 ± 0.72 3.65 ± 1.09 4.00 ± 0.83

Table 4: Results of the absolute grading of the human evaluation study, reported with the mean and standard deviation of the given marks over the 23 questionnaires that were completed, out of the 27. Participants could assign a score in the [1,5]-range. For rubric (6), the interval extremes were associated with the adjectives “confusing" and “clear"; for rubric (7) with “biased" and “neutral"; and for rubric (14) with “not satisfying" and “very satisfying". Rubrics vs STORM vs WebThinker vs WikiAutoGen Interest level 50.00% 50.00% 50.00% Coherence & 50.00% 50.00% 50.00% Organization Relevance 50.00% 50.00% 50.00% & Focus Broad 50.00% 50.00% 50.00% Coverage

which concern the positioning of the figures and of the overview table in the report, with an average score of 4.52 and 4.50, respectively. The third highest average score, equal to 4.48, is achieved on rubric (7), related to the neutrality of the tone, which is evaluated as satisfactorily neutral. The criterion that receives the lowest score, equal to 3.65, is the engagement and thought-provoking level of the technical report (rubric (15) in Table 4). However, it can be seen that the standard deviation is quite high, which can be explained by the highly subjective nature of the answer. Also the structure of the report, namely rubric (1), and the textual integration of the figures within the text, i.e. rubric (12), exhibit margin of improvement, reaching a score of 3.87. The factuality, the figures’ informativeness and the usefulness of the report (rubrics (14), (10) and (16), respectively) are very positively evaluated, with average scores of 4.22, 4.09, and 4.00, demonstrating how Wyvern’s design successfully accomplishes the objective of producing multimodal and grounded technical reports.

B

Table 5: Results of the automatic relative assessment, performed using Prometheus 2-7B as evaluator model

sidering the same four criteria presented in (Shao et al., 2024), which concern the interest level, the coherence and organization, the relevance and focus of the content, and the broad coverage of the topic. We consider both a relative assessment setup, discussed in the paper, and an absolute grading one. B.1

Relative assessment

In Figure 8 of the paper we presented the results of the pairwise relative evaluation, obtained using Qwen3-32B and DeepSeek-V3 as evaluator models. As discussed in the paper, for each pair of reports (one generated by Wyvern and the other by one of the baselines) we conduct the relative assessment twice, swapping the order of the reports, and assess the impact on the output of evaluators.

Automatic evaluation

We report in this section the complete results of the automatic evaluation of the reports, obtained con13

As shown in Table 2 of the paper, the two models did exhibit ordering bias, i.e. the outcome of the evaluation turned out to depend on the ordering of the reports under assessment. We carried out but omitted from the paper the same experiments using as evaluator Prometheus 2-7B (Kim et al., 2024b), a model specifically trained on a relative ranking task. Table 5 presents the results of this analysis, which shows that Prometheus 2-7B struggles even more to capture the human-assigned scores, giving the same preference rate to all the methods. By a manual inspection of the outcome, it was revealed that the evaluator always identifies as best report the first of the two options. Thus, since we consider both orderings when we perform the evaluation, the preference rate is always equal to 50.00%, regardless of both method and criterion under consideration. The information position hence alters the final output of the model, as already shown in the literature (Liu et al., 2024). We hypothesize that the cause of this behavior could lie in the fact that Prometheus 2-7B was trained with a limited maximum sequence length (4,096 tokens) (Kim et al., 2024b), potentially affecting its capability to handle longer inputs properly. B.2

the choice of the evaluator model can highly affect the results and ranking between various methodologies, making it hard to obtain a fair and robust quantitative interpretation of the evaluations. B.3

Claims evaluation

To verify whether automatic means of evaluation are effective for the grounding assessment, we conducted a manual evaluation comparing LLMs and human judgments on a sample of claims. Specifically, we randomly sampled 60 claims and verified whether the human annotations and the LLM judgments were consistent. The agreement rate was 93.33%, indicating strong alignment between the two in this specific task.

C

Cost analysis

We report the breakdown of the input and completion tokens of each module and agent of Wyvern in Figure 9 and Figure 10, respectively. Statistics are computed on a sample of 6 reports, using DeepSeek-V3 as base model and DeepSeek-R10528 as reasoning model. Agents 2 - 7 in the search module consume approximately the same number of input tokens, as they all analyze the collected documents individually. Within the report generation module, agents 11 and 18 have high input and completion tokens usage as they are responsible for the report’s text generation and refinement, based on the input references. Agent 12 produces a significant amount of completion tokens due to the generation of image descriptions. The grounding module (especially agent 20 ) required lengthy inputs mainly due to the two-stage grounding check, which may lead to considering many references (also in their full-text form); while the long outputs can be attributed to the long reasoning trace needed for the grounding assessment of each claim. Regarding the parsing of the resources, which falls outside of the tokens usage analysis, we note that it can contribute significantly on the end-toend latency of the framework, especially when a high number of long documents with many figures must be processed.

Absolute grading

We conduct an automatic evaluation in an absolute grading schema, similarly to (Shao et al., 2024). In these experiments, we provide only a single report to the evaluator model, which is prompted to assign a score in the [1,5]-range for each of the four considered criteria, i.e. Interest Level, Coherence and Organization, Relevance and Focus, and Broad Coverage. We employed as evaluator models Prometheus-13B (Kim et al., 2024a), Prometheus 2-7B (Kim et al., 2024b), Qwen332B, and DeepSeek-V3. For Prometheus-13B, we employ the same iterative trimming strategy as in (Shao et al., 2024) to reduce the length of text. However, we identify this evaluation setup as suboptimal, as for longer reports the majority of the content gets removed, impeding a fair evaluation. For this reason, we did not employ Prometheus13B for the relative assessment, but only on the absolute grading with the same evaluation setup used in (Shao et al., 2024). Table 6 presents the results of the experiments, that show that there is no uniform agreement between the evaluator models. For each rubric, distinct evaluators identify different optimal methods. This highlights how

D

Failure cases

While the overall evaluation of our framework is positive, as detailed in Section 5.1, the human evaluation study also served to identify the main weaknesses and directions of improvement. 14

Rubrics

Evaluator model

Interest Level Coherence & Organization Relevance & Focus Broad Coverage

Prometheus-13B Prometheus 2-7B DeepSeek-V3 Qwen3-32B Prometheus-13B Prometheus 2-7B DeepSeek-V3 Qwen3-32B Prometheus-13B Prometheus 2-7B DeepSeek-V3 Qwen3-32B Prometheus-13B Prometheus 2-7B DeepSeek-V3 Qwen3-32B

STORM 4.00 ± 0.00 4.22 ± 0.92 3.22 ± 0.63 3.33 ± 0.47 4.78 ± 0.42 4.67 ± 0.47 4.89 ± 0.31 4.56 ± 0.68 4.89 ± 0.31 5.00 ± 0.00 4.78 ± 0.42 4.56 ± 0.50 5.00 ± 0.00 4.44 ± 0.50 4.44 ± 0.50 4.44 ± 0.68

WebThinker 4.56 ± 0.50 4.44 ± 0.83 4.44 ± 0.83 4.33 ± 0.67 4.44 ± 0.50 4.67 ± 0.47 4.67 ± 0.47 5.00 ± 0.00 4.89 ± 0.31 5.00 ± 0.00 4.56 ± 0.50 5.00 ± 0.00 4.89 ± 0.31 4.22 ± 0.42 4.33 ± 0.47 4.89 ± 0.31

Score WikiAutoGen 4.00 ± 0.00 4.67 ± 0.67 3.44 ± 0.68 3.67 ± 0.67 4.33 ± 0.47 4.22 ± 0.42 4.22 ± 0.92 4.22 ± 0.63 4.78 ± 0.63 5.00 ± 0.00 4.00 ± 0.94 4.11 ± 0.31 5.00 ± 0.00 4.00 ± 0.00 4.00 ± 0.00 4.67 ± 0.47

Wyvern 4.07 ± 0.47 4.85 ± 0.45 4.85 ± 0.52 3.78 ± 0.50 3.96 ± 0.58 4.70 ± 0.46 4.85 ± 0.36 4.52 ± 0.63 4.11 ± 0.92 5.00 ± 0.00 4.81 ± 0.47 4.59 ± 0.56 4.19 ± 1.54 4.41 ± 0.49 4.37 ± 0.48 4.93 ± 0.26

Table 6: Results of the automatic absolute grading, performed with different evaluator models, namely Prometheus13B, Prometheus 2-7B, DeepSeek-V3, and Qwen3-32B. The results are reported in terms of the mean and standard deviation of the scores over all the reports generated by a given method. Underlined, the method achieving the highest average score with a specific evaluator on a given criterion.

Number of tokens

Input tokens

partially correct explanations and confusing terminology were also reported. Interestingly, we also came across an instance of a clearly false statement. After manually inspecting the cited reference, we found that the claim was indeed faithful to the source, and it was the source itself that contained the factual inaccuracy. As discussed in the Ethical considerations section, we emphasize on this matter that while Wyvern targets the groundedness of the claims, it does not incorporate a mechanism to filter out erroneous information retrieved from the web.

Completion Tokens

107 3.15e6

2.58e6

106 105

7.54e5 3.50e5

4.20e4

Search

7.22e4

Report gen.

Grounding

Figure 9: Input and completion tokens used by each module of Wyvern.

The main challenge noted by a small number of evaluators was the reports’ excessive length and redundancy, sometimes resulting in a fragmented structure. A few evaluators also identified an uneven balance between breadth and depth, with too much focus given to peripheral topics. Issues involving images, e.g. imprecise placement or explanation, instead stem mainly from insufficient textual descriptions in the input sources, which may lead to overly general figure descriptions by agent 12 . This in turn may lead to the placement of figures in sections not fully relevant to the figure’s topic, or to overly vague in-text explanations of their content. Additionally, the parsing tool occasionally crops images imprecisely, resulting in minor inconsistencies between the images’ content and their captions. Regarding claims grounding, instances of only 15

Input tokens

Completion tokens

1 2 3

Search

4 5 6 7 8 9

11 12

Report gen.

Agent

10

13 14 15 16 17 18

Grounding

19 20 21 22 101

102

103

104

105

106

107

108

Number of tokens

Figure 10: Input and completion tokens used by each agent of Wyvern.

16

Related documents

• Wyvern: An Agentic Framework for Generating Grounded Multimodal Reports#609613
• Cognitive-structured Multimodal Agent for Multimodal Understanding, Generation, and Editing#353065
• The Plot Twist: Jailbreaking Unified Multimodal Models with a Three-Act NarrativeAttack#1104892
• DAREBench: Deployment-Aware and Reliable Evaluation of Models as Agents#973669
• Same Semantics, Different Outcome: On the Modality Robustness of Multimodal LLMs under Knowledge Conflict#819989