GRADRAG: Cross-Component Prompt Adaptation for Coordinated Multi-Agent RAG

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

GRADRAG: Cross-Component Prompt Adaptation for Coordinated Multi-Agent RAG

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

artificialintelligence knowledgerepresentation reasoning

artificial intelligence, reasoning, knowledge representation

G RAD RAG: Cross-Component Prompt Adaptation for Coordinated Multi-Agent RAG Paolo Pedinotti* Bloomberg [email protected]

arXiv:2607.21324v1 [cs.CL] 23 Jul 2026

Abstract Retrieval-Augmented Generation (RAG) systems increasingly employ multiple LLM agents. Yet, most prior work optimizes components in isolation rather than coordinating improvements across the pipeline. We introduce G RAD RAG, a framework for crosscomponent prompt adaptation that models the RAG pipeline as a computational graph and propagates structured evaluation feedback to update upstream agents. An Evaluator critiques downstream answers and supporting evidence, producing actionable feedback that a Prompt Optimizer uses to iteratively update adaptive agents (e.g., retrievers, graph constructors, answerers). The Evaluator also triggers early stopping when the output is deemed satisfactory. We evaluate G RAD RAG on the SQ UALITY and QMS UM benchmarks under two retrieval paradigms: (i) flat (chunk-based) retrieval using IRCoT-style query refinement (Trivedi et al., 2023), and (ii) graph-based retrieval that constructs and iteratively enriches an entity–relation graph from the document. Across both settings, G RAD RAG consistently outperforms one-step refinement baselines that update only the final generator, achieving a 12– 15 percentage point net preference margin in LLM-judged pairwise comparisons, with most gains realized within two refinement iterations.

Introduction

Retrieval-Augmented Generation (RAG) has become a central paradigm for enabling large language models (LLMs) to reason over information beyond their parametric knowledge. Early RAG systems followed a simple retrieve-then-generate workflow in which a retriever selects relevant text and a generator produces an answer conditioned on it. More recently, RAG architectures have evolved * Paolo Pedinotti contributed to this work during his internship at Bloomberg. † Equal contribution.

Enrico Santus† Bloomberg [email protected]

into increasingly agentic systems, where multiple LLM-based components cooperate to formulate retrieval queries, organize evidence, perform multistep reasoning, and evaluate or revise intermediate outputs. Despite these advances, most RAG pipelines still optimize components in isolation. Refinements are typically applied locally—for example, through query rewriting, evidence filtering, or post-hoc answer editing via self-reflection or self-correction. Consequently, errors introduced early in the pipeline, such as incomplete retrieval or poorly structured evidence, often propagate downstream and limit overall performance. This reveals a coordination problem in multi-agent RAG pipelines: evaluation signals produced at the final generation stage are rarely used to improve upstream components such as retrievers or graph construction modules. In this work, we study document-grounded query answering and query-focused summarization, where retrieval operates within a given document (or document set) by selecting relevant spans for each query. This setting corresponds to benchmarks such as SQ UALITY and QMS UM, where each query is paired with long documents. Although these documents fit within modern LLM context windows, our goal is not to compare RAG with full-context inference but to use these benchmarks as a controlled testbed for analyzing how different RAG pipelines behave and how coordination between their components can be improved. We introduce G RAD RAG, a framework for cross-component prompt adaptation in agentic RAG systems (Figure 1). G RAD RAG models the RAG pipeline as a computational graph whose nodes correspond to agents such as the retriever, graph constructor, and answer generator. Instead of refining only the final answer, the framework propagates evaluation feedback to multiple upstream agents responsible for retrieval and evidence con-

Figure 1: G RAD RAG: cross-component prompt adaptation in an agentic RAG pipeline. Grey (forward pass): A Retrieval Agent gathers evidence from either document chunks (Vector RAG) or a structured entity–relation graph (GraphRAG, constructed by a separate agent that is not shown in the mock up), which is assembled into a context and passed to an Answer Generation Agent to produce a candidate answer. Orange (evaluation and control): An Evaluator reviews the answer and its supporting evidence and returns feedback with a binary Stop? decision. If Yes, the answer is accepted. If No, a Prompt Optimizer updates the prompts of one or more agents (as indicated by the evaluation), and the next forward pass is executed.

struction. At test time, an Evaluator agent analyzes the generated answer together with its supporting evidence and produces structured feedback identifying missing information, weak reasoning links, or irrelevant context. A Prompt Optimizer converts this feedback into prompt updates for adaptive agents across refinement iterations. The process operates entirely through prompt updates and does not modify model parameters. By allowing downstream critiques to influence upstream decisions, G RAD RAG enables coordinated improvements beyond one-step answer rewriting. Rather than optimizing retrieval, structuring, and generation independently, the framework aligns their behavior through feedback-driven prompt refinement within a bounded test-time budget. We evaluate G RAD RAG on SQ UALITY and QMS UM using two representative retrieval paradigms. The first is a flat (chunk-based) retrieval pipeline, where the system iteratively rewrites search queries and retrieves document chunks using a hybrid dense–lexical retriever following an IRCoT-style strategy (Trivedi et al., 2023). The second is a graph-based retrieval pipeline that extracts and iteratively enriches an entity–relation graph to assemble evidence for answering the query. Within each paradigm we com-

pare GradRAG against controlled one-step refinement baselines in which only the final answer generator is updated, isolating the effect of crosscomponent prompt adaptation. Across both settings, G RAD RAG consistently outperforms these baselines, with most gains realized within the first two refinement iterations. Section 5 shows that cross-component prompt adaptation yields a 12–15 percentage point net preference margin in LLMguided pairwise evaluations. G RAD RAG therefore provides a general framework for coordinating improvements across heterogeneous components in modular RAG pipelines. This work makes the following contributions: • We introduce a framework for crosscomponent prompt adaptation in agentic RAG, enabling evaluator feedback to update the prompts of multiple upstream agents at test time. • We show that coordinated agent refinement via shared evaluative feedback improves alignment between retrieval, structuring, and generation. • We propose a modular graph-based abstraction that applies the same feedback-driven optimization across both flat (chunk-based)

and graph-based RAG pipelines.

Related Work

2.1

Agentic Approaches and RAG

Recent work in Retrieval-Augmented Generation (RAG) increasingly incorporates agentic components. Asai et al. (2023) introduce a reflectionbased mechanism in which the model produces intermediate tokens representing retrieval decisions and evidence assessments, enabling iterative retrieval and answer refinement guided by a learned critic. This work established that RAG performance can benefit from multi-step, agentic decision-making. Subsequent approaches distribute retrieval and reasoning across specialized components. For example, Jeong et al. (2024) and Lee et al. (2025) propose routing agents that decide whether and how extensively to retrieve and which backend to query. Trivedi et al. (2023) frame retrieval as a sequential decision process that iteratively refines search queries, while Jiang et al. (2025) use Monte Carlo Tree Search guided by learned reward models. In this literature, multi-agent typically refers to pipelines composed of multiple LLM calls with distinct prompts, each specializing in a subtask such as retrieval routing, evidence assessment, or answer synthesis. Multi-agent retrieval optimization has become a common strategy. Yan et al. (2024) train a relevance judge that decides whether to continue querying or switch to generation, while Chang et al. (2025) generate partial answers from retrieved documents and use a judge to select the most informative ones. Despite their success, these approaches apply feedback locally, focusing on individual stages such as retrieval control or answer selection. Consequently, evaluation signals generated at the final answer stage rarely influence upstream components such as retrieval or evidence construction. In contrast, our work addresses this coordination problem directly: how feedback on downstream answer quality can be used to adapt multiple upstream components of a RAG pipeline in a unified manner. 2.2

LLMs as Critic Agents and Prompt Optimizers

The reflection pattern can be viewed as a threestage loop of generation, evaluation, and adaptation. Early systems (Madaan et al., 2023; Shinn et al.,

2023) demonstrated inference-time refinement in which models iteratively revise outputs using selfor evaluator-generated critiques (later described as textual gradients by Yuksekgonul et al. (2024), or verbal reinforcement). These approaches treat natural-language critiques as optimization signals guiding subsequent reasoning steps. This idea was later generalized into automatic prompt optimization, where an LLM searches for prompts that improve downstream performance (Zhou et al., 2023; Opsahl-Ong et al., 2024). Frameworks such as DSPy (Khattab et al., 2023) and TextGrad (Yuksekgonul et al., 2024) provide programming abstractions for this process: DSPy focuses on optimizing prompting structure and demonstrations, while TextGrad formalizes critique-driven prompt updates over computational graphs. Our work builds on these foundations but focuses specifically on coordinating prompt adaptation across multiple components of a RAG pipeline. G RAD RAG instantiates critique-driven prompt optimization within agentic RAG systems and empirically evaluates how coordinating updates across retrieval, structuring, and generation components improves retrieval-augmented reasoning over long documents.

Method: G RAD RAG

G RAD RAG is a test-time framework for coordinating multiple agents in Retrieval-Augmented Generation (RAG) pipelines through cross-component prompt adaptation. G RAD RAG builds on critique-driven prompt optimization and reflectionbased approaches, instantiating them in agentic RAG systems where feedback on downstream answer quality is used to adapt multiple upstream components in a unified manner. Overview. As illustrated in Figure 1, G RAD RAG executes a RAG pipeline in iterative refinement cycles. Each cycle consists of a forward pass that produces intermediate artifacts (e.g., retrieved text chunks or extracted graph structures) and a final answer, followed by an evaluation step. An Evaluator agent reviews the answer together with its supporting evidence and produces structured naturallanguage feedback. If refinement is required, a Prompt Optimizer converts this feedback into updated prompts for one or more adaptive agents. The next cycle then runs with the updated prompts. This process operates entirely through prompt updates

and does not modify model parameters. The full refinement loop is summarized in Algorithm 1. Computational graph abstraction. G RAD RAG models the RAG workflow as a computational graph whose nodes represent agents, programs, or data artifacts, and whose edges represent computational transformations. Nodes corresponding to LLM-based agents are associated with prompts, which may be either fixed or adaptive. In our framework, adaptive agents are those whose prompts can be updated across refinement cycles based on evaluator feedback. This abstraction allows heterogeneous components — such as classical retrievers and LLM-based agents — to coexist within the same pipeline. Cross-component prompt adaptation. Unlike one-step reflection approaches that refine only the final answer generator, G RAD RAG allows evaluator feedback to update the prompts of multiple upstream agents. For example, feedback indicating missing causal or explanatory information may trigger changes to both the retrieval strategy (to target more relevant evidence) and the answer generation style (to emphasize causal structure). Prompt updates are applied iteratively within a bounded testtime budget, enabling coordinated improvements across components. Early stopping. Following prior reflection-based architectures (Madaan et al., 2023; Shinn et al., 2023), G RAD RAG employs an early-stopping mechanism. After each evaluation step, the Evaluator returns a discrete decision indicating whether the current output is satisfactory. Refinement cycles stop when the Evaluator signals satisfaction or when a maximum number of iterations is reached (Algorithm 1, lines 6–8). 3.1

Vector RAG Technical Details

For flat retrieval, we implement a documentgrounded Vector RAG pipeline in which the input document is segmented into overlapping text chunks. A Retrieval Agent follows an IRCoT-style strategy (Trivedi et al., 2023), iteratively generating sub-queries based on the context retrieved so far. Each sub-query triggers a hybrid retrieval step that combines dense similarity search and lexical matching to select relevant chunks. Within G RAD RAG, the Retrieval Agent is adaptive: its prompt may be updated across refinement cycles based on evaluator feedback. For instance,

Algorithm 1 G RAD RAG: Cross-Component Prompt Adaptation Require: Query q, document(s) D, initial prompts {pa } for agents a ∈ A, max iterations T 1: for t = 1 to T do 2: Forward pass: 3: Execute RAG pipeline with current prompts {pa }: 4: – generate intermediate artifacts (e.g., retrieved chunks or extracted entity–relation graphs) 5: – assemble supporting evidence (e.g., via chunk aggregation or graph community summaries) 6: – produce answer yt 7: Evaluation: 8: Evaluator reviews (yt , supporting evidence) and returns feedback ft and decision dt 9: if dt = SATISFACTORY then 10: return yt 11: end if 12: Prompt adaptation: 13: for each adaptive agent a ∈ Aadapt do 14: Update prompt pa ← Optimize(pa , ft ) 15: end for 16: end for 17: return final answer yT

critiques indicating missing explanations or temporal relationships can lead the agent to generate more targeted sub-queries in subsequent iterations. Retrieved chunks are aggregated to form the context passed to the Answer Generation Agent, as specified in the forward pass of Algorithm 1. 3.2

GraphRAG Technical Details

Our graph-based RAG pipeline builds on prior graph-oriented approaches (Edge et al., 2025; Pedinotti and Santus, 2026). The document is segmented into larger text spans, from which a Graph Extraction Agent identifies entities and relations. Extracted entities are merged into a consolidated graph, and community detection is performed using the Leiden algorithm to identify groups of related entities. For each community, the system generates a natural-language summary and a set of candidate answers, each annotated with a helpfulness score indicating relevance to the query. Only answers with positive scores are retained and concatenated to form the evidence context supplied to the Answer Generation Agent. As in the vector-based setting, both the Graph Extraction Agent and the Answer Generation Agent are adaptive. Evaluator feedback highlighting missing entities, relations, or explanatory structure is used to refine the extraction prompt, guiding subsequent graph enrichment toward information that is most useful for answering the query. Over successive refinement cycles, this feedback-driven

adaptation effectively induces a task-oriented ontology, shaping which types of entities and relations are prioritized during graph construction (Pedinotti and Santus, 2026; Pedinotti et al., 2026). The resulting enriched graph is then used to assemble evidence for the final answer in the next forward pass (Algorithm 1, lines 2–5). Implementation. All pipelines are implemented using the Microsoft AutoGen framework (Wu et al., 2023). The prompts are provided in Appendices A and B to support reproducibility.

Experimental Setup

Research Questions. We study whether coordinating prompt adaptation across multiple agents improves RAG pipelines. Specifically, we ask: (1) Does cross-component prompt adaptation — where evaluator feedback updates upstream agents such as retrievers or graph extractors — improve performance compared to one-step refinement that updates only the final answer generator? (2) Are these improvements consistent across different RAG architectures, including flat (chunk-based) and graph-based retrieval pipelines? G RAD RAG Systems and Baselines. To answer these questions, we evaluate G RAD RAG under two retrieval paradigms: flat (chunk-based) retrieval and graph-based retrieval. For each paradigm, we compare a one-step refinement baseline, in which feedback updates only the answer generator, against a full G RAD RAG variant that propagates feedback to multiple agents. The systems are: • IRCoT (Trivedi et al., 2023) + One-Step Refinement: A vector-based RAG pipeline in which only the Answer Generation Agent is adaptive. The Retrieval Agent uses a fixed prompt, so improvements arise solely from iterative answer rewriting. • Full G RAD RAG Vector: The same pipeline, but evaluator feedback updates both the Answer Generation Agent and the Retrieval Agent, enabling refinement of sub-query generation and evidence selection. • GraphRAG (Edge et al., 2025) + One-Step Refinement: A graph-based RAG pipeline where only the Answer Generation Agent is adaptive. The Graph Extraction Agent uses a fixed prompt, so improvements arise only from refining the final answer.

Dataset

Docs

Questions

Avg Doc Length

Avg Resp Length

SQ UALITY

260

4,995 words

241 words

QMS UM

281

10,668 words

65 words

Table 1: Statistics about the test sets.

• Full G RAD RAG Graph: The same pipeline, but evaluator feedback updates both the Answer Generation Agent and the Graph Extraction Agent, enabling adaptive graph enrichment. This setup forms a controlled ablation: within each retrieval paradigm, the baseline and GradRAG variants share the same architecture, model, and retrieval process, differing only in whether evaluator feedback updates upstream agents. This isolates the effect of cross-component prompt adaptation. Task. We evaluate all systems on the QueryFocused Summarization (QFS) task. In QFS, the model receives a document (or document set) d and a query q, and must produce a summary that answers the query using information drawn from the document. The task lies at the intersection of abstractive summarization and question answering, requiring both global document understanding and targeted evidence selection. Datasets. We conduct experiments on two standard QFS datasets: SQ UALITY (Wang et al., 2022) and QMS UM (Zhong et al., 2021). These datasets represent complementary domains: narrative texts (SQ UALITY) and long meeting transcripts (QMS UM). Both datasets are also included in the ZeroSCROLLS (Shaham et al., 2023) and LongBench (Bai et al., 2024) benchmarks. Although these documents fit within the context windows of modern LLMs, we use these datasets as controlled benchmarks for studying the behavior of RAG pipelines and evaluating coordination between their components. Table 1 summarizes their main characteristics. Experimental Protocol. G RAD RAG operates in a test-time refinement setting. For each test instance (query–document pair), the system executes an iterative loop consisting of a forward pass followed by evaluation and, if necessary, prompt adaptation. During refinement, an Evaluator Agent analyzes the generated answer together with its supporting evidence and produces structured feedback identifying missing information or reasoning gaps. Importantly, the Evaluator does not have access

to the reference answer, preventing data leakage during optimization. If the Evaluator determines that the answer is unsatisfactory, its critique is used to update the prompts of the adaptive agents, and another refinement cycle is executed. Otherwise, the process terminates through an early-stopping decision. Each test instance is processed independently and optimized prompts are reset after each example, ensuring that no information is carried across test instances. This protocol enables a controlled comparison between one-step refinement and cross-component adaptation by isolating the effect of prompt updates within a single query. While this per-instance refinement increases computational cost, it avoids cross-example contamination. In deployment scenarios, efficiency could be improved by batching multiple instances before updating prompts, which we leave for future work. Evaluation Metrics. Traditional evaluation metrics for query-focused summarization, such as ROUGE and BERTScore, correlate poorly with human judgments because they primarily measure lexical or embedding overlap rather than higherlevel properties such as coherence, factuality, and reasoning (Kryscinski et al., 2019; Liu et al., 2023; Nguyen et al., 2024). During refinement, G RAD RAG uses an internal Evaluator Agent that analyzes generated answers together with their supporting evidence and produces critiques guiding prompt updates. This evaluator operates without access to gold reference answers to prevent data leakage. For final system comparison, we instead adopt an LLM-as-a-Judge evaluation paradigm in which an external LLM compares system outputs using a structured rubric. Given two candidate answers for the same input, the judge selects the better response based on overall answer quality and use of supporting evidence. We report win rates, defined as the proportion of pairwise comparisons in which one system is preferred over the other (Liu et al., 2025). To improve evaluation robustness, we follow recommendations from recent work on LLM-based evaluation (Baumann et al., 2025). Specifically, we report position consistency (Shi et al., 2025), which measures how often the judge’s preference remains unchanged when the order of candidate answers is reversed. We also apply a two-sided binomial sign test to assess statistical significance.

We also conduct human validation on a subset of examples. Annotators are shown the same pairwise outputs and asked to select the better response using the same rubric. Agreement between LLM and human preferences is measured using accuracy and Cohen’s κ computed on majority-vote labels. Finally, to analyze refinement dynamics (Section 5), we track several per-iteration diagnostics: average response length, lexical density (contentword ratio), ROUGE-1 between the system response and the gold reference answer, and topic entropy, defined as the Shannon entropy of topic distributions obtained by fitting LDA to the set of system responses at each iteration. Implementation Details and Hyperparameters. For flat (chunk-based) retrieval, documents are segmented into overlapping text chunks using fixed character lengths (default 400 characters with a 40-character overlap). Retrieval follows an IRCoTstyle strategy (Trivedi et al., 2023), in which the Retrieval Agent iteratively generates sub-queries based on the context retrieved so far. Each subquery triggers a retrieval step, where the Retrieval Agent selects the retrieval strategy to apply: dense similarity search via FAISS (Douze et al., 2025) or lexical matching via BM25 (Robertson et al., 1995). We perform three retrieval iterations and select the top retrieved chunks at each step, aggregating them to form the final context passed to the Answer Generation Agent. The choice of retrieval strategy is controlled by the agent’s prompt and may change across refinement iterations as a result of evaluator feedback. For graph-based retrieval, documents are segmented into larger overlapping spans (default 1000 characters with a 200-character overlap) to support more reliable entity and relation extraction. A Graph Extraction Agent identifies entities and relations from these spans, which are aggregated into a graph structure that is incrementally enriched across refinement iterations. The resulting graph is then used to assemble evidence for answer generation. Across all systems, the maximum number of refinement iterations is capped at three. Models. All generation agents (Retriever, Graph Extractor, and Answer Generator) use Gemini-2.5Flash (Comanici et al., 2025) (June 2025 version) via the Google Generative Language API.1 Repetitive subtasks—such as sub-query generation in 1

https://ai.google.dev/gemini-api/docs/models

flat retrieval, entity–relation extraction in graph retrieval, and community-level summarization and candidate-answer generation—use the more costefficient Gemini-2.5-Flash-Lite (July 2025 version) through the same API. For vector indexing, we use the text-embedding-3-small embedding model.2 For evaluation, we use the open DeepSeekV3.1 model (DeepSeek-AI, 2025) as an external LLM judge. The prompts used by the system and the evaluator are provided in Appendices A and B to support reproducibility. Computational Cost. We analyze the additional overhead introduced by cross-component prompt adaptation using the GraphRAG pipeline on a sample of 13 query–document pairs. A full refinement cycle requires on average 35s per query compared to 30.5s for the one-step baseline, corresponding to roughly a 10% increase. Most runtime (about 70%) is spent in the forward pipeline—graph extraction and answer generation—which is identical across systems. Token usage increases from 6,040 tokens per query in the baseline to 8,140 tokens in G RAD RAG, primarily due to critique processing during prompt updates. Retrieval Paradigm

SQ UALITY

QMS UM

Flat (Vector RAG) Full G RAD RAG win rate (%)

56.5

56.0

Position consistency (%) p-value

79.3 0.028

55.1 0.090

Full G RAD RAG win rate (%) Position consistency (%)

57.5 82.5

55.8 87.9

p-value

0.005

0.020

Graph-based RAG

Table 2: Pairwise evaluation results. Values report the percentage win rate of G RAD RAG against the corresponding one-step baseline.

2 https://platform.openai.com/docs/guides/ embeddings

(a) Length

(b) Lexical density

Metric Full G RAD RAG win rate (%) Position consistency (%) p-value

SQ UALITY

QMS UM

58.5 79.3 0.0038

61.5 59.7 0.0007

Table 3: Effect of an additional refinement iteration for flat retrieval. Method

SQ UALITY

QMS UM

Flat Retrieval IRCoT + One-step refinement 1.82 Full G RAD RAG Vector 1.84 Graph Retrieval GraphRAG + One-step refine1.83 ment Full G RAD RAG Graph 1.72

1.60 1.63

1.63 1.68

Table 4: Average number of refinement iterations before early stopping.

Results

Human Validation. Human evaluation results show substantial agreement between LLM-based judgments and human preferences. Across the annotated subset, the LLM achieves a 75.86% agreement rate with human evaluators, a Cohen’s κ of 0.59, and an F1 score of 0.8. Human preferences exhibit moderate variability, with an average annotation entropy of 0.642, indicating partial disagreement on more challenging examples, particularly those involving longer responses. Main Results. Table 2 reports pairwise comparison results between one-step baselines and full G RAD RAG variants. Comparisons are performed within each retrieval paradigm (flat vs. graphbased), isolating the effect of cross-component prompt adaptation from differences in retrieval architecture. Across both datasets and retrieval paradigms, full G RAD RAG variants consistently outperform their one-step counterparts. Win rates range be-

Figure 2: Evolution of system responses across iterations.

(d) Topic entropy

Retrieval Paradigm

SQ UALITY

Iteration 2 Flat (Vector) GradRAG win 54.0 rate (%) Graph-based GradRAG win 53.6 rate (%) Iteration 3 Flat (Vector) GradRAG win 56.5 rate (%) Graph-based GradRAG win 57.5 rate (%)

QMS UM

54.6 55.6

56.0 55.8

Table 5: Pairwise preference rates for G RAD RAG over one-step baselines when the number of refinement iterations is fixed to a given value (without early stopping).

tween 55% and 57%, corresponding to a net preference margin of approximately 12–15 percentage points when compared with the corresponding baseline preferences (e.g., 56% vs. 44%). These results indicate that allowing evaluator feedback to update multiple upstream agents yields consistent improvements over refining only the final answer generator. Evaluation robustness diagnostics are reported alongside preferences (Section 4). Three of the four comparisons yield statistically significant differences at the p < 0.05 level. The remaining case—flat retrieval on QMS UM—shows marginal significance, reflecting greater variability in evaluator preferences for this setting. To further investigate this case, we run an additional refinement iteration for flat retrieval on QMS UM. As shown in Table 3, the additional iteration increases the performance gap and yields statistically significant results, suggesting that the earlier marginal outcome was likely due to insufficient refinement rather than instability in the evaluation procedure. Effect of Refinement Budget. Table 4 reports the average number of refinement iterations before early stopping. Across systems, refinement typically terminates after fewer than two iterations, indicating that satisfactory outputs are often reached well before the maximum iteration budget. Table 5 reports pairwise preference rates as a function of a fixed number of refinement iterations, evaluated without early stopping. For flat (chunk-based) retrieval, performance generally improves as additional refinement iterations are permitted, with win rates increasing from it-

eration 2 to iteration 3 on both datasets. Graphbased G RAD RAG shows a similar pattern, achieving preference rates above 50% at both iteration limits across datasets. While gains are not strictly monotonic in all cases—most notably on QMS UM, where the win rate slightly decreases from iteration 2 to iteration 3—the graph-based variant remains consistently preferred over the one-step baseline. Overall, these results suggest that both flat and graph-based pipelines benefit from additional refinement under fixed budgets, while early stopping helps balance refinement gains against potential over-specialization in structured retrieval settings. Response Evolution Across Iterations. Figure 2 illustrates how system outputs evolve across refinement iterations. Response length increases sharply after the first refinement and then stabilizes, while lexical density increases steadily across iterations. ROUGE-1 scores improve monotonically, and topic entropy decreases over time, indicating progressively more focused responses. Taken together, these trends suggest that refinement primarily improves responses by increasing informational density and topical focus rather than by simply adding more content. This pattern is consistent across retrieval paradigms and supports the interpretation that cross-component prompt adaptation produces more targeted and coherent outputs.

Conclusion

We introduced G RAD RAG, a framework for coordinating multiple agents in Retrieval-Augmented Generation (RAG) pipelines through crosscomponent prompt adaptation. By modeling RAG systems as computational graphs and propagating evaluation feedback across agents, G RAD RAG enables coordinated refinement of retrieval, evidence construction, and answer generation at test time. Across both vector- and graph-based retrieval settings, G RAD RAG achieves a 12–15 percentage point net preference margin in pairwise comparisons. These results indicate that allowing evaluation feedback to influence upstream components—rather than refining only the final answer—consistently improves system outputs. More broadly, our findings highlight the importance of coordinating adaptation across heterogeneous components in modular RAG pipelines.

Limitations

While G RAD RAG introduces a general framework for agentic optimization in RAG systems, several aspects offer opportunities for further refinement: • Computational cost. The approach is computationally intensive, as both the forward execution and refinement phases rely on LLM generation. However, this cost scales predictably with the number of adaptive agents and remains manageable for moderate configurations. Moreover, efficiency gains can be achieved through parallelization. • Evaluation vs. deployment gap. Our evaluation protocol — where each refinement cycle is triggered after a single forward execution — was designed to ensure data isolation and interpretability. While this setup is less reflective of deployment conditions, it provides a clean benchmark for studying agentic feedback dynamics. In contrast, recent approaches to automatic prompt optimization, such as DSPy (Khattab et al., 2023), derive optimized prompts from multiple examples. Future research may extend G RAD RAG to batch- or continual-feedback regimes to better approximate real-world usage. • Evaluator configuration. The Evaluator Agent operates using general summarization principles (e.g., coherence, fluency) without external reward signals. This design isolates the intrinsic capability of LLMs to selfevaluate while avoiding data leakage. Future implementations could integrate additional supervision signals, such as reward models or heuristic metrics, to strengthen evaluation fidelity.

References Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-rag: Learning to retrieve, generate, and critique through self-reflection. Preprint, arXiv:2310.11511. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. LongBench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3119–3137, Bangkok, Thailand. Association for Computational Linguistics.

Joachim Baumann, Paul Röttger, Aleksandra Urman, Albert Wendsjö, Flor Miriam Plaza del Arco, Johannes B. Gruber, and Dirk Hovy. 2025. Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotation. Preprint, arXiv:2509.08825. Chia-Yuan Chang, Zhimeng Jiang, Vineeth Rakesh, Menghai Pan, Chin-Chia Michael Yeh, Guanchu Wang, Mingzhi Hu, Zhichao Xu, Yan Zheng, Mahashweta Das, and Na Zou. 2025. MAIN-RAG: Multi-agent filtering retrieval-augmented generation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2607–2622, Vienna, Austria. Association for Computational Linguistics. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, and 1 others. 2025. Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities. Preprint, arXiv:2507.06261. DeepSeek-AI. 2025. DeepSeek-V3 Technical Report. Preprint, arXiv:2412.19437. Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. 2025. The faiss library. Preprint, arXiv:2401.08281. Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. 2025. From local to global: A graph rag approach to query-focused summarization. Preprint, arXiv:2404.16130. Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong Park. 2024. Adaptive-RAG: Learning to adapt retrieval-augmented large language models through question complexity. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7036–7050, Mexico City, Mexico. Association for Computational Linguistics. Jinhao Jiang, Jiayi Chen, Junyi Li, Ruiyang Ren, Shijie Wang, Xin Zhao, Yang Song, and Tao Zhang. 2025. RAG-star: Enhancing deliberative reasoning with retrieval augmented verification and refinement. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7064–7074, Albuquerque, New Mexico. Association for Computational Linguistics. Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. 2023. Dspy: Compiling declarative language model calls into self-improving pipelines. Preprint, arXiv:2310.03714.

Wojciech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Neural text summarization: A critical evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 540–551, Hong Kong, China. Association for Computational Linguistics. Meng-Chieh Lee, Qi Zhu, Costas Mavromatis, Zhen Han, Soji Adeshina, Vassilis N. Ioannidis, Huzefa Rangwala, and Christos Faloutsos. 2025. HybGRAG: Hybrid retrieval-augmented generation on textual and relational knowledge bases. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 879–893, Vienna, Austria. Association for Computational Linguistics. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore. Association for Computational Linguistics. Yinhong Liu, Han Zhou, Zhijiang Guo, Ehsan Shareghi, Ivan Vulić, Anna Korhonen, and Nigel Collier. 2025. Aligning with human judgement: The role of pairwise preference in large language model evaluators. Preprint, arXiv:2403.16950. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: iterative refinement with self-feedback. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Curran Associates Inc. Huyen Nguyen, Haihua Chen, Lavanya Pobbathi, and Junhua Ding. 2024. A comparative study of quality evaluation methods for text summarization. Preprint, arXiv:2407.00747. Krista Opsahl-Ong, Michael J Ryan, Josh Purtell, David Broman, Christopher Potts, Matei Zaharia, and Omar Khattab. 2024. Optimizing instructions and demonstrations for multi-stage language model programs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9340–9366, Miami, Florida, USA. Association for Computational Linguistics. Paolo Pedinotti, Peter Baumann, Nathan Jessurun, Leslie Barrett, and Enrico Santus. 2026. MetaGraph: A large-scale meta-analysis of GenAI in financial NLP (2022–2025). In Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM), pages 848–861, San Diego, California, USA. Association for Computational Linguistics.

Paolo Pedinotti and Enrico Santus. 2026. StructSurvey: Structured agentic retrieval for automated survey paper generation. In Proceedings of the First Workshop on Structured Understanding, Retrieval, and Generation in the LLM Era (SURGeLLM 2026), pages 162–181, San Diego, California, United States. Association for Computational Linguistics. Stephen Robertson, S. Walker, S. Jones, M. M. HancockBeaulieu, and M. Gatford. 1995. Okapi at trec-3. In Overview of the Third Text REtrieval Conference (TREC-3), pages 109–126. Gaithersburg, MD: NIST. Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. 2023. ZeroSCROLLS: A zero-shot benchmark for long text understanding. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7977–7989, Singapore. Association for Computational Linguistics. Lin Shi, Chiyu Ma, Wenhua Liang, Xingjian Diao, Weicheng Ma, and Soroush Vosoughi. 2025. Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge. Preprint, arXiv:2406.07791. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: language agents with verbal reinforcement learning. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Curran Associates Inc. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2023. Interleaving retrieval with chain-of-thought reasoning for knowledgeintensive multi-step questions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10014–10037, Toronto, Canada. Association for Computational Linguistics. Alex Wang, Richard Yuanzhe Pang, Angelica Chen, Jason Phang, and Samuel R. Bowman. 2022. SQuALITY: Building a long-document summarization dataset the hard way. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1139–1156, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. 2023. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation. Preprint, arXiv:2308.08155. Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, and Zhen-Hua Ling. 2024. Corrective retrieval augmented generation. Preprint, arXiv:2401.15884. Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. 2024. Textgrad: Automatic "differentiation" via text. Preprint, arXiv:2406.07496.

Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. 2021. QMSum: A new benchmark for querybased multi-domain meeting summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5905–5921, Online. Association for Computational Linguistics. Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023. Large language models are human-level prompt engineers. Preprint, arXiv:2211.01910.

Prompts for Evaluator Agents

A.1

Flat Retrieval

3. How the answer could improve in factual precision, logical or causal connections, completeness, and stylistic quality. Output format: DECISION: [SATISFACTORY or NEEDS_REFINEMENT] CRITIQUE: [detailed explanation] The question was the following: {original_query} The answer of the system was: {generated_answer} When generating the answer, the system had access to the following context: {retrieved_context}

Answer Generation Agent critique prompt You are evaluating the prompt used for answer generation in a RAG system. Your goal is to evaluate the prompt and propose actionable improvements based on feedback obtained from the system's output. The feedback contains strategies to improve both the retrieved content and the style of the answer; focus only on the style. Current answer generation prompt: {current_prompt} Based on this information, determine if there is a problem with the answer generation prompt that needs to be fixed. Provide a detailed critique to the prompt reporting what you learned in the previous critique. The critique must be actionable: it must include concrete actions that would likely improve the output. If false, leave the critique empty.

2. Answer Evaluation: - Assess whether the answer uses the available context effectively. - Evaluate clarity, coherence, factual accuracy, completeness, and narrative flow. - Check if it connects events and entities logically, avoids contradictions, and reads fluently in tone and length.

Answer generation prompt optimizer prompt

Decision: - SATISFACTORY: The context includes all major information and connections needed for an ideal answer, and the answer expresses them clearly and coherently. - NEEDS_REFINEMENT: Any important fact, relation, or narrative link is missing, unclear, or the answer lacks fluency, structure, or accuracy.

The current critique of the answer generation process is: {answer_generation_critique}

Critique must include: 1. What an ideal answer would contain that is not fully supported by the context. 2. Which parts of the context are irrelevant or insufficient.

You are optimizing a prompt for answer generation in a RAG system. The prompt must be general, but it also has to be as adherent as possible to the following critique. The current prompt is: {current_prompt}

Based on this critique, generate a new prompt that will be used to instruct the LLM how to better generate answers from retrieved vector information. The prompt should incorporate the feedback to improve answer quality, relevance, and coherence. Provide only the optimized prompt without additional commentary. """

Retrieval agent critique prompt You are evaluating the queries made by an iterative content retriever in a RAG system. Your goal is to provide an accurate evaluation of the queries based on the following feedback. The original question of the user was: {question} QUERIES MADE: {queries_formatted} RESPONSE EVALUATOR FEEDBACK: {response_evaluator_critique} The previous critique for the answer generator was: {answer_generator_critique} Based on the response evaluator's feedback, provide: 1. TYPES OF QUERIES TO FOCUS ON: - What types of queries would address the issues mentioned in the feedback? - What aspects should future queries target to improve the answer? - Remember that each query must target a specific part of the text. A good query targets specific content that can be found in a single chunk. Your task is not to rate the queries, but to identify what types of new queries would best address the issues raised in the feedback. Relate your suggestions directly to the user original question and the retrieved content, specifying what missing details or narrative connections new queries should aim to uncover. Provide a clear, actionable analysis.

Retrieval agent prompt optimizer You are optimizing a prompt for a query agent in a vector RAG system. The current prompt is: {current_prompt} IMPORTANT CLARIFICATION: - The planner you are optimizing asks the model to generate a sub-query to answer the question, based on the context retrieved so far and the previous sub-queries. - The planner does NOT control the retrieval/embedding mechanism itself - Your instructions should focus on which aspects a sub-query must target You will receive: 1. The original question to answer 2. The sub-queries that were made 3. A critique of the retrieval plan identifying what worked and what needs to be improved. {original_question}

{previous_queries} {retrieval_plan_critique} Based on this information, generate an optimized prompt that instructs the query agent to create a good sub-query. Provide only the optimized prompt as instructions for the query planner, without additional commentary. Generate a full optimized prompt (not partial additions) that will replace the current one used by the query agent.

A.2

Graph Retrieval

Response evaluator agent You are an expert judge evaluating both the retrieved context and the generated answer for completeness, coherence, and overall quality. Your task: 1. Context Evaluation: - First, imagine an ideal, fully satisfying answer to the question - one that captures all key facts, causal and temporal connections, character motivations, outcomes, and thematic meaning. - Then compare this ideal answer with the retrieved context. Identify what information, links, or nuances are missing, underdeveloped, ambiguous, or irrelevant. - Note any missing elements that would prevent a reader from forming a complete understanding or that weaken the logical or emotional flow of the narrative. 2. Answer Evaluation: - Assess whether the answer uses the available context effectively. - Evaluate clarity, coherence, factual accuracy, completeness, and narrative flow. - Check if it connects events and entities logically, avoids contradictions, and reads fluently in tone and length. Decision: - SATISFACTORY: The context includes all major information and connections needed for an ideal answer, and the answer expresses them clearly and coherently. - NEEDS_REFINEMENT: Any important fact, relation, or narrative link is missing, unclear, or the answer lacks fluency, structure, or accuracy. Critique must include: 1. What an ideal answer would contain that is not fully supported by the context. 2. Which parts of the context are irrelevant or insufficient.

Answer generation critique prompt You are evaluating the prompt used for answer generation in a GraphRAG system. Your goal is to evaluate the prompt and propose actionable improvements based on feedback obtained from the system's output. The feedback contains strategies to improve both the retrieved content and the style of the answer; FOCUS ONLY ON THE STYLE. Current answer generation prompt: {current_prompt} Feedback from the response evaluation: {response_evaluator_output} Based on this information, determine if there is a problem with the answer generation prompt that needs to be fixed. First, provide your reasoning explaining why there is or isn't a problem. Then, provide a critique focusing on the specific issue.

Answer generation prompt optimizer You are optimizing a prompt for answer generation in a GraphRAG system. The current critique of the answer generation process is: {answer_generation_critique} Based on this critique, generate a new prompt that will be used to instruct the LLM how to better generate answers from retrieved graph information. The prompt should incorporate the feedback to improve answer quality, relevance, and coherence. Provide only the optimized prompt without additional commentary.

Graph Extraction agent critique prompt You are evaluating the prompt given to an LLM to enrich a given graph with the information extracted from the text. The prompt tells the system how to extract entities and relationships.

You will be provided with feedback explaining which information is missing in the graph. You have to think: which entity/relationship types would make this information available in the graph? Clearly specify the entities and relationship types in your answer. Include only a few entities and relationship types (not more than 6-7). Based on this, you have to identify entity and relationship types that should be included in the graph and are not specified in the current prompt. Focus only on the most crucial entity/relationship types to meet the evaluation requirement. Specify only a few entity/relationship types, the ones that are most important. For each entity/relationship, you have to include examples (each example is a phrase or a sentence). The feedback from the system is: {response_evaluator_output} The previous critique for the answer generator was: {answer_generator_critique} The current prompt is: {current_prompt} Please, determine which entities/relationships the current prompt is missing based on the feedback.

Graph Extraction agent prompt optimizer Your goal is to generate an instruction for an LLM that enriches a graph using information from the text. The instruction must have the following format: "Focus on:" + LIST OF ENTITY/RELATIONSHIP TYPES The instruction tells the model which entity/relationship types to expand the graph with. Include only A FEW entities and relationship types (NOT MORE THAN 6-7 IN TOTAL). You have to accompany each entity/relationship with a full explanation and some examples. Each example is a phrase or a sentence. Specify only the relationships and the entities, don't use formatted examples since they can mislead the model output format. Your suggestions must be based on this feedback: {graph_extraction_feedback} Provide only the instruction without additional commentary.

LLM-as-a-Judge Evaluation Details

Model • Model: deepseek-ai/DeepSeek-V3.1 with thinking mode enabled • Size: 685B parameters Prompt Template You are evaluating two answers to a question. Your task is to determine which answer better adheres to the gold-standard reference answer. **Question:** {question} **Gold Standard Reference Answer:** {gold_standard} **Answer A (from {system_a}):** {answer_a} **Answer B (from {system_b}):** {answer_b} **Instructions:** 1. First, provide your reasoning about which answer is better and why. 2. After your reasoning, you MUST end your response with EXACTLY one of these options on the last line, with nothing else: A B Do not add any punctuation, explanation, or other text on the final line. Just the single letter or word.

Record · ID 394470 · SHA-256 1c44a2820ac40b04
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.