RAG-Stack: Co-Optimizing RAG Serving Performance and Quality

RAG-Stack: Co-Optimizing RAG Serving Performance and Quality Haiqiang Zhang

Yuanqing Lei

ETH Zurich Zurich, Switzerland zhaiqiang@ethz.ch

Columbia University New York, United States yl5457@columbia.edu

Wanting Li

Tao Zhang

Wenqi Jiang

National University of Singapore Singapore, Singapore wantingli@u.nus.edu

ETH Zurich Zurich, Switzerland zhangta@ethz.ch

National University of Singapore Singapore, Singapore wenqi.jiang@nus.edu.sg

(a) Sequential RAG

arXiv:2608.03487v1 [cs.DB] 4 Aug 2026

Abstract

×N parallel retrieval

Retrieval-augmented generation (RAG), which augments LLM generation with information retrieved from databases, has become a widely used approach for knowledge-intensive applications. Modern RAG systems, however, expose many configuration choices, such as retrieval indexes, model selections, and how models invoke retrievals. Each configuration yields a different trade-off between answer quality and serving performance, making it challenging to choose the optimal setting for a specific application deployment. In this paper, we present RAG-Stack, a framework for efficiently discovering quality–performance Pareto frontiers across diverse RAG applications and serving systems. Specifically, it consists of RAG-PE, an iterative design-space exploration algorithm that selects the next RAG configuration to evaluate; RAG-IR, a workload abstraction for diverse RAG algorithms; and RAG-CM, a performance model that predicts the optimal deployment and serving performance on the given hardware. Together, these components allow RAG-Stack to search the joint algorithm–system configuration space without deploying every candidate and to transfer an existing Pareto frontier to a new serving system. Given the same number of optimization iterations across diverse datasets, the Pareto frontiers found by RAG-Stack cover 52.5–153.2% more of the normalized quality–performance space than those found by state-of-the-art configuration-search methods evaluated over the same RAG design space.

Rewrite (prefix)

Rewrite (decode)

Embed Model

Regular LLM serving

Retrieval

Rerank

Prefix

Decode

Finish[answer]

Query

Regular LLM serving

Prefix

Decode

Answer

Retrieval Tools

Embed Model

Retrieval

Rerank

Retrieval[query] next thought

(b) Agentic RAG

Figure 1: Modern RAG system comprises diverse model components and execution workflows.

Artifact Availability: The source code, data, and/or other artifacts have been made available at https://github.com/haiqiang-zhang/rag-stack.

1

Retrieval Serving

(Optional)

Introduction

Retrieval-augmented generation (RAG) augments large language models (LLMs) with information retrieved from external data sources [24, 43, 64]. By combining models and databases, RAG addresses several limitations of model-only systems. First, RAG can incorporate upto-date information that may not be encoded in a model’s parameters [64]. Second, by grounding generation in retrieved evidence, RAG can improve factual accuracy and reduce hallucinations [60]. Third, RAG enables models to use private or domain-specific corpora that were unavailable during pretraining [62, 72]. Together, these capabilities have made RAG a widely adopted approach for knowledge-intensive applications [16, 72, 74]. Although the core idea of RAG is simple, modern RAG systems expose a large design space spanning both (1) pipeline components 1

and (2) execution workflows (Figure 1), inducing a broad spectrum of trade-offs between answer quality and serving performance. At the component level, a RAG pipeline contains a retrieval system and a generative LLM and may additionally include stages such as query rewriting and result reranking. Each RAG component can introduce a unique choice of model, prompt, and stage-specific parameters. At the workflow level, these components can be orchestrated either as a (a) sequential pipeline, in which every query follows a fixed execution order, or as an (b) agentic pipeline, in which an LLM-based controller dynamically decides when to retrieve and whether additional retrieval rounds are needed [46]. Naively enabling every pipeline component, selecting the most capable models, or maximizing the number of retrieval rounds may improve answer quality, but doing so also increases serving cost and reduces system performance (worse latency and throughput). In this paper, we ask: How can we efficiently explore this quality–performance trade-off space and identify the Pareto frontier across diverse RAG applications and serving systems? However, identifying the Pareto frontier for a given RAG application and serving system requires solving three key problems. (P1) Existing RAG configuration exploration algorithms either neglect cross-stage interactions or overlook per-stage signals. One approach to explore RAG configurations is to proceed stage by stage: it identifies the best configuration for one stage, holds that configuration fixed, and then optimizes the next stage [3]. However, the best end-to-end configuration depends on interactions among stages and therefore cannot generally be obtained by optimizing each stage independently. For example, increasing the retrieval top-𝑘 can improve quality when a strong reranker filters irrelevant passages or a long-context LLM effectively uses the additional evidence, but can

RAG-Stack Mode 1 Input Optimizing RAG from Scratch

Users

Input1: Input2: Algorithm Config. Space RAG Dataset

Input3: Optimization Target

resources without requiring deployment on the target system. Finally, RAG-PE uses the measured quality and predicted performance to select the next configuration. This loop continues until a userspecified stopping criterion or search budget is reached. We now introduce the main components in RAG-Stack. First, RAG-PE (Plan Exploration) is the core multi-objective Bayesian optimizer in RAG-Stack. It takes as input the algorithm design space and feedback from previous trials, including quality measurements on the dataset and performance estimates produced by RAG-CM. Its output is the next algorithm configuration to evaluate, which is sent through the RAG execution path and subsequently represented by RAG-IR. RAG-PE addresses P1 by jointly optimizing end-to-end quality and serving performance while using intermediate stage-level quality signals to guide exploration. Second, RAG-IR (Intermediate Representation) bridges RAG-PE and RAG-CM by translating each RAG algorithm configuration into a workload representation comprising a workflow schema and an execution trace of stage invocations and input/output sizes. The output of RAG-IR is a system-agnostic workload representation that decouples each stage’s logical work from its physical deployment, enabling RAG-CM to explore deployment choices and re-estimate the performance of the same workload on new hardware. Together with RAG-CM, this representation addresses P2 and P3. Third, RAG-CM (Cost Model) is an ML–analytical fusion performance model. It takes as input the workload representation produced by RAG-IR, the system design space, and the user’s available hardware resources. It then searches the system design space internally and returns the best predicted deployment and serving performance for the current RAG configuration to RAG-PE, thus effectively addressing P2. Moreover, RAG-CM can re-evaluate the archived algorithm configurations under a new hardware configuration without repeating their quality evaluations, enabling efficient frontier transfer and further addressing P3. We evaluate RAG-Stack on RAGEval [74] and MS MARCO [9] using two system configurations: one equipped with four NVIDIA H100 GPUs and the other with eight NVIDIA A100 GPUs. With the same number of exploration iterations, the Pareto frontiers found by RAGStack, averaged across seeds, cover 52.5% and 153.2% more of the normalized quality–performance space on RAGEval and MS MARCO, respectively, than those found by state-of-the-art configurationsearch methods evaluated over the same RAG design space. When transferring a Pareto frontier to a new serving system, RAG-Stack reuses quality measurements from previous evaluations and performs only a small number of additional optimization iterations to discover the new frontier. The resulting adapted frontier covers 182.2% more of the normalized quality–performance space than the frontier obtained by re-optimizing from scratch on the new system. In summary, the paper makes the following contributions:

Input 4: System Resource

RAG-Stack Quality Evaluation

RAG-PE

Explore algorithm configs Algorithm Config.

Algorithm Config. Space Intermediate Representation

RAG-IR

Abstract the workload Performance Evaluation

Workflow Schema

RAG-CM

Search + Predict the Performance

Trace

System Config. Space

RAG-Stack Mode 2 Input Migrating to New System

Output Pareto-Optimal RAG Configurations

Input 1 Input 2: New System Resource

Figure 2: Overview of RAG-Stack and its two operating modes. degrade quality when noisy passages reach a weaker generator without reranking. By contrast, global RAG optimization captures these interactions by jointly tuning parameters across all stages using end-to-end accuracy as the optimization signal [12]. However, this end-to-end signal does not reveal which stage is responsible for a gain or loss, potentially leading to poorly directed exploration and wasted evaluations. (P2) Existing multi-objective RAG optimizers overlook the system design space. Existing quality–performance optimizers search over algorithmic configurations while holding the serving deployment fixed [12, 57], even though different algorithms may require different batching policies, parallelization strategies, and stage placements to achieve their best serving performance [30]. (P3) Deployment-based serving-performance measurements are costly and do not transfer across systems. Measuring serving performance requires deploying and benchmarking each candidate configuration, and the resulting measurements are specific to the target system [45]. When the application is moved to a system with different hardware, configurations on the original Pareto frontier may no longer be Pareto-optimal, requiring the search to be repeated to reconstruct the frontier [28, 50]. To address these problems, we present RAG-Stack, an efficient framework for discovering the quality–performance Pareto frontier across arbitrary RAG applications and serving systems (Figure 2). RAG-Stack takes four inputs from the user: (1) an algorithm design space, (2) a new RAG application (i.e., an evaluation dataset), (3) an optimization target comprising quality and performance requirements, and (4) the available system resources. RAG-Stack can either discover a Pareto frontier from scratch or transfer an existing frontier to a new system by reusing archived quality measurements. For a new RAG application, RAG-Stack discovers the Pareto frontier with an iterative optimization loop. First, RAG-PE proposes an algorithm configuration for quality evaluation on the application dataset. Next, RAG-IR abstracts the executed workflow, and RAG-CM predicts its best achievable serving performance under the available

• We present RAG-Stack, an end-to-end framework that efficiently discovers the Pareto frontier between RAG answer quality and serving performance for arbitrary applications and systems. • We design RAG-PE, a multi-objective, sub-metric-aware Bayesian optimizer that uses both end-to-end and intermediate signals to efficiently navigate the RAG configuration space.

2

2.2

0.1

4×

0

0.1

(a) 80

0.4

60 40

0.2 0.0

RAG retrievers use vector search to match queries with passages based on semantic similarity rather than exact lexical overlap. Before serving, corpus passages are embedded and indexed. At query time, the retriever embeds the query and returns the nearest indexed passages. Exact search scans the entire corpus and guarantees the true nearest neighbors under the chosen metric, but scales poorly. Production systems therefore rely on approximate nearest-neighbor (ANN) indexes, which prune most candidates to reduce latency and increase throughput, but may lower recall and consequently degrade answer quality when relevant evidence is missed. Two index families. ANN indexes are either clustering-based or graph-based. The IVF (inverted-file) family is clustering-based: it groups the vectors into many lists, and for each query scans only the few lists closest to it. Within this family, IVF-Flat keeps the full vectors and favors recall; IVF-PQ compresses each vector into a short code to save memory and bandwidth; and IVF-PQ FastScan speeds up the distance computation with SIMD-friendly kernels [19, 29]. HNSW is graph-based: it links the vectors into a navigable graph and answers a query by walking the graph greedily toward the nearest ones. HNSW often reaches high recall at low latency, but it needs extra memory to store the graph [51]. Knobs trade recall for speed. Larger nprobe in IVF or efSearch in HNSW examines more candidates, improving recall at the cost of latency and throughput; IVF-PQ reduces memory use but may lower recall. Because the best choice depends on the workload and deployment, FAISS is well suited to studying these trade-offs: it directly exposes both index families and their CPU/GPU knobs [19], unlike vector databases such as Milvus, whose serving layer hides many low-level choices [65].

(b) 8×

0.6

20 128 256 512 1K 2K

Chunk size

2× 0.0

1×

Median QPS ratio

2 Background and Motivation 2.1 Vector Search for Retrieval

0.2

Median QPS Median Quality

Median quality

• We introduce RAG-CM, a hybrid ML-analytical performance model that predicts serving performance and searches optimal deployment configurations on the given hardware.

0.5×

ct n n n RieoAn okffe/or ofsf/so. off/o s v . s SeEqxpanReraCnompre

Figure 3: Median quality–throughput tension across algorithm choices. (a) By chunk size. (b) Single-choice contrasts (quality: first−second; QPS: first/second); error bars: interquartile range.

parallelism they use, how requests are batched, and how the index and runtime knobs are configured. A large body of systems work optimizes this serving layer for a fixed RAG algorithm—tensor [59], pipeline [26, 27], and data [44] parallelism, continuous batching [70], and paged KV-cache management [40]—together with RAG-specific techniques such as adaptive pipeline parallelism [32] and disaggregated accelerators [31].

2.3

Motivation: Performance and Quality Co-optimization for RAG

The discussion above of RAG pipelines and serving shows that cooptimizing answer quality and serving performance better reflects the practical needs of RAG serving. This view also accounts for RAG serving-system design, because placement, batching, and parallelism determine how efficiently each pipeline runs. Manual cooptimization is difficult because measurements reveal trade-offs that intuition misses (Figure 3): agentic ReAct [69] can be faster than a sequential pipeline at a modest quality cost, query expansion sacrifices throughput for a quality gain that appears only in some configurations, and even a larger generator does not always improve quality [73]. A configuration that avoids such unrewarded cost can deliver the same quality at higher throughput and thus dominate alternatives; only non-dominated configurations are worth serving. Finding the frontier is a search problem. Let X denote a candidate space and y(x) ∈ R𝑚 the objective vector of x ∈ X, with all objectives oriented so that larger values are better. An objective point y dominates y′ if y ≥ y′ componentwise with at least one strict inequality [18]. A candidate is non-dominated if no other candidate’s objective point dominates its own; the objective points of all non-dominated candidates form the Pareto frontier F . The hypervolume HV(F ;r) is the volume of the objective-space region that is dominated by F and dominates a fixed reference point r ∈ R𝑚 that lower-bounds F ; a larger hypervolume indicates broader coverage of desirable trade-offs [17]. Throughout, objectives are min–max normalized per experiment and dataset, with r =0. The frontier thus provides a set of operating points from which users can choose. A general configuration-search method finds it iteratively: it proposes a candidate, evaluates its objective vector, and uses the observations collected so far to select the next candidate. Because this search

RAG Pipelines and Serving

A RAG pipeline is built from a few stages. The two core stages are retrieval (§2.1) and generation: the retriever runs a vector search over the corpus, and the retrieved passages—optionally after query rewriting or reranking—are placed in a prompt for an LLM that prefills and decodes. The same stages can be wired into different workflows. The simplest pipeline retrieves once and generates once. Iterative and active-retrieval workflows interleave several rounds of retrieval and generation to improve the answer [6]. An agentic pipeline goes further: an LLM controller decides at run time which stages to run, when to retrieve, and whether to iterate [13, 46, 61]. Choosing the components and the workflow is already a quality-tuning problem, and frameworks such as FlashRAG let users assemble, swap, and evaluate these pipelines to raise answer quality [33]. The choices that raise quality also raise the work per request. A larger top-𝑘, an added reranker, iterative retrieval, or a bigger generator each helps the answer, but each also adds retrieval work, model calls, prompt tokens, or decoding time. Serving then adds a second layer of choices that leave the answer unchanged but set how fast it is produced: how the stages are placed on the hardware, how much 3

Quality (ans. corr.)

chunk 512 chunk 256

+32%

0.25 k=8

+16%

no compression compression 0.3

k = 16 k=1

0.20

k=1 k=1

k=1

0.15

-14%

-20% k = 16

k=8

150

parameters within that deployment [57]; neither finds an algorithm configuration’s best achievable serving performance. Simply folding the full system design space into the multi-objective search is wasteful because system parameters do not affect answer quality, yet every variant would still incur a full quality evaluation. (P3) Deployment-based serving-performance measurements are costly and non-transferable across systems. RAG serving performance is typically obtained by deploying and timing each configuration end to end, making dense deployment search prohibitively expensive [45, 47]. Such measurements are system-specific: a configuration on the original Pareto frontier may become dominated on a new system, so migration requires remeasurement and frontier reconstruction; candidate hardware also cannot be evaluated before acquisition. Although prediction could avoid these costs, existing RAG performance models cover only narrow portions of the stack [30, 37], leaving no portable full-stack estimator.

250

350

25

30

35

40

Performance (QPS)

Figure 4: Cross-stage interactions in the algorithm design space. Each arrow traces one configuration as retrieval top-𝑘 rises; annotated percentages are the resulting quality change. requires only X and the evaluated objective values, it can in principle use any multi-objective optimization method. The next section shows why such methods, applied directly to RAG, fall short.

2.4

Limitations of Existing Approaches 3

Existing RAG-specific optimizers cover only parts of the quality– performance problem. A RAG configuration spans two design spaces: the algorithm design space, whose choices affect answer quality— chunking, top-𝑘, reranking, and the generator—and the system design space, whose choices affect how fast that answer is served— placement, batching, parallelism, and hardware. Quality-side tools search the algorithm design space against answer quality alone [3, 22, 33]. System-side tools such as RAGO search the system design space for serving performance, but only for a fixed RAG algorithm [30]. Online methods such as METIS adapt a small set of serving choices per query, but they also operate within a fixed deployment [57]. A natural alternative is to treat RAG tuning as a generic multiobjective black-box optimization problem. Recent work takes this route by applying standard multi-objective Bayesian optimization, including LogNEHVI, to search RAG hyperparameters for quality– cost trade-offs [12]. More broadly, possible black-box optimizers include plain sampling, Bayesian optimization [10, 42, 48, 53, 66], evolutionary search [14, 25], and multi-fidelity optimization with cheaper proxy runs [35]. These optimizers can also be steered by LLMs [1, 41, 52] or extended to the multi-objective setting through scalarization [38, 55], dominance [18, 71], or hypervolume criteria [4, 17]. These general tools are our starting point. Applied directly to RAG, however, they and the RAG-specific optimizers above run into three problems. (P1) Existing RAG optimizers either break cross-stage interactions or lose stage-level signals. Figure 4 shows that increasing retrieval top-𝑘 always reduces throughput, but its quality effect flips with chunk size and compression, demonstrating that stage choices cannot be optimized independently. Stage-wise optimizers such as AutoRAG retain stage-level feedback but miss configurations that work only through cross-stage combinations [3]; global optimization accounts for these interactions by tuning complete pipelines, but end-to-end feedback alone cannot reveal which stage drives a gain or loss, leading to poorly directed exploration and wasted evaluations. (P2) Existing multi-objective RAG optimizers overlook the system design space. Existing quality–performance RAG optimizers search algorithm choices but measure performance on one fixed deployment [12], while online methods adapt only a few serving

RAG-Stack: System Overview

We present RAG-Stack (Fig. 2), an efficient framework that finds a RAG system’s quality (𝑄)–performance (𝑃) Pareto frontier across the full algorithm and system design space, without deploying each candidate to measure its performance. This addresses the limitations in §2.4, which leave RAG without a practical way to co-optimize answer quality and serving performance across its full design space. Inputs. In from-scratch mode, a user drives RAG-Stack with a single declarative specification of four things: (i) the algorithm design space to search, a hierarchical, conditional space of components and parameters detailed in §4.1 (Fig. 5); (ii) an evaluation dataset of queries with reference answers; (iii) an optimization target—the quality and performance metrics to trade off, plus serving SLOs; and (iv) the available system resources—GPUs, CPU, and interconnect— bounding the system design space RAG-CM searches (Table 1). In system-transfer mode, the input is instead the previous Pareto result and a new system-resource specification, as described under Two operating modes below. Output. RAG-Stack returns the Pareto-optimal configurations it found—each a complete algorithm-and-system deployment—so the user picks the operating point that fits their workload, hardware, and SLO. Overview. RAG-Stack (Fig. 2) rests on one split: a RAG configuration’s parameters divide into an algorithm design space, whose choices change the produced answer and so move both quality and performance, and a system design space, whose choices change only how that fixed computation is served. RAG-Stack searches the two in an iterative loop over three components. Each round, RAG-PE (Plan Exploration, §4) proposes one algorithm configuration x; the quality evaluator runs that pipeline on the dataset and returns its quality 𝑄 (x); RAG-IR (Intermediate Representation, §5) abstracts the executed run into a workload representation; RAG-CM (Cost Model, §6) searches the system design space for that representation and predicts x’s best serving performance 𝑃 (x); and RAG-PE uses the measured 𝑄 (x) and predicted 𝑃 (x) to choose the next configuration. The loop stops when the optimization target is met or the budget is spent. Quality comes from this one evaluation run, but performance is always predicted by RAG-CM—no candidate is ever deployed to measure it. 4

Two operating modes. The loop above optimizes a RAG system from scratch, discovering its frontier on the current hardware. RAG-Stack also supports a system-transfer mode: to retarget an already-optimized system to new hardware, the user provides only the previous Pareto result and a new system-resource specification. RAG-Stack then reuses the quality results from the first run, first invoking RAG-CM to re-score the previous deployments under the new resources and subsequently running a small number of RAG-PE polishing iterations to refine the transferred frontier. Benefits. RAG-Stack’s design yields three benefits, each removing one limitation of prior optimizers. B1 (solving P1): stage-aware multi-objective optimization without breaking cross-stage interactions. RAG-PE still optimizes the end-to-end (𝑄,𝑃) Pareto frontier, so each trial is a complete pipeline configuration and crossstage interactions are preserved. At the same time, it reads stage-level sub-metrics to guide exploration: for example, low context recall with high faithfulness points to retrieval as the bottleneck rather than generation. These sub-metrics guide where RAG-PE searches next without becoming separate objectives, giving one optimizer both global frontier optimization and stage-level diagnosis (§4.2). B2 (solving P2): near wallclock-free exhaustive system search. Given the algorithm/system split above, RAG-PE spends expensive quality evaluations only on algorithm configurations, while RAG-CM exhaustively searches the system design space for each one inside the cost model. Because this search uses predicted performance rather than real deployments, it is nearly wall-clock-free, letting RAG-Stack report the best deployment-side operating point instead of inheriting the performance of one fixed deployment (§4.1, §6). B3 (solving P3): cheap system transfer and planning. Because RAG-CM predicts performance from a hardware description instead of a real run, an optimized system is retargeted to new hardware by re-scoring the quality archive, and candidate machines are compared before purchase—neither step builds a real deployment. We now detail the three components in turn: RAG-PE (§4), RAG-IR (§5), and RAG-CM (§6).

4

computation. An algorithm parameter changes the logical computation (Fig. 5); conversely, a parameter belongs to the system design space whenever it leaves the logical computation unchanged. System parameters change only how that fixed computation is executed and served (Table 1)—placement, batching, parallelism, thread counts, and hardware. Batching, for example, merely groups identical perrequest computations and so leaves the answer unchanged; varying parallelism or thread count may perturb the output slightly through floating-point reduction order, but the logical computation remains unchanged. Formally, let 𝐼𝑌 denote the parameters relevant to objective 𝑌 ∈ {𝑃,𝑄 }. Any parameter relevant to answer quality 𝑄 changes the logical computation and must also be considered for serving performance 𝑃; therefore, 𝐼𝑄 ⊆ 𝐼𝑃 . This yields Xfull = Xalgo × Xcm , where Xalgo := Î Θ contains parameters that affect both 𝑄 and 𝑃, while Xcm := Î 𝑗 ∈𝐼𝑄 𝑗 𝑗 ∈𝐼𝑃 \𝐼𝑄 Θ 𝑗 contains parameters that affect only 𝑃. RAG-Stack optimizes Xalgo against (𝑄,𝑃) and delegates Xcm to RAG-CM for 𝑃 alone. Table 1: System design space searched by RAG-CM. Design choice

Example range

Vector DB

Search threads Parallel mode

{16,32,64} {intra-query, inter-query}

Parallelism strategy LLM serving GPUs per stage Prefill–decode deployment

{TP, PP, DP, hybrid} {1,2,3,4} {collocated, disaggregated}

Placement

Stage-to-device mapping

Prefill → GPUs {0–7}

Request batch size Decode batch size Dynamic batching wait time

{1,2,...,256} {16,32,64,128,256} {0,1,2,5,10}ms

CPU model GPU inventory

{AMD EPYC, Intel Xeon} {4×H100, 8×A100}

Batching

Hardware (Optional)

RAG-PE: Plan Exploration

4.1.2 Search-space organization. RAG-PE parses the user’s declarative specification into the two design spaces above. System-only choices in Xcm are sent to RAG-CM (§6), which exhaustively searches them inside the cost model to return the best predicted performance 𝑃˜ (x) for a given algorithm configuration. RAG-PE therefore runs the expensive ground-truth optimization only over Xalgo , while RAG-CM handles the internal system search. It organizes Xalgo as a hierarchical design space (Fig. 5) governed by two rules: conditional activation and heuristic constraints that remove invalid or low-value trials. Under conditional activation, a child parameter is active only when its parent choice is active; for example, query-rewriter parameters are searched only if query rewriting is enabled. The heuristic constraints (starred nodes in Fig.√5) use smooth proxies for derived knobs: IVF sets nlist =factor· 𝑁 for corpus size 𝑁 , while PQ selects a valid divisor 𝑀 of the embedding dimension 𝐷.

RAG-PE partitions the design space (§4.1), runs the RAG-specific optimizer (§4.2), and coordinates quality evaluation with performance modeling (§5, §6).

4.1

Category

Search Space

The space of RAG configurations is large and hierarchical. RAG-PE splits it into an algorithm part and a system part, searches the algorithm part directly, and delegates the system part to RAG-CM. We define this split and then organize the part RAG-PE searches as a hierarchy. 4.1.1 Two Design Spaces: Algorithm and System Design Spaces. Let Î Xfull = 𝑛𝑗=1 Θ 𝑗 be the full configuration space of the RAG stack, with each Θ 𝑗 denoting the domain of one parameter (e.g., nprobe, thread_count, top_k), and let 𝑄 (x) and 𝑃 (x) be the answer-quality and serving-performance objectives under the user’s chosen metrics (e.g., latency, throughput, or SLO satisfaction). We partition the parameters by what each one changes: the logical computation that produces the answer, or only the physical execution of that fixed

4.2

The Optimizer

As discussed in §2.4, RAG’s algorithm design space is entangled by cross-stage interactions: the best setting for one stage depends on the choices made in the others (Fig. 4). Optimizing stages independently 5

RAG workflow

Algo. Design Space

agentic sequential

corpus chunker

query rewriter

chunk size

chunk type

128

2048

…

enabled

LLM model

1.5B

disabled

temperature

…

top-K

index

embedding model

IVF

HNSW

384D

…

quantization

multi query

HyDE

14B

nlist factor

1

4

pq

6

…

1024D

4

…

enabled

8

passage compressor

main LLM

temperature

disabled

enabled

types

top-K

LLMLingua-2

ColBERT

1

heuristic aware: M should be relative to the embedding dimension D — chosen so that dsub = D/M falls in the sweetspot range (≈4–16), subject to D mod M = 0.

nbit

32

…

disabled

tart

flat

dsub (D/M) heuristic aware: nlist = factor · √N; N depends on corpus size & chunk size, only factor is tuned

passage reranker

retrieval

…

64

1.5B

LLM model

…

14B

rate

0.3

…

0.7

Selected Retrieval-side Parameters Selected Model-side RAG Parameters

Figure 5: Algorithm design space [23, 29, 36, 51, 54] searched by RAG-PE. The highlighted selected nodes are one example, showing how a single evaluation’s algorithm configuration is assembled: each trial fixes one value at every active branch of the hierarchy (retrieval-side in purple, model-side in yellow), and the starred callouts mark heuristic-aware parameters coupled across branches. Let D𝑡 denote the data after 𝑡 rounds. The hypervolume improvement (HVI) of a candidate outcome y over F is HVI(y | F , r) = HV(F ∪ {y};r) − HV(F ;r). HVI alone cannot score a candidate, because the outcome y is unknown before the evaluation. Expected hypervolume improvement (EHVI) resolves this by averaging HVI over the GP posterior prediction at x, measured against the frontier of the outcomes observed so far [17]. EHVI, however, trusts those observations to be exact, while our quality scores are noisy: the observed outcomes need not form the true frontier. Noisy EHVI (NEHVI) also treats the frontier as uncertain, averaging HVI over posterior draws of the evaluated configurations and the candidate: 𝛼 NEHVI (x) =Ef ∼𝑝 ( f | D𝑡 ) [HVI(f (x) | Ff ,r)]. We estimate this expectation using 𝑁 Sobol draws from the joint GP posterior; each draw f 𝑗 induces a frontier F𝑗 over the evaluated configurations [17]. In practice, we maximize LogNEHVI, a numerically stabilized variant of NEHVI, using the BoTorch implementation [4, 10]. Each round, RAG-PE fits the GPs, picks the next configuration by maximizing the acquisition over the whole space, x𝑡 +1 =argmaxx ∈ Xalgo 𝛼 LogNEHVI (x), evaluates it, and appends the result. These considerations lead us to choose GP+LogNEHVI over SMAC’s 4.2.1 Preliminary: multi-objective Bayesian optimization. Multi-objective combination of a random-forest (RF) surrogate and ParEGO [38, 48]. Bayesian optimization (MOBO) follows the general Pareto-search
LogNEHVI directly rewards expected improvement in the noisy formulation in §2.3. In our setting, X =Xalgo and y(x) = 𝑃 (x),𝑄 (x) , Pareto hypervolume, whereas ParEGO targets the frontier indirectly using the serving-performance and answer-quality objectives dethrough scalarized objectives [17]. The RF instead learns similarity fined above. Because evaluating quality is expensive, MOBO seeks through tree partitions; its piecewise-constant predictions can create to identify the frontier using as few evaluations as possible. plateaus in the acquisition landscape, leaving little signal for local MOBO consists of two components. The first is a cheap probarefinement. bilistic surrogate for each objective—a Gaussian process (GP) fitted Despite this principled acquisition rule, the standard loop is poorly to all configurations evaluated so far. Because Xalgo contains many directed when applied to RAG’s large hierarchical design space. The categorical variables and has a hierarchical structure, each GP uses a surrogate is fitted only on the two end-to-end objectives, so its posmixed kernel: a Matérn kernel for the numeric knobs and a Hamming terior uncertainty indicates where the search has not looked, but kernel for the categorical ones. Together, they provide an appropriate not why a configuration succeeds or which pipeline stage is worth notion of similarity for this design space [42]. changing. Under a tight quality-evaluation budget, it can therefore The second ingredient is an acquisition function 𝛼 that scores any spend trials on configurations that are novel yet unlikely to be useful. unevaluated configuration by its expected gain in this hypervolume.

can therefore miss configurations that work well only as a complete pipeline. RAG-PE consequently uses multi-objective Bayesian optimization (MOBO) as its global foundation, evaluating and ranking complete configurations against the end-to-end (𝑄,𝑃) Pareto frontier. MOBO also suits the small evaluation budget: its probabilistic surrogate and acquisition function use prior observations and uncertainty to select the configuration expected to improve the frontier most. A global optimizer alone, however, sees only the end-to-end objectives and lacks directional information about which stage should change. RAG-PE therefore augments the global MOBO with stagelevel feedback: stage diagnostics guide exploration toward promising changes, while the global acquisition function still arbitrates complete configurations according to how much they are expected to improve the (𝑄,𝑃) frontier. Stage information thus directs the search without becoming a separate objective or breaking cross-stage interactions. We first introduce the MOBO foundation and then present our stage–global co-aware extensions.

6

Pareto-tension crossover

2

3

The stage-guided candidates and quasi-random restarts R𝑡rand seed discrete local search (DLS, 2 ). Let z (𝑘 ) denote the discrete configuration after 𝑘 DLS steps, and let N (z) denote its hierarchyvalid one-parameter neighbors. Each c𝑚 initializes a DLS trajectory at z (0) = c𝑚 . Writing N + (z) = N (z) ∪ {z}, DLS greedily applies z (𝑘+1) = argmaxz ∈ N + ( z (𝑘 ) ) 𝛼 LogNEHVI (z) until convergence. Thus, the agent chooses a promising region and DLS refines the configuration within it. Pareto-tension crossover ( 3 ) starts from the best observed configuration for each objective and applies the single-knob crossover that most strongly moves it toward the other objective. For 𝑜 ∈ {𝑃,𝑄 }, denote that anchor by x𝑜★ =argmaxx ∈ D𝑡 𝑜 (x) and the other objective ★ denote the hierarchy-valid crossover that re¯ Let x𝑜,𝑗 =x𝑜★ ⊕ 𝑗 𝑥𝑜,𝑗 by 𝑜. ¯ places only knob 𝑗 with its value from x𝑜★¯ . It selects 𝑗𝑜★ =argmax 𝑗 𝜏𝑜,𝑗 , where hats denote min–max normalization and 𝜏𝑜,𝑗 = 𝑜¯ˆ (x𝑜,𝑗 ) − 𝑜¯ˆ (x𝑜★) − [𝑜ˆ (x𝑜★) − 𝑜ˆ (x𝑜,𝑗 )] + , yielding R𝑡× . The candidate pool is
C𝑡 = B𝑡 ∪ R𝑡LLM ∪ DLS R𝑡LLM ∪R𝑡rand ∪ R𝑡× , with DLS applied independently to each seed and source labels retained for arbitration ( 4 ).

4 Discrete Local Search (DLS)

1

Agentic Analyzer

Gaus

sian

Proc

esse

GPs know e2e-score only

s (GP

s)

Agentic Analyzer know per-stage and e2e score Input

Rewrite (prefix)

per-stage score of Q & P

Embed Model

Retrieval

Arbitrate e2e-score of Q&P

Rerank

Candidate Pool

Main LLM

Eval. Result (Q & P)

5

Forced channels LogNEHVI Quality reuse

Output Next Algo. Config for Evaluation

Figure 6: The optimizer of RAG-PE. 4.2.2 Our optimizer: stage–global co-aware MOBO. RAG-PE extends conventional MOBO for RAG with stage-aware diagnostics, heterogeneous candidate generation, and global arbitration, making exploration more directed while preserving end-to-end Pareto optimization. This design is necessary because stage effects are nonseparable—the benefit of changing one stage depends on the choices made in the others (Fig. 4). Greedy stage-wise optimization can therefore miss configurations that work well only as a complete pipeline [3]. Composite-function BO also exploits intermediate outputs, but assumes that the final objective is a known function of those outputs [7, 39]. RAG provides useful per-stage diagnostics, but no known function maps them to end-to-end quality and performance. RAG-PE therefore uses these diagnostics to direct candidate generation while leaving complete-configuration comparison to the global MOBO objective. RAG-PE combines three mechanisms, shown in Fig. 6 and detailed below.

(3) Arbitration. On ordinary rounds, LogNEHVI scores every candidate in C𝑡 , and RAG-PE evaluates the candidate with the largest expected hypervolume improvement, independent of its generator. When the number of consecutive ordinary evaluations with no realized normalized-hypervolume gain reaches a stagnation threshold 𝜏𝑠 , RAG-PE activates its forced channels. These channels target the largest gap between adjacent points on the normalized realized Pareto frontier and draw candidates from the agent and Paretotension crossover. The candidates bypass LogNEHVI and are ranked by the product of their range-normalized posterior standard deviations, with hierarchy-aware novelty breaking ties. Finally, quality reuse skips generation and judging when the retrieved passages and quality-relevant downstream configuration match an earlier evaluation: RAG-PE reuses its quality score, recomputes performance with RAG-CM, records the point, and repeats arbitration.

(1) Sub-metric Awareness. RAG-PE optimizes only end-to-end quality 𝑄 and performance 𝑃, while its agentic analyzer ( 1 ) uses stage-level sub-metrics to diagnose where a configuration falls short. The quality diagnostics are context recall and precision for retrieval and answer faithfulness for the Main LLM; the performance diagnostics are RAG-CM’s per-resource capacities 𝜅𝑟 (§6.5), which expose the bottleneck stage and remaining headroom. We record them as s𝑖 𝑡 for the GPs alongside each observation, yielding D𝑡 ={(x𝑖 ,𝑃𝑖 ,𝑄𝑖 )}𝑖=1 𝑡 and H𝑡 ={(x𝑖 ,𝑃𝑖 ,𝑄𝑖 ,s𝑖 )}𝑖=1 for the analyzer. Reading H𝑡 , the analyzer forms 𝑀𝑡 stage-guided candidate configurations R𝑡LLM ={c1,...,c𝑀𝑡 }. Each c𝑚 =b𝑚 ⊕𝜹𝑚 ,𝑚 =1,...,𝑀𝑡 , is constructed by applying a targeted edit 𝜹𝑚 to a previously evaluated base b𝑚 and snapping the result to the valid grid; ⊕ overwrites only the edited parameters [41, 49]. Thus, sub-metrics guide candidate generation without becoming surrogate inputs or optimization objectives.

5

RAG-IR: Intermediate Representation

RAG-IR bridges quality evaluation and RAG-CM through a common workload representation with two properties. It is system-agnostic, separating logical work from deployment so RAG-CM can search deployment choices and re-cost the same workload on new hardware without rerunning it. It is RAG-workflow-agnostic, encoding sequential and agentic pipelines with a common schema so RAGPE can explore workflows while RAG-CM evaluates them without workflow-specific modeling logic (Fig. 2). RAG-IR records an order-free workflow schema for throughput and per-request execution traces for latency (§6.5). Workflow schema. An order-free summary of each stage’s performance-relevant attributes and aggregate logical work, used by RAG-CM to predict throughput. Per-request trace. For each request 𝑞, RAG-IR records an execu(𝑞) tion DAG G𝑖 . Each node is one stage invocation and stores its stage type and input/output token counts; edges record dependencies between calls. Repeated calls in iterative or agentic pipelines appear as separate nodes, preserving their order and multiplicity. RAG-IR forwards the schema and traces, together with x𝑖 , to RAG-CM (§6).

(2) Heterogeneous Candidate Pool. Standard acquisition optimization uses multiple random restarts, but all candidates are refined against the same surrogate and therefore inherit the same model bias. RAG-PE instead constructs C𝑡 from four complementary channels: Sobol coverage, stage-guided candidates, DLS-based acquisitionguided local refinement, and Pareto-tension crossover (Fig. 6). Two channels determine where to search: Sobol breadth draws a hierarchy-aware, space-filling candidate set B𝑡 , while the agentic analyzer supplies stage-guided candidates R𝑡LLM ( 1 ). 7

6

5

RAG-CM: Cost Model

GPU ↔ CPU / GPU ↔ GPU

6.1

Communication Type (PCIe, NVLink …)

Communication Modelling

RAG-CM is the performance-estimation component of RAG-Stack. For a fixed RAG algorithm configuration x𝑖 , it estimates the best attainable serving performance on a target system by searching the system design space Xcm , whose parameters affect serving performance but not answer quality. Its inputs are the workload representation produced by RAG-IR (a workflow schema and per-request execution traces G𝑖 ), the algorithm configuration x𝑖 , and a resource specification of the target system’s devices, communication topology, and serving resources. We denote by s ∈ Xcm one concrete assignment of all systemdesign parameters (Table 1). For each pair (x𝑖 ,s), RAG-CM predicts throughput and end-to-end latency, p̃(x𝑖 ,s). Why a cost model for performance. Predicting performance instead of measuring it serves four roles in RAG-Stack. (R1) A smaller optimizer space. By moving the system design space into RAGCM, the optimizer only searches the algorithm design space Xalgo , so the scarce ground-truth budget is spent only on parameters that affect quality (§4). (R2) RAG serving planning and transfer. RAGCM lets users plan RAG serving before acquiring the target system and efficiently retarget it to new hardware by re-scoring the serving performance of previously identified Pareto configurations on the new system. (R3) Faster optimization iterations. Because RAG-CM predicts performance without deploying each candidate or benchmarking its performance, every optimization iteration avoids this overhead, substantially reducing wall-clock time relative to deployment-based optimizers. (R4) Exhaustive system search. RAG-CM exhaustively searches the system design space for each algorithm configuration, yielding higher performance and more effective optimization. ˜ The four-layer structure. RAG-CM computes p̃, and hence 𝑃, in four layers (Fig. 7): an algorithm layer that produces one hardwareagnostic Operator Work Profile per RAG stage (§6.1), a performance layer that maps each Operator Work Profile to time on the host (§6.3), a communication layer that prices the data moved between operators (§6.4), and an assembly layer that sweeps the system design space Xcm and composes the other three layers into the throughput and latency of each algorithm–system configuration pair (§6.5).

Input RAG-IR

Embed Model

Retrieval

Rerank

Prefix

Decode

2

1 IVF Family Operator

A

Prefill Operator

ops

Decode Operator A

HNSW Operator

A

bytes

bytes location

ML

access block

Algorithm Modelling

parallel fraction

3

ops

bytes

4 NUMA

compute mode Operator Work Profile

6

big.LITTLE

CPU Topology

RAG-CM Assembly

Effective Cache

4

Get E2E RAG Performance

Roofline Model Scattered sequential Memory

Search System Design Space

flops Parallel Model

bandwidth

Performance Modelling

Output Latency

Throughput

Figure 7: Overview of RAG-CM. A: analytical; ML: fused machine learning and analytical. Table 2: IVF-family search stages (Flat: IVF-Flat; PQ: IVF-PQ; FS: IVF-PQ Fastscan; FSR: Fastscan with residual). Stage

Compute Mode

Peak Used by

1. Coarse quantizer BLAS GEMM if per-thread batch ≥ 20, else FP32 all hand-vectorized SIMD distance kernel 2. LUT build

non-residual, built once/query (FS); residual-dependent (PQ, FSR)

FP32 PQ, FS, FSR

3a. PQ scan

ADC table lookup (PQ) / vpshufb (FS, FSR)

int

PQ, FS, FSR

3b. Flat scan

full-vector SIMD L2

int

Flat all

4. Top-𝑘 heap

branch-heavy integer

int

5. Refinement

exact SIMD L2 on candidates

FP32 optional

from the FAISS source: the ops, bytes, and access pattern of each kernel are fixed by the configuration, so a closed-form formula gives the profile for each such stage listed in Table 2, with variant-specific formulas where the kernels differ; for example, standard PQ builds its distance table per probed list, whereas Fastscan builds it once per query. Only the scan stage requires special treatment because its work cannot be read directly from the configuration, so we model it below.

Algorithm Modeling

Given RAG-IR’s workflow and execution trace, the algorithm layer models every RAG stage and outputs one hardware-agnostic Operator Work Profile per stage—a record of that stage’s per-phase operation counts, data movement, and execution characteristics ( 1 in Fig. 7; §6.2). RAG-CM uses GenZ [11] for model-inference operators; the following subsections present its retrieval models, grouped into Clustering Indices (§6.1.1) and Graph Indices (§6.1.2).

PQ/flat scan: data-aware imbalance. The scan’s cost is driven by 𝑁 scan , the number of database vectors a query touches—a vector count, not an op or byte total. Under a fixed PQ configuration, scoring each scanned vector requires reading its 𝑏 code -byte PQ code and summing one precomputed distance-table entry per subquantizer, so the scan’s ops are 𝑁 scan times the per-vector work and its bytes are 𝑁 scan𝑏 code ; this is how 𝑁 scan sets the scan operator’s profile. IVF makes 𝑁 scan far smaller than the corpus size 𝑁 : a query scans only the 𝑛 probe of 𝑛 list inverted lists nearest to it, a fraction 𝜌 = 𝑛 probe /𝑛 list of all vectors. The naive estimate 𝑁 scan = 𝜌𝑁 assumes equal-sized lists, but real embeddings cluster, so a query’s nearest lists are larger than average and it scans more; we correct with a cell-imbalance factor 𝑓 measured from the built index, so that 𝑁 scan = 𝑓 (𝑛 probe )𝜌 𝑁 , where 𝑓 is the ratio of the vectors a real

6.1.1 Clustering Indices. All three clustering indices share one search skeleton: a query is run through a coarse quantizer that selects the 𝑛 probe inverted lists nearest to it, the candidate vectors in those lists are scanned to compute distances, a top-𝑘 heap keeps the best candidates, and an optional re-rank recomputes exact distances on the survivors. They are analytical: given the configuration, the work in every stage is fixed in closed form. Table 2 lists every stage, its compute kernel, the peak it is bounded by, and which variants use it. For most stages, the corresponding Operator Work Profile follows directly 8

level; (3) the access-block size, which distinguishes sequential from scattered bandwidth; and (4) the parallel fraction for Amdahl scaling. The performance layer maps these fields to effective compute and memory rates (𝜋eff,𝛽 eff ). Data-dependent corrections modify 𝑊 and 𝑉 rather than the profile schema, allowing one performance model to serve all operators.

query actually scans—obtained from the index’s per-list sizes and the nearest-list assignments of a query sample—to the balanced 𝜌𝑁 . This 𝑓 is the only index-dependent input the model needs, and it is a pure property of the corpus and embedding—never of the hardware—while the remaining index knobs enter the cost model as analytical parameters. RAG-CM therefore caches 𝑓 once per (corpus, embedding)—only a handful of combinations across a whole design space—and from then on predicts the scan count for any configuration without ever building an index again.

6.3

The performance layer maps an Operator Work Profile to wall-clock time in two composable steps ( 3 , 4 in Fig. 7): a roofline bounds each phase by the slower of its compute and memory time, and Amdahl’s law composes the per-phase times across threads under the chosen parallel mode. The intermediate points below exist only to feed these two: the hardware model supplies the roofline’s two rates (𝜋eff , 𝛽 eff ), and the parallel model sets how the per-phase results combine into end-to-end latency and throughput.

6.1.2 Graph Indices. HNSW is ML-analytical. A query first descends greedily through the 𝐿 ≈ log𝑀 𝑁 upper layers—a few hops each over 𝑀-neighbor lists—to reach a good entry point, then runs a bounded best-first (beam) search of width ef 𝑠 at the base layer, which performs almost all of the work. Where IVF’s scan count follows in closed form from the configuration, the size of the base-layer neighborhood an HNSW query explores is data-dependent—it grows with the data’s intrinsic geometry and shrinks on clustered corpora where the beam converges fast—and has no closed form. The model therefore predicts the two quantities that drive the Operator Work Profile: the per-query distance computations 𝑛 dis and graph hops 𝑛 hops . These fold into the profile’s𝑊 and 𝑉 exactly as IVF’s 𝑁 scan does (§6.2)—𝑛 dis setting the dominant base-layer FLOPs and random-vector reads, and 𝑛 hops the visited-set and heap overhead. Concretely, the HNSW operator model estimates 𝑛ˆ dis =𝑐 𝑀 𝑔ˆdis (z) and 𝑛ˆ hops =𝑔ˆhops (z) with feature vector z = (𝑑,𝑀,ef 𝑠 ,LID): a learned predictor 𝑔ˆ emits the workload counts n̂ = (𝑛ˆ dis,𝑛ˆ hops ), and a factor 𝑐 𝑀 obtained through one-time per-corpus calibration fixes the scale of 𝑛ˆ dis . These counts determine the HNSW Operator Work Profile; the following paragraph defines 𝑔ˆ and its lightweight calibration.

Roofline kernel. For a phase with𝑊 and data volume𝑉 ,  operations  the model predicts 𝑇 =𝑇ovh +max 𝜋𝑊 , 𝛽𝑉 , where 𝜋eff and 𝛽 eff are eff eff the effective compute throughput and bandwidth, and 𝑇ovh captures fixed per-call costs. Hardware and execution factors. To derive these effective rates and compose phase times, RAG-CM accounts for (1) CPU topology, including heterogeneous cores and NUMA effects; (2) effective cache capacity under sharing among concurrent threads; (3) scattered– sequential memory access, which is sequential within a block but random across blocks; and (4) parallel execution, using separate compute and memory thread counts with Amdahl scaling.

6.4

Workload predictor. The predictor 𝑔ˆ is two independent gradientboosted regressors, 𝑔ˆdis and 𝑔ˆhops ; both are predicted because their ratio is not constant—it drifts with ef 𝑠 —and each contributes a different part of the Operator Work Profile. The lone data-dependent feature is the corpus local intrinsic dimensionality LID, estimated by an Amsaleg MLE on a query sample; it carries the geometry that lets a single model generalize across corpora—from high-LID synthetic training data to the low-LID, tightly clustered distributions of real image and text embeddings, on which a configuration-only estimate over-counts because the beam terminates sooner than the ambient 𝑑 suggests. Crucially, every feature in z is either a searchspace knob or a property of the corpus itself: the predictor uses no statistic of a constructed graph (degree, layer counts), so it scores candidate configurations whose indexes have not been built—exactly the regime the optimizer explores. The predictor requires only a onetime calibration for each corpus, which is reused across all unbuilt configurations and therefore adds little overhead.

Communication Modeling

Given a deployment, RAG-CM’s communication model estimates the time to move every inter-stage and intra-LLM payload across its selected devices ( 5 in Fig. 7). It derives these costs from the concrete topology: every GPU pair carries its own bandwidth and startup latency (𝛽𝑖 𝑗 , ℓ𝑖 𝑗 ), and GPU–CPU hops use a host-transfer class— covering tensor-parallel (TP) and pipeline-parallel (PP) collectives, data-parallel (DP) replicas, GPU–GPU stage handoffs such as the prefill→decode KV-cache transfer under 1P/1D disaggregation, and CPU-side boundaries. A cross-device edge𝑒 sums its fastest endpointdisjoint pair bandwidths into 𝛽𝑒 and takes its slowest startup as ℓ𝑒 ; LLM-internal TP/PP traffic folds into the prefill/decode service time rather than adding assembly edges; and each explicit crossdevice edge is priced as ℓ𝑒 + 𝑀𝑒 /𝛽𝑒 over its payload 𝑀𝑒 , with text and dataframe boundaries (e.g., retrieval→reranker) priced as host transfers.

6.5 6.2

Performance Modeling

Operator Work Profile

RAG-CM Assembly

The assembly layer ( 6 in Fig. 7) has two functions: it enumerates and materializes each s ∈ Xcm as one deployment and execution plan, then composes stage-level operator and communication predictions into steady-state, end-to-end serving performance. For each candidate it prices, assembly predicts the saturation throughput 𝑇˜ (s) and ˜ the corresponding mean latency 𝐿(s) once serving reaches steady state; startup behavior is outside the model. RAG-CM uses the performance objective and SLOs in the user’s optimization target to select

Each operator emits a hardware-agnostic Operator Work Profile, the interface between the algorithm and performance layers. Computed once per operator phase, it contains work (𝑊 ,𝑉 )—the operation count and bytes moved after cache reuse—and four execution characteristics: (1) the compute mode (e.g. blas/simd), which selects compute efficiency; (2) the bytes location, determined by the workingset size and unique byte volume 𝑉uniq ≤𝑉 , which selects the memory 9

among these configurations and exposes the selected performance to RAG-PE as 𝑃˜ (x𝑖 ).

uses additional datasets, which we describe in §7.4. The quality objective 𝑄 is RAGAS answer correctness [20], which scores factual agreement with the gold answer rather than lexical overlap—so valid paraphrases are not penalized—in [0,1], averaged over queries. As per-stage diagnostics, RAG-PE reads three RAGAS sub-metrics (§4.2): context recall, context precision, and faithfulness. All LLM-as-judge scoring uses DeepSeek-V4-Flash. Algorithm Design Spaces. Our algorithm design space follows Fig. 5; here, we specify only the choices not detailed in the figure, while all other components and parameter ranges remain as shown. For RAGEval, the chunk size is chosen from {128,256, 512,1024,2048}; the embedding model is BGE-small [2] (384-d), allmpnet-base-v2 [63] (768-d), or BGE-M3 [15] (1024-d); the reranker is TART [5], ColBERT [36], SentenceTransformer [58], or FlagEmbedding [2, 15]; and the query rewriter and main LLM independently use a Qwen2.5 [8] model from {1.5,3,7,14}B. For MS MARCO, we fix the chunk size at 2048 and the embedding model to all-mpnet-base-v2 to keep indexing its 8.84M passages tractable. System Design Spaces. We instantiate the system axes of Table 1. The end-to-end and optimizer-ablation experiments (§7.2, §7.3) share the four-H100 SysA space: RAG-CM maximizes throughput over collocated and disaggregated GPU-stage placements, all feasible stageto-GPU mappings and LLM parallelism plans (at most four GPUs per stage), request and decode batch sizes in {1,2,4,...,256} with 𝐵 decode ≥ 𝐵 request , and dynamic-batching waits in {2,10,50}ms; FAISS threads follow min(retrieval batch, physical cores) under inter-query parallelism. For transfer to the eight-A100 SysB (§7.5), only the GPU budget and per-stage cap grow from four to eight. Baseline system space. The baselines in §7.2 have no performance model, so a fair comparison lets them search the algorithm design space jointly with the system axes as one entangled space. But every candidate they propose must be deployed and measured, and the full system design space would spend most of this budget on poor system variants. We therefore compress all system axes that RAG-CM searches into two categorical choices, expert-curated to be strong on SysA: eight deployment presets (collocated TP/PP plans and 1P/1D–2P/2D prefill/decode disaggregation, with non-main-LLM stages either sharing GPUs with the main LLM or using GPUs left unused by it) crossed with six 𝐵 request ×𝐵 decode batch presets.

6.5.1 Steady-state performance under saturated closed-loop serving. We model a saturated closed-loop deployment with a fixed number of concurrent clients, each issuing its next query after the previous response. The resulting 𝑇˜ is the maximum sustainable throughput, and 𝐿˜ is the steady-state response latency at that operating point. This target characterizes the deployment’s attainable capacity without introducing an external arrival rate. An open-system latency instead depends on the offered load and can make the same deployment appear lightly loaded, near saturation, or unstable. 6.5.2 Trace-driven assembly. For each algorithm–system pair (x𝑖 ,s), RAG-CM instantiates a closed discrete-event model from RAG-IR’s workload representation and the operator and communication models (§6.4). The simulation captures four mechanisms: (1) Continuous and dynamic batching: LLM stages use continuous batching, while other stages dispatch batches by size or timeout; (2) Automatic prefix caching: repeated calls prefill only the uncached prompt suffix within KV-cache capacity; (3) Decoupled batch limits: generator decode and other stages use independent batch-size limits [30]; and (4) Shared-resource contention: collocated stages time-share 𝐿(s, 𝑁 ) denote GPUs and CPUs. At concurrency 𝑁 , 𝑇b(s, 𝑁 ) and b steady-state throughput (QPS) and mean end-to-end latency, respectively. To scale to thousands of candidates, RAG-CM uses a cheap analytical model to shortlist the best batch settings per deployment topology, then simulates only those candidates to throughput saturation and reports only simulation results.

7

Evaluation

Our evaluation is organized around four questions. RQ1 (End-to-end value). Across random seeds and under the same evaluation budget, does RAG-Stack consistently find better quality–performance Pareto frontiers than existing optimization methods? RQ2 (Optimizer ablation). With the rest of RAG-Stack held fixed, does the optimizer inside RAG-PE outperform existing alternatives? RQ3 (Cost-model accuracy). How accurate is RAG-CM, in absolute error and, more importantly, in ranking candidate deployments? RQ4 (System transfer). After migrating to new hardware, how much does RAG-Stack improve normalized hypervolume over reoptimizing from scratch under the same evaluation budget?

7.1

7.2

End-to-End Optimization

Figure 8 compares the measured quality–throughput Pareto frontiers under the protocol in §7.1. Following §2.3, we compute hypervolume from linearly normalized quality and raw throughput per dataset; throughput is log-scaled only in the plots for readability. We then compute normalized hypervolume independently for each seed and report the three-seed mean. Against GP+LogNEHVI [17], the strongest baseline on both datasets, RAG-Stack averages 0.514 versus 0.364 on RAGEval and 0.635 versus 0.259 on MS MARCO; the corresponding per-seed relative improvements average 52.5% and 153.2%, respectively. RAG-Stack’s per-seed hypervolume exceeds the strongest baseline’s in all six seed–dataset pairs. As a secondary benefit, RAG-Stack completes the end-to-end search in 4.59 hours on RAGEval and 3.60 hours on MS MARCO, faster than all baselines except Greedy-Forward (Table 3). This efficiency holds even though RAG-CM evaluates 570–5,400 system

Experimental Setup

Hardware. We use two NUMA servers as measured deployment targets with different CPU and GPU resources: SysA has two AMD EPYC 9124 sockets, 32 CPU cores in total (16 cores per socket, 3.0 GHz base and up to 3.7 GHz boost), two NUMA nodes, and four NVIDIA H100 GPUs; SysB has two AMD EPYC 7742 sockets, 128 CPU cores in total (64 cores per socket, 2.25 GHz base and up to 3.4 GHz boost), two NUMA nodes, and eight NVIDIA A100 80GB GPUs. Datasets and Quality Metrics. For the end-to-end evaluation, we use 100 queries each from RAGEval [74] to avoid contamination from LLM training data [56, 67], and MS MARCO [9] to evaluate on a larger, more diverse dataset. The RAG-CM accuracy evaluation 10

+

1

-

+

RAG-PE (ours) SMAC

1 + 0

2

5

Normalized HV

.

+

6

-

4

( .1

3

7



Greedy-Lookback Greedy-RAG-PE

0.6

0.30

0.4 0.15 0.00

  

0.2

RAGEval 0

10

20

30

40

0.0 50 0

MSMARCO 10

20

30

40

50

Evaluation budget

Figure 9: Optimizer ablation: dominated hypervolume versus evaluation budget, holding the rest of RAG-Stack fixed and varying only the optimizer inside RAG-PE. Lines are means over 3 seeds; bands show seed min/max.

+)

Figure 8: End-to-end quality–performance Pareto frontiers (log-scaled performance). Translucent markers show per-seed frontiers; each line pools seeds 43–45. Selected RAG-Stack configurations are re-measured on SysA. Table 3: Mean end-to-end cold-start runtime on SysA. Optimizer

GP+LogNEHVI Greedy-Forward

RAGEval (hours)

MS MARCO (hours)

Greedy-Forward [3] GP+LogNEHVI [17] SMAC [48]

2.25±0.23 5.60±0.40 8.03±0.42

3.16±0.06 5.68±0.51 7.03±0.37

RAG-Stack (ours)

4.59±0.51

3.60±1.79

each fixed 2048-character passage forms one chunk—then parallelizes generation with TP2×PP2 and encoding with TP4. 7 instead combines IVF-PQ-FS with ReAct, whose retrieval errors the small generator cannot correct. Across the overlapping quality range in Figure 8, serving at equal quality is 1.5–7.5× cheaper on MS MARCO than on RAGEval, likely due to public-benchmark leakage into LLM training data [56, 67]. Public benchmarks may therefore understate the serving cost of quality, motivating RAGEval as a fresh-data control.

configurations per candidate, because it prices them with the model rather than deploying each one. To explain these trade-offs, we inspect representative RAG configurations in Figure 8 and attribute their gains to algorithm and system choices. On RAGEval (left), RAG-Stack point 2 dominates the baselinebest point 1 with higher quality at 81× the throughput. Both use a 14B generator, but 1 adds 3B multi-query expansion to HNSW (ef 𝑠 =128, top-𝑘=16) without reranking and uses a fixed 1P/2D plan (one prefill and two decode replicas, batch 16×32, with 14B prefill on one H100). 2 disables expansion and uses 2048-character chunks with ColBERT after top-𝑘=8 IVF-PQ to rank the retrieved documents by relevance, raising answer correctness from 0.594 to 0.638, while RAG-CM selects a faster serving plan. GP+LogNEHVI is seed-dependent: only a few of its points rise above RAG-Stack’s frontier, all of them from seed 43 and confined to narrow throughput ranges, the widest at 3 (245.6 vs. 204.1 QPS; 0.335 vs. 0.326 quality). RAG-Stack still has higher seed-43 hypervolume, and SMAC and Greedy-Forward never win on both axes. 3 replaces RAG-Stack’s top-𝑘=1 plus ColBERT with reranker-free top-𝑘=4 retrieval over shorter chunks, then adds 1P/2D disaggregation and a larger batch. Once seed 43 finds this region, LogNEHVI concentrates later evaluations nearby; the other seeds never enter it—stochastic discovery followed by exploitation, not a stable baseline advantage. On MS MARCO (right), RAG-Stack point 5 dominates the qualitybest baseline point 4 with globally best quality at 8.6× the throughput: 4 relies on top-𝑘=64, whereas 5 uses top-𝑘=16 and a RAGCM-selected disaggregated layout. At the throughput extreme, 6 dominates the baselines’ fastest point 7 with 1.65× the throughput and 2.2× the quality. Both use a 1.5B generator; 6 retains accurate, uncompressed HNSW and cuts only to top-𝑘=1—sufficient because

7.3

Optimizer Ablation

To isolate the contribution of RAG-PE’s optimizer, we keep every other part of RAG-Stack unchanged and replace only this optimizer with each baseline in turn. We compare RAG-PE with SMAC [48], GP+LogNEHVI [10], Greedy-Forward, and Greedy-Lookback [3]; the last alternates backward and forward passes after its initial forward pass. Greedy-RAG-PE is a hybrid that switches from Greedy-Forward to RAG-PE after 30 of 50 evaluations. Each method runs with seeds 43– 45, and Figure 9 reports mean normalized hypervolume by budget. RAG-PE is the most consistent optimizer across both datasets: against the four standalone baselines, it attains the highest mean hypervolume at 77 of 80 post-warm-start budgets. At budget 50, it reaches 0.375 versus Greedy-Forward’s 0.327 on RAGEval and 0.681 versus Greedy-Lookback’s 0.599 on MS MARCO, relative gains of 14.8% and 13.8%. No single baseline is consistently strongest: the strongest method differs between the two datasets in this ablation, while the end-to-end comparison identifies a different strongest baseline (§7.2). In contrast, RAG-PE remains strong across settings and contributes independently to RAG-Stack’s end-to-end advantage.. Because dominated hypervolume also credits configurations in regions no deployment would use (extremely low quality or throughput), Figure 10 re-reads the same archives from a deployment perspective: the best quality each optimizer can serve once a minimum throughput is required. Figure 11 shows that initialization is critical: the Sobol-plus-agent warm start accounts for the largest share of the final hypervolume. The remaining channels also make non-trivial contributions to the final hypervolume. 11

GP+LogNEHVI Greedy-Forward

Best quality

0.6

Greedy-Lookback Greedy-RAG-PE

Quality (ans. corr.)

RAG-PE (ours) SMAC

0.40 0.35

0.4

0.30 0.2

RAGEval 100

MSMARCO

0.25

101

101

102

102

Figure 10: Best attainable quality versus throughput requirement: per optimizer, the highest-quality configuration meeting the requirement within the budget (lines: mean over 3 seeds; bands: seed min/max). Forced

Init (Sobol + agent)

17.9

Pareto-tension crossover

19.2

Unguided (Sobol + DLS)

18.1 7.8

19.9

DLS(agent) 0

10

Init

18.8

16.6

Agent proposal

20

7.6 0

% of evaluations

10

20

30

7.5

Figure 11: Per-channel contributions to RAG-PE’s results in §7.3: shares of all quality evaluations (left) and final hypervolume (right).

IVF-Flat (Latency) IVF-PQ (Latency) IVF-PQ-FS (Latency) HNSW (Latency)

𝑛 Err. ↓

𝜌𝑠 ↑

SysB 𝑟 ↑ Err. ↓

137 6.7% 0.998 1.000 137 11.3% 0.996 0.997 137 17.0% 0.989 0.997 1080 17.0% 0.986 0.971

11.4% 13.2% 12.8% 20.1%

𝜌𝑠 ↑

𝑟↑

0.993 0.996 0.994 0.977

0.997 0.998 0.998 0.957

8

Sequential RAG (Latency) 201 11.9% 0.973 0.955 13.4% 0.965 0.950 Sequential RAG (QPS) 201 11.0% 0.975 0.969 14.6% 0.965 0.950 Agentic RAG (Latency) 30 8.1% 0.966 0.987 13.8% 0.984 0.979 Agentic RAG (QPS) 30 12.2% 0.969 0.974 20.3% 0.967 0.980 RAG Overall (Latency) 231 11.4% 0.978 0.958 13.5% 0.970 0.952 RAG Overall (QPS) 231 11.2% 0.978 0.952 15.3% 0.968 0.952

7.4

MSMARCO 5

10

20

50

Performance (QPS)

100

200

System Transfer

We evaluate system transfer by migrating optimized RAG configurations from SysA to SysB. RAG-Stack re-scores the existing quality archive with RAG-CM instantiated for SysB, spends 20 additional evaluations refining the frontier, and re-measures the selected configurations on SysB. The uncalibrated variant uses only SysB’s hardware description, whereas the calibrated variant additionally calibrates RAG-CM on the target machine. Figure 12 compares both variants with from-scratch optimization on SysB under 20- and 50-evaluation budgets. With the same 20-evaluation budget, uncalibrated and calibrated transfer improve normalized HV over from-scratch optimization by 108.4% (0.331 vs. 0.159) and 182.2% (0.448 vs. 0.159), respectively.

Table 4: RAG-CM prediction accuracy. Error is MAPE; 𝜌𝑠 and 𝑟 denote Spearman and Pearson correlation; 𝑛 is the number of configurations.

Component

0.4

40

% of final HV

SysA

From scratch (20 evals)

0.6

varying models, indexes, batching, parallelism, colocation, disaggregation, and dynamic batching; each deployment is compared against measured saturated closed-loop QPS and mean end-to-end latency. For guiding system space search, preserving the relative order of configurations matters more than minimizing absolute prediction error. Across both systems, calibrated RAG-CM attains Spearman correlations of 0.968–0.978 for latency and QPS, while keeping overall MAPE at 11.2–15.3%. Even for random-access-heavy IVF-PQ-FS and HNSW, where MAPE rises to 20.1%, Spearman correlation remains at least 0.977. Thus, the residual prediction error largely preserves configuration ordering, providing the ranking signal that RAG-PE needs.

47.8

23.8

From scratch (50 evals)

Transfer (uncalib., 20 evals)

Figure 12: System transfer from SysA to SysB. The transferred Pareto configurations are re-measured end-to-end on SysB; all points shown are measured.

Throughput requirement (QPS)

Pool win

Transfer (calib., 20 evals)

Limitations and Discussion

Our current support for agentic RAG is trace-driven: RAG-IR must execute the pipeline to observe its runtime control flow before RAG-CM can predict its performance. Consequently, RAG-PE cannot densely sample the performance objective before quality evaluation—which would let a multi-task Gaussian process (MTGP) exploit abundant performance-only observations to improve sample efficiency—or filter SLO-violating configurations before they enter the candidate pool. We have implemented trace-free static RAG-IR and RAG-CM paths for sequential RAG, whose control flow is known from the configuration alone; extending such pre-execution performance prediction to agentic RAG remains future work.

RAG-CM Accuracy

Table 4 evaluates calibrated RAG-CM predictions on SysA and SysB at the operator and RAG pipeline levels. The operator study spans four vector-search corpora (SIFT1M [29], GloVe, GIST1M [29], and Deep1M [68]) and two embedded RAG corpora (ELI5 [21] and TriviaQA [34]), varying batch size and CPU parallelism across HNSW and IVF variants (full grids in the artifact). The RAG pipeline study crosses 77 serving configurations with RAGEval [74], ELI5, and TriviaQA, covering sequential and ReAct-style agentic workflows and

9

Conclusion

We present RAG-Stack, a system for efficiently discovering quality– performance Pareto frontiers across RAG algorithms and serving 12

systems. RAG-Stack combines RAG-PE for sub-metric-aware multiobjective exploration, RAG-IR for representing executed sequential and agentic workflows, and RAG-CM for predicting serving performance and searching system configurations without deploying every candidate. Experiments show that RAG-Stack improves normalized hypervolume over the strongest end-to-end baseline, with three-seed mean gains of 52.5% on RAGEval and 153.2% on MS MARCO.

Text Embeddings Through Self-Knowledge Distillation. In Findings of the Association for Computational Linguistics: ACL 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 2318–2335. https://doi.org/10.18653/v1/2024.findings-acl.137 [16] Jian Chen, Peilin Zhou, Yining Hua, Loh Xin, Kehui Chen, Ziyuan Li, Bing Zhu, and Junwei Liang. 2024. FinTextQA: A Dataset for Long-form Financial Question Answering. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 6025–6047. https://doi.org/10.18653/v1/2024.acl-long.328 [17] Samuel Daulton, Maximilian Balandat, and Eytan Bakshy. 2021. Parallel Bayesian Optimization of Multiple Noisy Objectives with Expected Hypervolume Improvement. https://doi.org/10.48550/arXiv.2105.08195 arXiv:2105.08195 [cs.LG] [18] K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan. 2002. A Fast and Elitist Multiobjective Genetic Algorithm: NSGA-II. IEEE Transactions on Evolutionary Computation 6, 2 (April 2002), 182–197. https://doi.org/10.1109/4235.996017 [19] Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. 2025. The Faiss Library. https://doi.org/10.48550/arXiv.2401.08281 arXiv:2401.08281 [cs] [20] Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. 2024. RAGAs: Automated Evaluation of Retrieval Augmented Generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, Nikolaos Aletras and Orphee De Clercq (Eds.). Association for Computational Linguistics, St. Julians, Malta, 150–158. https://doi.org/10.18653/v1/2024.eacl-demo.16 [21] Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. ELI5: Long Form Question Answering. https://doi.org/10.48550/arXiv.1907.09190 arXiv:1907.09190 [cs.CL] [22] Jia Fu, Xiaoting Qin, Fangkai Yang, Lu Wang, Jue Zhang, Qingwei Lin, Yubo Chen, Dongmei Zhang, Saravan Rajmohan, and Qi Zhang. 2024. AutoRAG-HP: Automatic Online Hyper-Parameter Tuning for Retrieval-Augmented Generation. https://doi.org/10.48550/arXiv.2406.19251 arXiv:2406.19251 [cs.CL] Pre[23] Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2022. cise Zero-Shot Dense Retrieval without Relevance Labels. https: //doi.org/10.48550/arXiv.2212.10496 arXiv:2212.10496 [cs.IR] [24] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2024. Retrieval-Augmented Generation for Large Language Models: A Survey. https://doi.org/10.48550/arXiv.2312.10997 arXiv:2312.10997 [cs.CL] The CMA Evolution Strategy: A Tutorial. [25] Nikolaus Hansen. 2023. https://doi.org/10.48550/arXiv.1604.00772 arXiv:1604.00772 [cs.LG] [26] Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. 2018. PipeDream: Fast and Efficient Pipeline Parallel DNN Training. https://doi.org/10.48550/arXiv.1806.03377 arXiv:1806.03377 [cs.DC] [27] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. 2019. GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism. https://doi.org/10.48550/arXiv.1811.06965 arXiv:1811.06965 [cs.CV] [28] Pooyan Jamshidi, Miguel Velez, Christian Kästner, and Norbert Siegmund. 2018. Learning to Sample: Exploiting Similarities across Environments to Learn Performance Models for Configurable Systems. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2018). Association for Computing Machinery, New York, NY, USA, 71–82. https://doi.org/10.1145/3236024.3236074 [29] Herve Jégou, Matthijs Douze, and Cordelia Schmid. 2011. Product Quantization for Nearest Neighbor Search. IEEE Transactions on Pattern Analysis and Machine Intelligence 33, 1 (Jan. 2011), 117–128. https://doi.org/10.1109/TPAMI.2010.57 [30] Wenqi Jiang, Suvinay Subramanian, Cat Graves, Gustavo Alonso, Amir Yazdanbakhsh, and Vidushi Dadu. 2025. RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Serving. https://doi.org/10.48550/arXiv.2503.14649 arXiv:2503.14649 [cs] [31] Wenqi Jiang, Marco Zeller, Roger Waleffe, Torsten Hoefler, and Gustavo Alonso. 2025. Chameleon: A Heterogeneous and Disaggregated Accelerator System for Retrieval-Augmented Language Models. https://doi.org/10.48550/arXiv.2310.09949 arXiv:2310.09949 [cs] [32] Wenqi Jiang, Shuai Zhang, Boran Han, Jie Wang, Bernie Wang, and Tim Kraska. 2025. PipeRAG: Fast Retrieval-Augmented Generation via Adaptive Pipeline Parallelism. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.1 (KDD ’25). Association for Computing Machinery, New York, NY, USA, 589–600. https://doi.org/10.1145/3690624.3709194 [33] Jiajie Jin, Yutao Zhu, Guanting Dong, Yuyao Zhang, Xinyu Yang, Chenghao Zhang, Tong Zhao, Zhao Yang, Zhicheng Dou, and Ji-Rong Wen. 2025. FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Research. In Companion Proceedings of the ACM on Web Conference 2025. 737–740. https://doi.org/10.1145/3701716.3715313. arXiv:2405.13576 [cs.CL] [34] Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension.

Acknowledgments We sincerely thank Yongjun He and Gustavo Alonso for their insightful discussions and invaluable suggestions throughout the development of this work.

References [1] 2026. Algorithmicsuperintelligence/Openevolve. Algorithmic SuperIntelligence Labs. [2] 2026. FlagOpen/FlagEmbedding. FlagOpen. [3] 2026. Marker-Inc-Korea/AutoRAG. Markr.AI. [4] Sebastian Ament, Samuel Daulton, David Eriksson, Maximilian Balandat, and Eytan Bakshy. 2025. Unexpected Improvements to Expected Improvement for Bayesian Optimization. https://doi.org/10.48550/arXiv.2310.20708 arXiv:2310.20708 [cs] [5] Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannaneh Hajishirzi, and Wen-tau Yih. 2023. Task-Aware Retrieval with Instructions. In Findings of the Association for Computational Linguistics: ACL 2023, Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, Toronto, Canada, 3650–3675. https://doi.org/10.18653/v1/2023.findings-acl.225 [6] Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. https://doi.org/10.48550/arXiv.2310.11511 arXiv:2310.11511 [cs] [7] Raul Astudillo and Peter Frazier. 2019. Bayesian Optimization of Composite Functions. In Proceedings of the 36th International Conference on Machine Learning. PMLR, 354–363. [8] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023. Qwen Technical Report. https://doi.org/10.48550/arXiv.2309.16609 arXiv:2309.16609 [cs.CL] [9] Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2018. MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. https://doi.org/10.48550/arXiv.1611.09268 arXiv:1611.09268 [cs.CL] [10] Maximilian Balandat, Brian Karrer, Daniel Jiang, Samuel Daulton, Ben Letham, Andrew G Wilson, and Eytan Bakshy. 2020. BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization. In Advances in Neural Information Processing Systems, Vol. 33. Curran Associates, Inc., 21524–21538. [11] Abhimanyu Bambhaniya, Ritik Raj, Geonhwa Jeong, Souvik Kundu, Sudarshan Srinivasan, Suvinay Subramanian, Midhilesh Elavazhagan, Madhu Kumar, and Tushar Krishna. 2025. Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM Models. https://doi.org/10.48550/arXiv.2406.01698 arXiv:2406.01698 [cs] [12] Matthew Barker, Andrew Bell, Evan Thomas, James Carr, Thomas Andrews, and Umang Bhatt. 2025. Faster, Cheaper, Better: Multi-Objective Hyperparameter Optimization for LLM and RAG Systems. https://doi.org/10.48550/arXiv.2502.18635 arXiv:2502.18635 [cs] [13] Maciej Besta, Lorenzo Paleari, Jia Hao Andrea Jiang, Robert Gerstenberger, You Wu, Jón Gunnar Hannesson, Patrick Iff, Ales Kubicek, Piotr Nyczyk, Diana Khimey, Nils Blach, Haiqiang Zhang, Tao Zhang, Peiran Ma, Grzegorz Kwaśniewski, Marcin Copik, Hubert Niewiadomski, and Torsten Hoefler. 2025. Affordable AI Assistants with Knowledge Graph of Thoughts. https://doi.org/10.48550/arXiv.2504.02670 arXiv:2504.02670 [cs] [14] Hans-Georg Beyer and Hans-Paul Schwefel. 2002. Evolution Strategies – A Comprehensive Introduction. Natural Computing 1, 1 (March 2002), 3–52. https://doi.org/10.1023/A:1015059928466 [15] Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity 13

https://doi.org/10.48550/arXiv.1705.03551 arXiv:1705.03551 [cs.CL] [35] Kirthevasan Kandasamy, Gautam Dasarathy, Jeff Schneider, and Barnabas Poczos. 2017. Multi-Fidelity Bayesian Optimisation with Continuous Approximations. https://doi.org/10.48550/arXiv.1703.06240 arXiv:1703.06240 [stat] ColBERT: Efficient and Ef[36] Omar Khattab and Matei Zaharia. 2020. fective Passage Search via Contextualized Late Interaction over BERT. https://doi.org/10.48550/arXiv.2004.12832 arXiv:2004.12832 [cs.IR] [37] Junkyum Kim and Divya Mahajan. 2026. VectorLiteRAG: LatencyAware and Fine-Grained Resource Partitioning for Efficient RAG. https://doi.org/10.48550/arXiv.2504.08930 arXiv:2504.08930 [cs] [38] J. Knowles. 2006. ParEGO: A Hybrid Algorithm with on-Line Landscape Approximation for Expensive Multiobjective Optimization Problems. IEEE Transactions on Evolutionary Computation 10, 1 (Feb. 2006), 50–66. https://doi.org/10.1109/TEVC.2005.851274 [39] Akshay Kudva, Wei-Ting Tang, and Joel A. Paulson. 2026. Multi-Objective Bayesian Optimization for Networked Black-Box Systems: A Path to Greener Profits and Smarter Designs. https://doi.org/10.48550/arXiv.2502.14121 arXiv:2502.14121 [stat] [40] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. https://doi.org/10.48550/arXiv.2309.06180 arXiv:2309.06180 [cs.LG] [41] Jiale Lao, Yibo Wang, Yufei Li, Jianping Wang, Yunjia Zhang, Zhiyuan Cheng, Wanghu Chen, Mingjie Tang, and Jianguo Wang. 2025. GPTuner: An LLMBased Database Tuning System. SIGMOD Rec. 54, 1 (April 2025), 101–110. https://doi.org/10.1145/3733620.3733641 [42] Julien-Charles Lévesque, Audrey Durand, Christian Gagné, and Robert Sabourin. 2017. Bayesian Optimization for Conditional Hyperparameter Spaces. In 2017 International Joint Conference on Neural Networks (IJCNN). 286–293. https://doi.org/10.1109/IJCNN.2017.7965867 [43] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems (NIPS ’20). Curran Associates Inc., Red Hook, NY, USA, 9459–9474. [44] Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, and Soumith Chintala. 2020. PyTorch Distributed: Experiences on Accelerating Data Parallel Training. https://doi.org/10.48550/arXiv.2006.15704 arXiv:2006.15704 [cs.DC] [45] Shaobo Li, Yirui Zhou, Yuan Xu, Kevin Chen, Daniel Waddington, Swaminathan Sundararaman, Hubertus Franke, and Jian Huang. 2026. RAGPerf: An End-to-End Benchmarking Framework for Retrieval-Augmented Generation Systems. https://doi.org/10.48550/arXiv.2603.10765 arXiv:2603.10765 [cs.PF] [46] Yangning Li, Weizhi Zhang, Yuyao Yang, Wei-Chieh Huang, Yaozu Wu, Junyu Luo, Yuanchen Bei, Henry Peng Zou, Xiao Luo, Yusheng Zhao, Chunkit Chan, Yankai Chen, Zhongfen Deng, Yinghui Li, Hai-Tao Zheng, Dongyuan Li, Renhe Jiang, Ming Zhang, Yangqiu Song, and Philip S. Yu. 2025. Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMs. https://doi.org/10.48550/arXiv.2507.09477 arXiv:2507.09477 [cs] [47] Ning Liang, Fabian Wenz, Jana Giceva, and Lisa Wu Wills. 2025. Athena: A Plug-and-Play Advisor for Retrieval-Augmented Generation Using VectorDB. In 2025 IEEE International Symposium on Workload Characterization (IISWC). 28–41. https://doi.org/10.1109/IISWC66894.2025.00013 [48] Marius Lindauer, Katharina Eggensperger, Matthias Feurer, André Biedenkapp, Difan Deng, Carolin Benjamins, Tim Ruhopf, René Sass, and Frank Hutter. 2022. SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization. https://doi.org/10.48550/arXiv.2109.09831 arXiv:2109.09831 [cs] [49] Tennison Liu, Nicolás Astorga, Nabeel Seedat, and Mihaela van der Schaar. 2024. Large Language Models to Enhance Bayesian Optimization. https://doi.org/10.48550/arXiv.2402.03921 arXiv:2402.03921 [cs] [50] Zhichao Lu, Gautam Sreekumar, Erik Goodman, Wolfgang Banzhaf, Kalyanmoy Deb, and Vishnu Naresh Boddeti. 2021. Neural Architecture Transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence 43, 9 (Sept. 2021), 2971–2989. https://doi.org/10.1109/TPAMI.2021.3052758 [51] Yu A. Malkov and D. A. Yashunin. 2018. Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs. https://doi.org/10.48550/arXiv.1603.09320 arXiv:1603.09320 [cs] [52] Alexander Novikov, Ngân Vũ, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco J. R. Ruiz, Abbas Mehrabian, M. Pawan Kumar, Abigail See, Swarat Chaudhuri, George Holland, Alex Davies, Sebastian Nowozin, Pushmeet Kohli, and Matej Balog. 2025. AlphaEvolve: A Coding Agent for Scientific and Algorithmic Discovery. https://doi.org/10.48550/arXiv.2506.13131 arXiv:2506.13131 [cs.AI] [53] Miles Olson, Elizabeth Santorella, Louis C. Tiao, Sait Cakmak, Mia Garrard, Samuel Daulton, Zhiyuan Jerry Lin, Sebastian Ament, Bernard Beckerman, Eric Onofrey, Paschal Igusti, Cristian Lara, Benjamin Letham, Cesar Cardoso, Shiyun Sunny Shen, Andy Chenyuan Lin, Matthew Grange, Elena Kashtelyan,

David Eriksson, Maximilian Balandat, and Eytan Bakshy. 2025. Ax: A Platform for Adaptive Experimentation. In Proceedings of the Fourth International Conference on Automated Machine Learning. PMLR, 21/1–25. [54] Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, and Dongmei Zhang. 2024. LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression. In Findings of the Association for Computational Linguistics: ACL 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 963–981. https://doi.org/10.18653/v1/2024.findings-acl.57 [55] Biswajit Paria, Kirthevasan Kandasamy, and Barnabás Póczos. 2019. A Flexible Framework for Multi-Objective Bayesian Optimization Using Random Scalarizations. https://doi.org/10.48550/arXiv.1805.12168 arXiv:1805.12168 [cs] [56] Zehan Qi, Rongwu Xu, Zhijiang Guo, Cunxiang Wang, Hao Zhang, and Wei Xu. 2025. Long$^2$RAG: Evaluating Long-Context & Long-Form Retrieval-Augmented Generation with Key Point Recall. https://doi.org/10.48550/arXiv.2410.23000 arXiv:2410.23000 [cs] [57] Siddhant Ray, Rui Pan, Zhuohan Gu, Kuntai Du, Shaoting Feng, Ganesh Ananthanarayanan, Ravi Netravali, and Junchen Jiang. 2025. METIS: Fast Quality-Aware RAG Systems with Configuration Adaptation. https://doi.org/10.48550/arXiv.2412.10543 arXiv:2412.10543 [cs.LG] [58] Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings Using Siamese BERT-Networks. https://doi.org/10.48550/arXiv.1908.10084 arXiv:1908.10084 [cs.CL] [59] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2020. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. https://doi.org/10.48550/arXiv.1909.08053 arXiv:1909.08053 [cs.CL] [60] Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval Augmentation Reduces Hallucination in Conversation. In Findings of the Association for Computational Linguistics: EMNLP 2021, Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, Punta Cana, Dominican Republic, 3784–3803. https://doi.org/10.18653/v1/2021.findings-emnlp.320 [61] Aditi Singh, Abul Ehtesham, Saket Kumar, Tala Talaei Khoei, and Athanasios V. Vasilakos. 2026. Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG. https://doi.org/10.48550/arXiv.2501.09136 arXiv:2501.09136 [cs] [62] Shamane Siriwardhana, Rivindu Weerasekera, Elliott Wen, Tharindu Kaluarachchi, Rajib Rana, and Suranga Nanayakkara. 2023. Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answering. Transactions of the Association for Computational Linguistics 11 (2023), 1–17. https://doi.org/10.1162/tacl_a_00530 [63] Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. MPNet: Masked and Permuted Pre-training for Language Understanding. https://doi.org/10.48550/arXiv.2004.09297 arXiv:2004.09297 [cs.CL] [64] Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, and Thang Luong. 2024. FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation. In Findings of the Association for Computational Linguistics: ACL 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 13697–13720. https://doi.org/10.18653/v1/2024.findings-acl.813 [65] Jianguo Wang, Xiaomeng Yi, Rentong Guo, Hai Jin, Peng Xu, Shengjun Li, Xiangyu Wang, Xiangzhou Guo, Chengming Li, Xiaohai Xu, Kun Yu, Yuxing Yuan, Yinghao Zou, Jiquan Long, Yudong Cai, Zhenxiang Li, Zhifeng Zhang, Yihua Mo, Jun Gu, Ruiyi Jiang, Yi Wei, and Charles Xie. 2021. Milvus: A Purpose-Built Vector Data Management System. In Proceedings of the 2021 International Conference on Management of Data (SIGMOD ’21). Association for Computing Machinery, New York, NY, USA, 2614–2627. https://doi.org/10.1145/3448016.3457550 [66] Shuhei Watanabe. 2026. Tree-Structured Parzen Estimator: Understanding Its Algorithm Components and Their Roles for Better Empirical Performance. https://doi.org/10.48550/arXiv.2304.11127 arXiv:2304.11127 [cs.LG] [67] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. 2025. WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex Instructions. https://doi.org/10.48550/arXiv.2304.12244 arXiv:2304.12244 [cs] [68] Artem Babenko Yandex and Victor Lempitsky. 2016. Efficient Indexing of BillionScale Datasets of Deep Descriptors. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2055–2063. https://doi.org/10.1109/CVPR.2016.226 [69] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. https://doi.org/10.48550/arXiv.2210.03629 arXiv:2210.03629 [cs.CL] [70] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A Distributed Serving System for Transformer-Based Generative Models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). 521–538. [71] Qingfu Zhang and Hui Li. 2007. MOEA/D: A Multiobjective Evolutionary Algorithm Based on Decomposition. IEEE Transactions on Evolutionary Computation 14

[73] Tao Zhang, Kaixian Qu, Zhibin Li, Jiajun Wu, Marco Hutter, Manling Li, and Fan Shi. 2026. Using Large Language Models for Embodied Planning Introduces Systematic Safety Risks. https://doi.org/10.48550/arXiv.2604.18463 arXiv:2604.18463 [cs.AI] [74] Kunlun Zhu, Yifan Luo, Dingling Xu, Yukun Yan, Zhenghao Liu, Shi Yu, Ruobing Wang, Shuo Wang, Yishan Li, Nan Zhang, Xu Han, Zhiyuan Liu, and Maosong Sun. 2025. RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework. https://doi.org/10.48550/arXiv.2408.01262 arXiv:2408.01262 [cs]

11, 6 (Dec. 2007), 712–731. https://doi.org/10.1109/TEVC.2007.892759 [72] Tianyang Zhang, Zhuoxuan Jiang, Shengguang Bai, Tianrui Zhang, Lin Lin, Yang Liu, and Jiawei Ren. 2024. RAG4ITOps: A Supervised Fine-Tunable and Comprehensive RAG Framework for IT Operations and Maintenance. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, Franck Dernoncourt, Daniel Preoţiuc-Pietro, and Anastasia Shimorina (Eds.). Association for Computational Linguistics, Miami, Florida, US, 738–754. https://doi.org/10.18653/v1/2024.emnlp-industry.56

15

Related documents

• ViBE: Co-Optimizing Workload Skew and Hardware Variability for MoE Serving#246529
• Prepared Meals to Prime and Perpetuate Reduced Dietary Sodium Intake in Hemodialysis#126000
• RAG-Anything: All-in-One RAG Framework#130733
• Strategic maintenance for manufacturing excellence#355658
• Prescribing Architecture : Designing Healthcare-Grade Systems with Java and FHIR / Abdelfattah Ragab#487305