Process Reward Agents for Steering Knowledge-Intensive Reasoning Jiwoong Sohn1∗ Tomasz Sternal1∗ Kenneth Styppa1,2∗ Torsten Hoefler1† Michael Moor1† 1
arXiv:2604.09482v1 [cs.AI] 10 Apr 2026
2
ETH Zürich, Switzerland Heidelberg University, Germany
Abstract
checked against axioms, syntactic rules, or compiler constraints, medical reasoning rarely admits rigorous verification. This limitation is consequential, as clinically correct decisions must be defensible throughout the entire reasoning trace, not only in the final answer. Establishing correctness often requires synthesizing multiple, layered sources of evidence, including primary scientific literature, clinical guidelines, and institution-specific protocols, within a landscape of knowledge that evolves continuously. Consequently, it becomes prohibitively expensive to repeatedly finetune each new LRM backbone to remain aligned with updated medical knowledge. Likewise, simply injecting retrieved documents into a thereby bloating policy context does not guarantee that the model will attend to the right evidence at the right time, nor does it provide a mechanism to detect and correct mistakes as they emerge. Prior work has explored the use of process reward models (PRM) (Yun et al., 2025; Jiang et al., 2025) in medical reasoning. Med-PRM trains a process reward model for post hoc evaluation of policy-generated reasoning traces, incorporating external medical evidence via retrieval. Meanwhile, Med-S3 jointly trains a policy model and a reward model through a self-evolving framework, but does not incorporate search. Importantly, both approaches rely on post hoc evaluation, as reward signals are applied only after a complete reasoning trajectory has been generated. This formulation limits intervention during reasoning, allowing errors to accumulate before any corrective signal is applied. Moreover, it precludes finegrained control over the generation process, restricting the model’s ability to explore alternative reasoning paths or prioritize evidence. Building on this view, we introduce a retrievalaugmented process reward framework in which a Process Reward Agent (PRA) interacts with a frozen reasoning model. At each reasoning step, the PRA
Reasoning in knowledge-intensive domains remains challenging as intermediate steps are often not locally verifiable: unlike math or code, evaluating step correctness may require synthesizing clues across large external knowledge sources. As a result, subtle errors can propagate through reasoning traces, potentially never to be detected. Prior work has proposed process reward models (PRMs), including retrieval-augmented variants, but these methods operate post hoc, scoring completed trajectories, which prevents their integration into dynamic inference procedures. Here, we introduce Process Reward Agents (PRA), a test-time method for providing domain-grounded, online, step-wise rewards to a frozen policy. In contrast to prior retrieval-augmented PRMs, PRA enables searchbased decoding to rank and prune candidate trajectories at every generation step. Experiments on multiple medical reasoning benchmarks demonstrate that PRA consistently outperforms strong baselines, achieving 80.8% accuracy on MedQA with Qwen3-4B, a new state of the art at the 4B scale. Importantly, PRA generalizes to unseen frozen policy models ranging from 0.5B to 8B parameters, improving their accuracy by up to 25.7% without any policy model updates. More broadly, PRA suggests a paradigm in which frozen reasoners are decoupled from domain-specific reward modules, allowing the deployment of new backbones in complex domains without retraining. All code and data are publicly available at https://process-reward-agents.github.io/.
1
Introduction
Despite the success of large reasoning models (LRM), the absence of mechanisms for validating intermediate reasoning steps remains a major challenge to reliable reasoning, particularly in high-stakes domains such as healthcare. In contrast to formal proofs or software programs, where each step can be mechanically ∗ Equal contribution † Co-last authors
1
observes the current reasoning trace, optionally deIn parallel, post-training has improved medicides whether to search for external medical evidence, cal reasoning through supervised fine-tuning (SFT) and assigns a local reward signal to guide generation and reinforcement learning from verifiable rewards in real time. This approach enables the evaluation of (RLVR) (Chen et al., 2024; Zhang et al., 2025a; Liu intermediate reasoning steps before errors propagate. et al., 2025a; Huang et al., 2025; Thapa et al., 2025). Our contributions are threefold: (i) we formulate Some systems also couple grounding and training by retrieval-grounded, step-wise evaluation as an online constructing reasoning traces from structured knowlcontrol problem for medical reasoning; (ii) we propose edge graphs (Wu et al., 2025). These approaches concentrate on improving medPRA, which decouples evidence search and verification from a frozen policy to guide generation in real time; ical reasoning either through post-training or by inand (iii) we demonstrate that PRA enables inference- jecting retrieved documents directly into the policy time branching and pruning strategies that generalize context (Lewis et al., 2020; Zakka et al., 2024). An alternative design, in which retrieval and evidence across tasks and backbone models. We evaluate PRA across multiple medical reason- selection are jointly integrated with step-wise verificaing benchmarks. Under a matched policy sampling tion by a separate online controller, remains underexbudget, PRA consistently outperform strong decod- plored. We address this gap by decoupling retrieval ing baselines. In particular, PRA achieves 80.8% from reasoning and assigning it to a process reward accuracy on MedQA with Qwen3-4B-Instruct, repre- agent that evaluates partial reasoning traces using senting state-of-the-art performance at the 4B scale. retrieved evidence. Overall, these results suggest that online, step-wise rewards provide a stable and transferable mechanism 2.2 Reward Models for improving medical reasoning. Additionally, we show that PRA generalizes to un- Reward modeling provides an interface for allocating seen, frozen policy models spanning 0.5B to 8B pa- additional compute at inference time. Unlike outcome rameters, improving MedQA accuracy by up to 25.7%. reward models, which score a trace solely based on its These gains expose underutilized reasoning capacity final answer, process reward models assign rewards to in smaller models, since generation requires neither intermediate reasoning steps (Lightman et al., 2023). policy retraining nor context editing. Under matched This step-level signal is particularly well suited for policy sampling budgets, PRA continues to improve tree search frameworks (Liu et al., 2025b). with inference-time scaling, whereas self-consistency Training PRMs typically requires step-level supervisaturates early. Ablations on reward granularity and sion. Early work relied on human-annotated reasoning timing indicate that these gains are driven by apply- traces, but the high cost and limited scalability of exing rewards at intermediate steps rather than only at pert annotation motivated automated alternatives. completion. We detail the PRA framework and its Subsequent approaches label intermediate steps usinference-time interaction in the following sections. ing Monte Carlo rollouts from partial states, treating the fraction of correct completions as a proxy for step correctness (Wang et al., 2023). However, because models can arrive at the correct answer even 2 Related Work when some intermediate steps are incorrect, such labels can be noisy (Zhang et al., 2025b). More recent 2.1 Medical Reasoning work explores alternative supervision signals, includReasoning models applied in the medical domain face ing LLM-as-a-judge annotations (Yang et al., 2025), several domain-specific challenges. Clinically correct hybrid pipelines that combine Monte Carlo-based sigdecisions must be grounded in both ever-expanding nals with judge-based labels (Zhang et al., 2025b), biomedical literature and contextual constraints such and retrieval-augmented judges that ground step evalas guidelines and common practice (Norman and Eva, uations in external evidence (Yun et al., 2025). 2010; Fisher and Wennberg, 2003; Lu, 2011). This has A critical challenge for process reward models is motivated retrieval-based methods that provide cu- generalization across policies. Applying a PRM offrated and up-to-date evidence at inference time (Zakka policy, that is, scoring reasoning traces generated by et al., 2024; Kim et al., 2025; Gao et al., 2026). Recent a policy different from the one used during training, work has also introduced carefully curated retrieval often degrades performance due to distributional miscorpora for targeted access, including MIRIAD (Zheng match, particularly in settings where PRMs are used et al., 2025), as well as structured medical knowledge to guide inference-time search (Liu et al., 2025b; Snell graphs such as MedGraphRAG (Wu et al., 2024). et al., 2024). In mathematical reasoning, retrieval 2
Figure 1: Overview of our approach. A Process Reward Agent (PRA) observes the reasoning trace generated by the frozen policy model (reasoner), decides when to search for external evidence, and assigns step-level rewards. This interaction can steer the policy at inference time, enabling more robust and controllable reasoning, particularly in knowledge-intensive domains like medicine. has been used to provide similar questions and steps as warm-up context for PRM judgment, improving generalization across models and problem types (Zhu et al., 2025). In medicine, however, existing retrieval-augmented process reward models typically retrieve evidence only after a complete reasoning trace has been generated and apply rewards post hoc (Yun et al., 2025). Consequently, online, retrieval-grounded step-wise evaluation that remains robust under policy shift remains unexplored. We address this gap by decoupling retrieval from the policy and performing search-based stepwise evaluation during generation, yielding a portable reward signal for online tree search across diverse medical reasoning policies.
3
Process Reward Agents
3.1
Problem Formulation
τ = (s1 , . . . , sK ). For notational simplicity, we index τ in a cumulative way, i.e., τt = (s1 , . . . , st ). Also, we define the last step sK of a completed reasoning trace to be the model’s final answer. In addition, assume access to a fixed knowledge base D containing domain-specific documents relevant to the questions. To make effective use of both the policy π and the documents in D, we aim to design a parameterized inference procedure Gϕ that takes as input a question, the fixed policy, and the knowledge base, and outputs a final answer: Gϕ : (q, π, D) 7→ ŷq .
(1)
The objective is to find parameters ϕ that maximize the expected correctness of the produced answer: h i max Eq∼P (Q), ŷq ∼Gϕ (q,π,D) C(ŷq , yq ) . (2) ϕ
3.2
Let q ∈ Q denote a question, and let yq ∈ Y denote its ground-truth answer, where Q and Y are the spaces of possible questions and answers, respectively. We assume answers are verifiable. Concretely, there exists a correctness function C, where C(ŷq , yq ) = 1 if ŷq correctly matches yq , and C(ŷq , yq ) = 0 otherwise. Let π be a reasoning model with frozen parameters that autoregressively generates reasoning steps. We refer to the resulting sequence as a reasoning trace
Process Reward Agents
We instantiate Gϕ as a process reward agent (PRA) that separates reasoning from evidence acquisition by delegating retrieval and evaluation to a dedicated model. The PRA consists of two components: an action controller µact ϕ and a reward scoring function rwd µϕ , both implemented as separate token-level readouts from a single model with shared parameters ϕ. The controller observes a partial reasoning trace and 3
selects an action: ât ∼ µact ϕ (τt ), where ât ∈ {search, reward}.
with its own beam of traces at potentially different reasoning depths. Rather than processing questions independently, we maintain a single global queue of all active traces. At each iteration, traces are partitioned by pending stage, namely π generation, ρ retrieval, or µϕ evaluation (readout), and each stage is executed as a single batched operation regardless of which question, beam, or reasoning step a trace belongs to. After completion, traces re-enter the queue with updated stage tags. This synchronized stage-level batching sustains high GPU utilization even as traces become desynchronized due to variable-length reasoning, early termination, and conditional retrieval. Pseudocode is provided in Appendix Figure B.
(3)
When ât = search, the most relevant documents Dt are retrieved from D; when ât = reward, we set Dt = ∅. Finally, the scoring function µrwd evaluates ϕ the most recent reasoning step st using the partial reasoning trace τt and the set of documents Dt . The scoring function then evaluates the most recent reasoning step st conditioned on the partial trace and the (possibly empty) evidence set: r̂t = µrwd ϕ (τt , Dt ).
(4)
The resulting step-wise rewards r̂t steer tree search at inference time, ranking and pruning candidate trajectories online during generation. This approach has three advantages: (1) the PRA can be trained or updated to reflect changes in the knowledge base D without modifying the frozen policy π, so that domain adaptation reduces to retraining a single reward module; (2) the policy is never conditioned on retrieved documents and receives no gradient signal from the PRA, different reasoning backbones can be substituted at deployment time with no retraining; and (3) conditional activation of retrieval through the controller introduces a new axis of inference-time scaling: search can be invoked selectively per step, trading off computational cost against reward signal quality within the same tree search budget.
4
Experiments
4.1
Experimental Setup
Datasets and Knowledge Base We use MedQA Jin et al. (2020) to construct the training dataset. For each question in the MedQA training split (10,178 questions), we generate a single reasoning trace using Qwen3-4B-Instruct as the frozen reasoning model, following the policy prompt shown in Appendix Figure 7. For every partial reasoning trace τtj , we retrieve a corresponding set of relevant documents. We evaluate in-distribution performance on the MedQA test split Jin et al. (2020), ensuring that all evaluation questions are held out from the training set. To assess generalization, we further evalu3.3 PRA-Guided Tree Search ate on several out-of-distribution datasets, including MedBullets Chen et al. (2025), MedMCQA Pal et al. We use beam search Boulanger-Lewandowski et al. (2022), MMLU-Med Hendrycks et al. (2021); Singhal (2012); Graves (2012) as an inference-time-scaling et al. (2023), GPQA Rein et al. (2023), and clinical method. A beam of width B maintains B partial case datasets from The Lancet and The New England j reasoning traces. Each trace τt is scored by its cumuJournal of Medicine Thapa et al. (2025). lative reward: Our knowledge base aggregates multiple medical t t corpora, including medical textbooks Singhal et al. X X (j) (j) (j) R(τt ) = rˆi (j) = µrwd (τ , D ) (5) (2023), StatPearls Xiong et al. (2024), clinical practice ϕ i i i=1 i=1 guidelines Chen et al. (2023), and a rare disease corpus Wang et al. (2024). At every step t, the frozen policy π extends each of the B traces with b candidate next steps (branching factor), producing B × b candidates. The PRA scores Retrieval Retrieval is performed using the Medevery candidate, and the top-B traces by cumulative CPT dense retriever and reranker Jin et al. (2023). reward R are retained; the rest are pruned. Gen- For each corpus, we retrieve 200 candidate documents, eration terminates when all traces in the beam are rerank the combined candidate documents and retain complete, and the trace with the highest cumulative the top 64 documents. The query used for retrieval reward yields the final answer. consists of the question q and the last two reasoning To enable efficient evaluation over an entire bench- steps in the partial reasoning trace. This retrieval mark, PRA-guided tree search coordinates three mod- configuration is fixed and used consistently across all els, the frozen policy π, process reward agent µϕ , and experiments, including training, inference, ablations, the retriever ρ, across many concurrent questions, each and baseline comparisons. 4
Policy
Qwen3-4B-Instruct
Method Direct Direct + SC CoT CoT + SC RAG RAG + SC PRA (Ours)
ID
OOD
Average
MedQA
Medbullets
MedMCQA
MMLU
GPQA
Lancet
NEJM
61.6 61.3 72.7 74.8 72.2 76.7 80.8
48.8 48.7 56.5 58.7 55.7 58.4 63.6
55.6 55.8 61.1 62.7 63.3 64.8 66.2
77.4 77.3 83.7 84.9 85.4 86.2 86.6
51.1 50.8 60.8 51.8 59.2 54.4 64.4
60.4 60.2 62.4 63.5 62.1 61.0 67.0
45.3 46.4 62.7 63.2 65.7 66.9 68.3
57.2 57.2 65.7 65.7 66.2 66.9 71.0
Table 1: Main results on medical reasoning benchmarks. PRA outperforms direct answering, chain-of-thought (CoT), and retrieval-augmented generation (RAG) baselines on the in-distribution MedQA benchmark and six out-of-distribution benchmarks, achieving the best average score overall. Using Qwen3-4B-Instruct as the base policy model, PRA improves over the strongest baseline, RAG + SC, by 4.1 points on average. Baselines We compare against standard reasoning and retrieval baselines. Direct prompting generates answers without explicit reasoning. CoT uses Chainof-Thought prompting to elicit step-by-step reasoning. RAG augments the input with the retrieved documents before generation. For each baseline, we also evaluate with Self-Consistency (SC), which samples multiple reasoning paths and selects the most frequent answer. For fair comparison, SC samples 64 traces, which match the compute budget of our PRA with beam search (B = 4, branching factor b = 16).
additional evidence changes the evaluator’s posterior belief. Since our goal is to identify unnecessary retrieval, we treat search as necessary only when conditioning on retrieved documents induces a sufficiently large posterior update. Let m denote the margin between the logprobabilities of the correct and incorrect reasoning labels when no documents are provided, and let md denote the corresponding margin when the teacher model is conditioned on retrieved documents:
Label Generation We obtain reasoning and search labels for every reasoning step using Qwen3-235BInstruct as a teacher model. Reasoning labels are generated by conditioning the teacher model on the partial reasoning trace produced by the reasoning model up to the evaluated step, together with the corresponding set of retrieved documents. For each step, we instruct the teacher model to classify the reasoning step as either correct(1) or incorrect(0) by emitting a single token (prompt in Appendix Figure 8). We use this binary output directly as the reasoning label. In addition, we extract the log-probabilities log p(0) and log p(1) assigned to the two reasoning labels, which are used for search label generation. To obtain search labels, we additionally instruct the teacher model on the same partial reasoning trace without providing any retrieved documents, while keeping the prompt structure otherwise identical. This yields a second set of log-probabilities for the same reasoning labels, enabling us to directly measure the impact of retrieval. We compute search labels using the log-probabilities obtained from these two evaluations. Intuitively, if retrieval does not affect the teacher’s assessment of the reasoning step(posterior), then invoking search is unnecessary. From a Bayesian perspective, the margin shift between two information sets measures how much the
We measure the influence of retrieval on the reasoning process by computing the margin shift:
m = log p(1) − log p(0).
∆m = m − md .
(6)
(7)
A large |∆m| indicates that the search substantially affected the assessment of the teacher, while a small |∆m| suggests that retrieved documents had little effect on the reasoning labels. To obtain the final binary labels, we label a step as requiring search if ( search, |∆m| > ϵglobal , at = (8) reward, |∆m| ≤ ϵglobal . where ϵglobal is set to the median of |∆m| across all training dataset, yielding 50% of reasoning steps labeled as requiring search. PRA Training We fine-tune Qwen3-4B-Instruct on the reasoning and search labels generated by the teacher model for training PRA. For each reasoning step, the model is trained to predict two binary outputs, the reasoning label and the search label. In the main experiments, we fix the search label to 1 for every step, corresponding to an always-search setting in which PRA retrieves evidence before evaluating each reasoning step, thereby ensuring maximal access to external evidence during online process-level reward 5
guidance. For further analysis on Search–Accuracy Trade-off (Figure 3), we instead use search labels derived from the margin-shift criterion described above. This allows PRA to learn when retrieval is necessary, enabling selective search at inference time. The detailed training hyperparameters and prompt templates are provided in Appendix Section C and Section D
generation, allowing it to steer reasoning toward more promising trajectories and recover from early errors. In contrast, Self-Consistency is constrained by the policy’s initial sampling distribution and can only aggregate over completed samples. MedQA 0.82
Reward Readout Let ℓ(1) ∈ R|V| denote the logit vector at the first output slot of PRA, used for predicting the reasoning reward. The step-wise reward r̂t = µrwd ϕ (τt , Dt ) is obtained by applying a two-way softmax to the logits of tokens 0 and 1 and taking the normalized score assigned to token 1. We interpret r̂t as the reward score for the correctness of the current reasoning step.
0.80
Accuracy
0.78 0.76 0.74 0.72
Self-Consistency
PRA Action Readout Let ℓ(2) ∈ R|V| denote the logit ±1 Standard Error 0.70 vector at the second output slot of PRA, used for pre1 2 4 8 16 32 64 Number of Samples (Traces) dicting the search action. The controller distribution µact ϕ (τt ) is obtained by applying a two-way softmax to the logits of tokens 0 and 1, where 1 corresponds to Figure 2: Performance on MedQA under inference search and 0 corresponds to reward. We then sample time scaling. PRA continues to benefit from additional compute, while Self-Consistency saturates quickly. For the action ât ∼ µact ϕ (τt ). SC, we estimate per-question expected accuracy via Monte Carlo sampling (1,000 trials); shaded regions 4.2 Results show ±1 SE computed via bootstrap resampling over We evaluate PRA against reasoning and retrieval questions. baselines across seven medical benchmarks (Table 1). PRA consistently outperforms all baselines on both in-distribution and out-of-distribution benchmarks. Generalization to Unseen Datasets PRA To the best of our knowledge, our framework is the demonstrates strong generalization to medical reasonfirst to enable a 4B-scale model to exceed 80% ac- ing benchmarks unseen during training as presented curacy on MedQA, establishing new state-of-the-art in Table 1. Across all six out-of-distribution benchperformance for models of this size. marks, PRA consistently outperforms baselines by an While Self-Consistency improves performance when average of 4.8 points. applied to Direct, CoT, and RAG baselines on most benchmarks, we observe performance degradation Generalization to Unseen Policy Models We with increased number of sampling on challenging evaluate whether PRA enables the drop-in replacebenchmarks such as GPQA and Lancet. On these ment of frozen policy models without requiring any benchmarks, the policy model frequently produces retraining. Table 2 shows that PRA, despite being incorrect or incomplete responses across repeated trained exclusively on reasoning traces from Qwen3samples, causing Self-Consistency to amplify errors 4B, generalizes well across a diverse set of policy modthrough majority voting. In contrast, PRA maintains els spanning both smaller and larger sizes. In several stable improvements even on difficult benchmarks by cases, models that perform poorly under standard deguiding generation toward valid completions. coding exhibit especially large relative improvements when paired with PRA, with the largest gains obInference Time Scaling Behavior Figure 2 com- served for smaller policy models. For example, on pares the performance of PRA and Self-Consistency Qwen2.5-0.5B-Instruct, PRA improves MedQA acas the sampling budget increases. Self-Consistency curacy from 28.4 to 54.1, corresponding to a 90.5% shows little improvement once the number of samples relative improvement over chain-of-thought. exceeds 8, whereas PRA continues to benefit from This strong transfer is particularly notable because additional compute. We attribute this difference to PRA does not modify the policy model itself. Instead, the fact that PRA applies step-wise rewards during it operates purely at inference time through step-wise
6
selection within beam search: at each step, the policy model autoregressively proposes candidate continuations, and PRA selects among them without altering the generation procedure, injecting additional context, or updating model parameters. As a result, all generated outputs remain within the policy model’s original output distribution. Unlike retrieval-augmented generation methods, which modify the model’s input context, PRA exerts control solely through inference-time guidance. These results suggest that substantial gains in reasoning performance can be achieved by more effectively exploiting the latent capabilities of existing models, and that the reasoning potential of smaller policy models is considerably stronger than their standalone decoding performance may indicate. Policy
Method
Acc.
∆
Llama-3.1-8B-Instruct
CoT + Self-Consistency + PRA
67.0 75.1 80.1
– +8.1 +13.1
Qwen3-4B-Instruct†
CoT + Self-Consistency + PRA
72.7 74.8 80.8
– +2.1 +8.1
Llama-3.2-3B-Instruct
CoT + Self-Consistency + PRA
56.0 66.2 75.4
– +10.2 +19.4
CoT + Self-Consistency + PRA
49.5 54.0 69.9
– +4.5 +20.4
augmentation), indicating that search enables more effective inference-time scaling of the policy’s reasoning. We further compare the trained PRA against its untrained backbone, Qwen3-4B-Instruct, when used as the reward agent within beam search. Despite employing a different inference structure, the untrained reward agent achieves performance comparable to Self-Consistency, and when combined with retrieval, matches the accuracy of Self-Consistency with RAG. This suggests that inference-time restructuring alone is insufficient to substantially improve performance beyond the policy’s native distribution, and that retrieval provides an orthogonal source of improvement. In contrast, combining search with a trained process reward agent yields a clear additional gain. PRA, which integrates reward agent training and retrieval, achieves the highest accuracy (80.8), demonstrating that training the reward agent is critical for effective inference-time scaling with beam search and retrieval of external evidence. Reward Agent
Trained?
Search?
Method
# Sample
Acc.
× × × × Qwen3-4B Qwen3-4B PRA (ours)
× × × × × × ✓
× × ✓ ✓ × ✓ ✓
CoT CoT + SC RAG RAG + SC PRA PRA PRA
1 64 1 64 64(B × b) 64(B × b) 64(B × b)
72.7 74.8 72.2 76.7 74.4 76.7 80.8
Table 3: Ablation study of PRA training and search on MedQA. All methods use the same frozen Qwen34B-Instruct policy model and differ only in whether CoT 36.2 – Llama-3.2-1B-Instruct + Self-Consistency 44.0 +7.8 a reward agent is used, whether it is trained, and + PRA 57.8 +21.6 whether search(retrieval) is enabled. Training the CoT 28.4 – reward agent accounts for the majority of the perQwen2.5-0.5B-Instruct + Self-Consistency 31.9 +3.5 formance gain, and combining it with search yields + PRA 54.1 +25.7 further improvements, achieving the highest accuracy with PRA. Table 2: Cross-model generalization on MedQA. PRA † trained with Qwen3-4B-Instruct generalizes effectively to both larger and smaller policy models, with larger gains observed for smaller models. All non- 5.2 Ablation on Inference daggered policies are unseen during PRA training. We further investigate whether the gains from PRA arise primarily from improved reward modeling or from how rewards are applied at inference time. To disentangle these factors, we fix the same 5 Analysis trained process reward agent (PRA) and vary only its inference-time usage along two axes: reward level and 5.1 Ablation on Training reward timing. Reward level specifies what is scored. Table 3 isolates the effects of reward agent, training, Outcome-level assigns a single reward to each comand search while keeping the policy model fixed to pleted reasoning trace, whereas process-level assigns Qwen3-4B-Instruct. Under single-sample decoding, rewards to intermediate reasoning steps. Reward timChain-of-Thought (CoT) and retrieval-augmented gen- ing specifies when rewards are computed and used. eration (RAG) achieve comparable accuracy. However, Post hoc computes rewards only after a full trace has increasing the number of samples yields larger gains been generated, while online computes step-wise rewhen inference is augmented with search (i.e., retrieval wards during generation (here, within beam search), Qwen2.5-3B-Instruct
7
suited to knowledge-intensive evaluation and can be viewed as a practical upper bound on accuracy.
CoT + SC PRA (Last) PRA (Min) PRA (Max) PRA (Average) PRA (Ours)
Reward Level
Reward Time
Search
Acc.
× Outcome Process Process Process Process
Post hoc Post hoc Post hoc Post hoc Post hoc Online
× ✓ ✓ ✓ ✓ ✓
74.8 75.7 74.3 77.5 77.6 80.8
Pareto
0
50
0.2 0.0
100
dep
0.4
Accuracy (%)
0.6
77 75
85
0.8
79
Accuracy Search Freq.
80
100 75 50 25
75 0.0
Search Frequency (%)
0.5
1.0
0
dep
Figure 3: Search–accuracy trade-off on MedQA. We sweep the search threshold and report accuracy versus search frequency; the Pareto frontier highlights the best operating points for a given search budget.
5.3 Method
(b) 1.0
81
Search Freq. (%)
(a) Accuracy (%)
allowing them to guide reasoning as it unfolds. All settings use identical sampled traces and, when applicable, the same search mechanism; search queries are formed from the accumulated reasoning steps available at the time the reward is computed. Table 4 shows that outcome-level PRA yields only modest improvements over Self-Consistency. For process-level, we aggregate step-wise rewards post hoc using different reduction operators (min, max, or average). This improves performance, but still underperforms settings in which rewards are applied online. Our full method, which applies step-wise rewards during generation, achieves the highest accuracy. Overall, these results indicate that the majority of the gain stems not only from stronger reward signals, but from enabling online, process-level control over the reasoning process itself.
Analysis on Margin Shift
We analyze how margin shift varies across reasoning traces on MedQA. Specifically, we compute ∆m, which quantifies how the inclusion of retrieved evidence changes the teacher model’s decisions between reasoning traces.
| m| by Position
Mean | m|
Table 4: Ablation of outcome-level and process-level inference-time usage of the same trained PRA on MedQA. All PRA variants share identical reward model parameters; only the timing and granularity of reward application differ.
Search–Accuracy Trade-off While retrieval at every reasoning step improves performance in knowledgeintensive settings, it can be costly. We therefore investigate whether PRA can learn to invoke search selectively, trading off retrieval cost against answer accuracy. To this end, we train PRA with binary search labels derived from the margin-shift criterion described in Section 4.1, enabling step-wise decisions about when retrieval is necessary. At inference time, PRA outputs a search score at each step and triggers retrieval only when this score exceeds a threshold θdep . We sweep θdep from 0 to 1 in increments of 0.1 to vary the frequency of search calls. Figure 3 shows a clear trade-off between search frequency and answer accuracy: reducing search generally lowers accuracy, although the Pareto frontier indicates that selective retrieval can achieve comparable, and sometimes slightly higher, accuracy with fewer search calls. This experiment provides further analysis of selective retrieval in PRA. In the main results (Table 1), we use an always-search configuration, reflecting the assumption that retrieval is broadly beneficial in knowledge-intensive reasoning. This setting is well
2.4 2.2 2.0 1.8 1.6 1.4 1.2 1.0
Correct Incorrect 0
50
Trajectory Position (%)
100
Figure 4: Margin shift across step positions in reasoning trajectories on MedQA, separated by traces with correct and incorrect final answers. Correct traces show larger margin shifts at later steps, whereas incorrect traces show the opposite pattern.
Trajectory Position and Answer Correctness. Figure 4 reports the average margin shift at different step positions within the reasoning trajectory, separated by traces with correct and incorrect final answers. We observe markedly different trends between the two groups. For traces that ultimately produce correct answers, margin shift increases toward later steps, indicating that retrieved evidence plays a larger role in the teacher model’s evaluation as reasoning progresses. In contrast, for incorrect traces, margin 8
Mean | m|
| m| by Difficulty
3.0 2.8 2.6 2.4 2.2 2.0 1.8 1.6 1.4
uates partial reasoning traces and steers generation without modifying the policy model, its parameters, or its input space. By routing retrieval and evidence usage to the reward agent rather than the policy, PRA enables fine-grained verification of intermediate steps while avoiding the sensitivity to retrieval noise and context length of standard retrieval-augmented generation. Across multiple medical reasoning benchmarks, PRA consistently outperforms strong reasoning and retrieval baselines, achieving state-of-the-art performance for 4B-scale models on MedQA and delivering robust gains on diverse out-of-distribution datasets. We further showed that PRA generalizes across unseen policy backbones, revealing substantial underutilized reasoning capacity in smaller models. Ablation studies indicate that these gains arise primarily from applying process-level rewards online during generation rather than from post hoc scoring alone. Finally, we characterized the trade-off between retrieval cost and accuracy under selective search, showing that PRA can adaptively reduce search while preserving performance along a Pareto frontier, positioning PRA as a practical and modular approach for reliable, evidencegrounded reasoning in knowledge-intensive domains without retraining the underlying reasoning model.
Correct Incorrect
0
50
Solve Rate (%)
100
Figure 5: Mean absolute magin shift over reasoning steps across questions of varying difficulty, where difficulty is defined by the fraction of policy-generated reasoning samples that reach the correct answer. Correct traces consistently exhibit larger margin shifts than incorrect traces, and margin shift for correct traces is highest on harder questions and gradually decreases as solve rate increases.
shift decreases at later steps, suggesting that flaws in the reasoning become more apparent to the teacher model even without additional evidence. Notably, at the final step, which typically corresponds to a concise answer selection or conclusion, retrieved evidence has little effect on margin shift, consistent with this step containing minimal substantive reasoning.
Impact Statement This work proposes process reward agents for steering knowledge-intensive reasoning, especially medical reasoning, with frozen reasoning models. The intended impact of this approach is to increase the reliability and verifiability of reasoning traces procuced by language models, with a focus on the high stakes application domain of healthcare, where individual steps need to meet a high bar to enable trust and appropriate reliance on AI systems. By explicitly rewarding the reasoning process, and grounding individual steps in latest guidelines and literature, this work has the potential to reduce unfounded reasoning traces, catch hallucinations, and overall increase the quality of generated reasoning traces by means of domain-grounded test-time scaling. Despite these promises, process reward agents may not fully eliminate the risk of hallucinations or remove all incorrect intermediate steps. Also, this work should be considered as a method contribution and not a ready-to-deploy system to support medical decision making. Ultimately, this work aims to make AI systems more reliable and better grounded in available external knowledge. Insofar, we hope that the presented methods and results will improve the safety of using AI in knowledge-intensive and high-stakes domains such as medicine.
Difficulty and Answer Correctness. Figure 5 reports margin shift across questions of varying difficulty, again separated by traces with correct and incorrect final answers. Question difficulty is defined by the fraction of reasoning samples from the policy model (Qwen3-4B-Instruct) that reach the correct answer. Consistent with the trajectory-position analysis, retrieved evidence induces larger margin shifts for correct traces, particularly on more difficult questions, while its effect remains substantially smaller for incorrect traces. One plausible interpretation could be that incorrect reasoning trajectories contain internal inconsistencies or errors that are detectable by the teacher model without strong reliance on external evidence.
6
Conclusion
We presented Process Reward Agents (PRA), a framework for guiding frozen reasoning models through knowledge-intensive reasoning tasks using online, stepwise, and domain-grounded process rewards. PRA reframes inference-time reasoning as a controllable search process in which a dedicated reward agent eval9
References
Shuyang Jiang, Yusheng Liao, Zhe Chen, Ya Zhang, Yanfeng Wang, and Yu Wang. Meds3 : Towards Nicolas Boulanger-Lewandowski, Yoshua Bengio, and medical slow thinking with self-evolved soft dualPascal Vincent. High-dimensional sequence transsided process supervision, 2025. URL https:// duction, 2012. URL https://arxiv.org/abs/ arxiv.org/abs/2501.12051. 1212.1936. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Hanjie Chen, Zhouxiang Fang, Yash Singla, and Mark Weng, Hanyi Fang, and Peter Szolovits. What Dredze. Benchmarking large language models on Disease does this Patient Have? A Large-scale answering and explaining challenging medical quesOpen Domain Question Answering Dataset from tions, 2025. URL https://arxiv.org/abs/2402. Medical Exams, September 2020. URL http: 18060. //arxiv.org/abs/2009.13081. arXiv:2009.13081 [cs]. Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, and Qiao Jin, Won Kim, Qingyu Chen, Donald C Comeau, Benyou Wang. Huatuogpt-o1, towards medical Lana Yeganova, W John Wilbur, and Zhiyong Lu. complex reasoning with llms. arXiv preprint Medcpt: Contrastive pre-trained transformers with arXiv:2412.18925, 2024. large-scale pubmed search logs for zero-shot biomedical information retrieval. Bioinformatics, 39(11): Zeming Chen, Alejandro Hernández Cano, Angebtad651, 2023. lika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, AnYubin Kim, Hyewon Jeong, Shan Chen, Shuyue Stella dreas Köpf, Amirkeivan Mohtashami, and others. Li, Chanwoo Park, Mingyu Lu, Kumail AlMeditron-70b: Scaling medical pretraining for large hamoud, Jimin Mun, Cristina Grau, Minseok Jung, language models. arXiv preprint arXiv:2311.16079, et al. Medical hallucinations in foundation mod2023. els and their impact on healthcare. arXiv preprint arXiv:2503.05777, 2025. Elliott S Fisher and John E Wennberg. Health care quality, geographic variations, and the challenge of Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio supply-sensitive care. Perspectives in biology and Petroni, Vladimir Karpukhin, Naman Goyal, Heinmedicine, 46(1):69–79, 2003. rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation Fan Gao, Sherry T. Tong, Jiwoong Sohn, Jiahao for knowledge-intensive nlp tasks. Advances in neuHuang, Junfeng Jiang, Ding Xia, Piyalitt Ittichairal information processing systems, 33:9459–9474, wong, Kanyakorn Veerakanjana, Hyunjae Kim, 2020. Qingyu Chen, Edison Marrese Taylor, Kazuma Kobayashi, Akkiko Aizawa, and Irene Li. Medcoreasoner: Reducing language disparities in medi- Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, cal reasoning via language-informed co-reasoning, John Schulman, Ilya Sutskever, and Karl Cobbe. 2026. URL https://arxiv.org/abs/2601.08267. Let’s verify step by step. In The Twelfth International Conference on Learning Representations, Alex Graves. Sequence transduction with recurrent 2023. neural networks, 2012. URL https://arxiv.org/ abs/1211.3711. Che Liu, Haozhe Wang, Jiazhen Pan, Zhongwei Wan, Yong Dai, Fangzhen Lin, Wenjia Bai, Daniel RueckDan Hendrycks, Collin Burns, Steven Basart, Andy ert, and Rossella Arcucci. Beyond distillation: PushZou, Mantas Mazeika, Dawn Song, and Jacob Steining the limits of medical llm reasoning with minimalhardt. Measuring massive multitask language unist rule-based rl. arXiv preprint arXiv:2505.17952, derstanding, 2021. URL https://arxiv.org/abs/ 2025a. 2009.03300. Xiaoke Huang, Juncheng Wu, Hui Liu, Xianfeng Tang, Runze Liu, Junqi Gao, Jian Zhao, Kaiyan Zhang, and Yuyin Zhou. m1: Unleash the potential of Xiu Li, Biqing Qi, Wanli Ouyang, and Bowen test-time scaling for medical reasoning with large Zhou. Can 1b llm surpass 405b llm? rethinking language models. arXiv preprint arXiv:2504.00869, compute-optimal test-time scaling. arXiv preprint 2025. arXiv:2502.06703, 2025b. 10
Zhiyong Lu. Pubmed and beyond: a survey of web tools for searching biomedical literature. Database, 2011:baq036, 2011.
Eliciting factual medical reasoning steps in llms via knowledge graphs, 2025. URL https://arxiv. org/abs/2504.00993.
Geoffrey R Norman and Kevin W Eva. Diagnostic Junde Wu, Jiayuan Zhu, Yunli Qi, Jingkun Chen, Min error and clinical reasoning. Medical education, 44 Xu, Filippo Menolascina, and Vicente Grau. Medi(1):94–100, 2010. cal graph rag: Towards safe medical large language model via graph retrieval-augmented generation. Ankit Pal, Logesh Kumar Umapathi, and MalaikanarXiv preprint arXiv:2408.04187, 2024. nan Sankarasubbu. MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical do- Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong main Question Answering, March 2022. URL http: Zhang. Benchmarking retrieval-augmented genera//arxiv.org/abs/2203.14371. arXiv:2203.14371 tion for medicine. In Lun-Wei Ku, Andre Martins, [cs]. and Vivek Srikumar, editors, Findings of the association for computational linguistics ACL 2024, pages David Rein, Betty Li Hou, Asa Cooper Stickland, Jack6233–6251, Bangkok, Thailand and virtual meetson Petty, Richard Yuanzhe Pang, Julien Dirani, ing, August 2024. Association for Computational Julian Michael, and Samuel R. Bowman. Gpqa: A Linguistics. graduate-level google-proof q&a benchmark, 2023. URL https://arxiv.org/abs/2311.12022. Zhaohui Yang, Chenghua He, Xiaowen Shi, Linjing Li, Qiyue Yin, Shihong Deng, and Daxin Jiang. Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara MahBeyond the first error: Process reward models for davi, Jason Wei, Hyung Won Chung, Nathan Scales, reflective mathematical reasoning. arXiv preprint Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, arXiv:2505.14391, 2025. and others. Large language models encode clinical knowledge. Nature, 620(7972):172–180, 2023. Jaehoon Yun, Jiwoong Sohn, Jungwoo Park, HyunPublisher: Nature Publishing Group. jae Kim, Xiangru Tang, Daniel Shao, Yong Hoe Koo, Ko Minhyeok, Qingyu Chen, Mark GerCharlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral stein, Michael Moor, and Jaewoo Kang. MedKumar. Scaling llm test-time compute optimally PRM: Medical reasoning models with stepwise, can be more effective than scaling model parameters. guideline-verified process rewards. In Christos arXiv preprint arXiv:2408.03314, 2024. Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors, Proceedings of the Rahul Thapa, Qingyang Wu, Kevin Wu, Harrison 2025 Conference on Empirical Methods in Natural Zhang, Angela Zhang, Eric Wu, Haotian Ye, Suhana Language Processing, pages 16554–16571, Suzhou, Bedi, Nevin Aresh, Joseph Boen, et al. DisentanChina, November 2025. Association for Computagling reasoning and knowledge in medical large tional Linguistics. ISBN 979-8-89176-332-6. doi: language models. arXiv preprint arXiv:2505.11462, 10.18653/v1/2025.emnlp-main.837. URL https: 2025. //aclanthology.org/2025.emnlp-main.837/. Guanchu Wang, Junhao Ran, Ruixiang Tang, ChiaYuan Chang, Chia-Yuan Chang, Yu-Neng Chuang, Cyril Zakka, Rohan Shad, Akash Chaurasia, Alex R Dalal, Jennifer L Kim, Michael Moor, Robyn Fong, Zirui Liu, Vladimir Braverman, Zhandong Liu, and Curran Phillips, Kevin Alexander, Euan Ashley, Xia Hu. Assessing and enhancing large language et al. Almanac—retrieval-augmented language modmodels in rare disease question-answering, 2024. els for clinical medicine. Nejm ai, 1(2):AIoa2300068, URL https://arxiv.org/abs/2408.08422. 2024. Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Sheng Zhang, Qianchu Liu, Guanghui Qin, Tristan Naumann, and Hoifung Poon. Med-rlvr: Math-shepherd: Verify and reinforce llms step-byEmerging medical reasoning from a 3b base step without human annotations. arXiv preprint model via reinforcement learning. arXiv preprint arXiv:2312.08935, 2023. arXiv:2502.19655, 2025a. Juncheng Wu, Wenlong Deng, Xingxuan Li, Sheng Liu, Taomian Mi, Yifan Peng, Ziyang Xu, Yi Liu, Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Hyunjin Cho, Chang-In Choi, Yihan Cao, Hui Ren, Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Xiang Li, Xiaoxiao Li, and Yuyin Zhou. Medreason: Zhou, and Junyang Lin. The lessons of developing 11
process reward models in mathematical reasoning. arXiv preprint arXiv:2501.07301, 2025b. Qinyue Zheng, Salman Abdullah, Sam Rawal, Cyril Zakka, Sophie Ostmeier, Maximilian Purk, Eduardo Reis, Eric J. Topol, Jure Leskovec, and Michael Moor. Miriad: Augmenting llms with millions of medical query-response pairs, 2025. URL https: //arxiv.org/abs/2506.06091. Jiachen Zhu, Congmin Zheng, Jianghao Lin, Kounianhua Du, Ying Wen, Yong Yu, Jun Wang, and Weinan Zhang. Retrieval-augmented process reward model for generalizable mathematical reasoning. arXiv preprint arXiv:2502.14361, 2025.
12
A
Table of Notations
Table 5 summarizes the notation used throughout the main manuscript. Symbol
Description
System q Q Y yq ŷq C(ŷq , yq ) π Gϕ µϕ µact ϕ µrwd ϕ D Dt
question space of questions space of answers ground-truth answer for question q predicted answer for question q correctness function; 1 if ŷq matches yq , else 0 policy (frozen) parameterized inference procedure (q, π, D) → Y Process Reward Agent (PRA) PRA controller component PRA scoring function knowledge base (collection of documents) set of retrieved documents at step t
Traces & Steps τ τt K t st sK j
reasoning trace partial reasoning trace up to step t number of reasoning steps in a trace step index (1 . . . K) reasoning step at position t final step of the reasoning trace beam index
PRA Actions & Rewards r̂t ât rt at (j) R(τt )
predicted reward at step t predicted action at step t reward label at step t action label at step t cumulative reward of partial trace j up to step t
Training & Labels ℓ(1) ℓ(2) V m md ∆m ϵglobal θdep
logit vector at the first output slot (reward) logit vector at the second output slot (action) vocabulary of the PRA margin without retrieval margin with retrieved documents margin shift: m − md global threshold for search labels search threshold at inference
Inference B b
beam width branching factor Table 5: Summary of notations
13
B
Stage-Level Batching
Figure 6 presents simplified pseudocode for PRA-guided beam search. Each question is managed by a Trace object that maintains a beam of partial reasoning traces and a stage tag ∈ {reason, reward, search, done}. At each iteration, the global queue is drained, traces are partitioned by stage, and each partition is dispatched as a single batched operation to π, µϕ , or ρ. PRA Beam Search Pseudocode class Trace: stage: REASON | REWARD | SEARCH | DONE beams: [{steps, cum_score, is_done}] candidates: [(beam_idx, text, score)] evidence: [str] | None def after_reason(self, outputs): self.candidates = outputs self.stage = REWARD def after_reward(self, scored): if should_search(scored): self.stage = SEARCH else: self.expand_and_prune(scored) def after_search(self, docs): self.evidence = docs self.stage = REWARD def expand_and_prune(self, scored): self.beams = top_B(scored, key=cum_score) self.stage = DONE if all_done else REASON # ---- Global queue ---queue = [Trace(q) for q in questions] while queue: buckets = partition(queue, key=stage) batch_retrieve(buckets[SEARCH]) batch_generate(buckets[REASON]) batch_score(buckets[REWARD]) queue = [t for t in queue if t.stage != DONE]
Figure 6: Simplified pseudocode. Each Trace manages one question and cycles through four stages. The global queue collects all active traces, partitions them by pending stage, and dispatches each partition as a batched operation to the policy (π), retriever (ρ), or reward agent (µϕ ), regardless of per-trace step index.
C
Additional Training Details
We fine-tune Qwen3-4B-Instruct to predict the reasoning and search labels described in Section 4.1. Training is performed with a learning rate of 3 × 10−5 using a cosine decay schedule with 100 warmup steps. We use a weight decay of 0.01, an effective batch size of 16, and train for 3 epochs in bfloat16 precision. In the main experiments, we use the prompt shown in Figure 9 and train PRA in the always-search setting, where the search label is fixed to 1 for every reasoning step. For the Search–Accuracy Trade-off analysis, we instead train PRA using search labels derived from the margin-shift criterion described in Section 4.1. 14
D
Prompt Templates
Figure 7, 8 and 9, show the prompts used throughout our experiments. Policy Prompt System: Solve the following question step-by-step. Do not analyze individual options in a single step. Each step of your explanation must start with ’Step number:’ format. You must conclude the answer using the phrase ’the answer is (option alphabet)’ at the end of your step. User: === QUESTION === {question text} A: {option A} ...
Figure 7: Policy prompt used for all PRA experiments. The prompt instructs explicit step-wise reasoning for easier parsing during search, and a standardized final answer format for answer extraction.
15
Teacher Prompt System: You are a medical expert responsible for evaluating the quality of the last reasoning step in a solution to a medical question. You are provided with relevant documents, the question, and the reasoning trace including prior steps. Your task is to critically assess only the last reasoning step, considering its logical coherence, medical validity, and consistency with the evidence. You must only return one score, and output nothing else: Reasoning Score: Score 1 if the last step is logically coherent, medically sound, and aligns with the provided evidence; otherwise, score 0. Output only a single digit of your reasoning score in the following format: 1 or 0 (1: correct, 0: incorrect) User: === DOCUMENTS === Doc 1: {document 1} ... === QUESTION === {question text} A: {option A} ... === CORRECT ANSWER === ({answer idx}): {answer text} === REASONING TRACE === Step 1: {step 1 text} ...
Figure 8: Teacher prompt used for all PRA experiments. The prompt evaluates the last reasoning step given retrieved documents, the question with options, correct answer, and the reasoning trace up to and including the current step.
16
PRA Prompt System: You are an evaluator responsible for assessing the quality of the last reasoning step in a solution to a medical question. You are provided with relevant documents, the question, and the reasoning path (including prior steps and their rewards, if they exist). Your task is to critically assess only the last reasoning step, considering its logical coherence, medical validity, and consistency with the evidence. You must only return two scores, and output nothing else: 1. Reasoning Reward: Score 1 if the last step is logically coherent, medically sound, and aligns with the provided evidence; otherwise, score 0. 2. Search Reward: Score 1 if, in order to evaluate the last reasoning step, you needed to refer to the provided evidence (i.e., the step required searching for or validating with external information), or if the reasoning step itself explicitly involves searching, retrieval, or referencing outside knowledge; otherwise, score 0. Provide your evaluation as two numbers, separated by a comma and a space, with no additional explanation or text. The first number is the Reasoning Reward, and the second is the Search Reward, as in the following examples: 0,0 1,0 0,1 1,1 For instance: If Reasoning Reward = 0 and Search Reward = 1, write: 0,1 If both are 1: 1,1 If both are 0: 0,0 If Reasoning Reward = 1 and Search Reward = 0: 1,0 User: === DOCUMENTS === Doc 1: {document 1} ... === QUESTION === {question text} A: {option A} ... === REASONING TRACE === Step 1: {step 1 text} ...
Figure 9: PRA prompt used in all experiments. The documents section appears only when search is triggered at the current step.
17