Cycle-Consistent Search: Question Reconstructability as a Proxy Reward for Search Agent Training Sohyun An1,2,∗ , Shuibenyang Yuan1 , Hayeon Lee1 , Cho-Jui Hsieh2,† , Alexander Min1,† 1
arXiv:2604.12967v1 [cs.AI] 14 Apr 2026
∗
Meta Superintelligence Labs, 2 UCLA Work done at Meta, † Joint last author
Reinforcement Learning (RL) has shown strong potential for optimizing search agents in complex information retrieval tasks. However, existing approaches predominantly rely on gold supervision, such as ground-truth answers, which is difficult to scale. To address this limitation, we propose Cycle-Consistent Search (CCS), a gold-supervision-free framework for training search agents, inspired by cycle-consistency techniques from unsupervised machine translation and image-to-image translation. Our key hypothesis is that an optimal search trajectory, unlike insufficient or irrelevant ones, serves as a lossless encoding of the question’s intent. Consequently, a high-quality trajectory should preserve the information required to accurately reconstruct the original question, thereby inducing a reward signal for policy optimization. However, naive cycle-consistency objectives are vulnerable to information leakage, as reconstruction may rely on superficial lexical cues rather than the underlying search process. To reduce this effect, we apply information bottlenecks, including exclusion of the final response and named entity recognition (NER) masking of search queries. These constraints force reconstruction to rely on retrieved observations together with the structural scaffold, ensuring that the resulting reward signal reflects informational adequacy rather than linguistic redundancy. Experiments on question-answering benchmarks show that CCS achieves performance comparable to supervised baselines while outperforming prior methods that do not rely on gold supervision. These results suggest that CCS provides a scalable training paradigm for training search agents in settings where gold supervision is unavailable. Date: April 15, 2026 Correspondence: Sohyun An at [email protected]
1
Introduction
Recent advances in Large Language Models (LLMs) (Brown et al., 2020; Chung et al., 2024) have led to the emergence of search agents (Li et al., 2025; Jin et al., 2025a; Song et al., 2025; Zheng et al., 2025; Chen et al., 2025) that navigate complex information environments through iterative planning and tool use (Yao et al., 2022). Unlike conventional single-step retrieval systems, these agents actively formulate search queries, inspect retrieved observations, and adapt subsequent actions to answer complex, multi-faceted questions. Reinforcement Learning (RL) has become a standard framework for optimizing such sequential decision-making processes (Schulman et al., 2017; Shao et al., 2024). However, existing RL-based search agents typically rely on gold supervision, such as ground-truth answers, to define reward signals. This reliance creates a fundamental scalability bottleneck: in specialized or rapidly evolving domains, such supervision is often prohibitively expensive or unavailable, making it difficult to construct reliable rewards for search trajectories. To address this limitation, we propose Cycle-Consistent Search (CCS), a gold-supervision-free framework for training search agents using the internal structure of the search process itself. Our approach is inspired by cycle-consistency techniques from unsupervised machine translation (He et al., 2016; Lample et al., 2017; Han et al., 2021) and image-to-image translation (Zhu et al., 2017; Liu et al., 2017; Huang et al., 2018), where the quality of a transformation is assessed by whether the original input can be recovered through an inverse mapping. We extend this idea to search by treating a search trajectory τ as an information-preserving encoding of the source question q. As illustrated in Figure 1, trajectories that omit essential steps for multi-hop 1
Question (𝑞): What is the population of the city where the architect who designed the Burj Khalifa was born? Appropriate search Action (𝑎1 ): Who is the architect of the Burj Khalifa? Observation (𝑜1 ): …Adrian Smith is the chief architect who designed the Burj Khalifa… Action (𝑎2 ): Where was Adrian Smith born? Observation (𝑜2 ): …Adrian Smith was born in Chicago, United States… Action (𝑎3 ): What is the population of Chicago? Observation (𝑜3 ): …the population of Chicago is approximately 2.7 million… Action (𝑎4 ): …Adrian Smith, the architect who designed the Burj Khalifa, was born in Chicago, which has a population of approximately 2.7M…
Insufficient search
Irrelevant search
Action (𝑎1 ): Who is the architect of the Burj Khalifa? Observation (𝑜1 ): …Adrian Smith is the chief architect who designed the Burj Khalifa… Action (𝑎2 ): Where was Adrian Smith born? Observation (𝑜2 ): …Adrian Smith was born in Chicago, United States… Action (𝑎3 ): ...Adrian Smith is the architect who designed the Burj Khalifa and he was born in Chicago …
Action (𝑎1 ): Who was the chief structural engineer of the Burj Khalifa? Observation (𝑜1 ): …William F. Baker served as the chief structural engineer… Action (𝑎2 ): Where was William F. Baker born? Observation (𝑜2 ): …William F. Baker was born in Fulton, Missouri, United States… Action (𝑎3 ): What is the population of Fulton? Observation (𝑜3 ): …the population of Fulton is approximately 12,000… Action (𝑎4 ): …William F. Baker, who served as the chief structural engineer of the Burj Khalifa, was born in Fulton, which has a population of approximately 12,000…
Figure 1 Examples of search trajectories. A high-quality trajectory (left) constitutes a lossless encoding of the question’s
intent, preserving all necessary information required to reconstruct the original question. In contrast, trajectories that involve insufficient (middle) or irrelevant (right) search steps fail to preserve this information, resulting in incomplete or distorted representations of the question.
questions (middle) or drift toward irrelevant information (right) fail to preserve the information needed to recover the original question, whereas high-quality trajectories (left) do. Formally, our central hypothesis is that, under a high-quality policy π, the source question should be recoverable from the resulting trajectory: ϕ π q− →τ − → q̂ ≈ q, where ϕ denotes a reconstruction function that maps a search trajectory back to the source question. Under this view, reconstruction quality provides a proxy for trajectory quality. A central challenge in applying cycle-consistency to search is information leakage. Search trajectories may contain superficial lexical cues that enable reconstruction without reflecting genuine search quality. For example, search queries often overlap with the wording of the original question, and final responses may partially restate the question for context, allowing the original question to be reconstructed even when the underlying search process is uninformative or low quality. To mitigate this issue, we introduce two information bottlenecks: excluding the final response from the trajectory and applying Named Entity Recognition (NER) masking (Yamada et al., 2020; Sun et al., 2019; Liu et al., 2020) to search queries, replacing entities such as names and locations with generic tags (e.g., [LOC]). These constraints reduce direct lexical shortcuts and encourage reconstruction to depend on retrieved observations together with the structural scaffold (Figure 2). Under this formulation, training a search agent amounts to enforcing cycle-consistency over search trajectories, with reconstruction quality on an information-bottlenecked trajectory τ̃ serving as a proxy reward for trajectory quality. We evaluate CCS on seven representative question-answering benchmarks and a deep research benchmark. The results show that CCS achieves performance comparable to supervised baselines while outperforming existing methods that do not rely on gold supervision. These findings suggest that cycle-consistency can provide a scalable proxy objective for training search agents in settings where gold supervision is unavailable. In summary, our contributions are as follows: • We introduce Cycle-Consistent Search (CCS), a gold-supervision-free framework for training search agents by enforcing cycle-consistency between questions and search trajectories. • We identify information leakage as a practical challenge in cycle-consistent search training and use information bottlenecks to reduce lexical shortcuts. • We show empirically, across seven question-answering benchmarks and a deep research benchmark, that CCS achieves performance competitive with supervised methods while outperforming prior approaches that do not rely on gold supervision.
2
2
Related Works
From Multi-Step Retrieval to Interactive Search Agents Prior work in information retrieval has largely centered on single-shot retrieve-and-read pipelines (Lee et al., 2019; Guu et al., 2020; Karpukhin et al., 2020a). This paradigm expanded with the emergence of LLM-based systems that perform retrieval over multiple steps. Prompted tool-use frameworks such as ReAct (Yao et al., 2022) and multi-step retrieval methods such as IRCoT (Trivedi et al., 2023) showed that LLMs can improve retrieval coverage on complex queries by iteratively issuing retrieval actions conditioned on intermediate reasoning or generations. These approaches established the broader view that retrieval can be formulated as a sequential, interactive process rather than a single retrieval step. Building on this perspective, subsequent work increasingly developed interactive search agents that actively generate queries, consume observations, and iteratively refine their information needs over the course of search. Search-o1 (Li et al., 2025) demonstrated the effectiveness of this search-centric agent design through prompting-based iterative query formulation and evidence collection over the web. More recent approaches, including Jin et al. (2025a), Song et al. (2025), Zheng et al. (2025), and Chen et al. (2025), have further advanced this paradigm by training search agents with reinforcement learning (Schulman et al., 2017; Shao et al., 2024). In these approaches, search is explicitly modeled as sequential decision-making over query actions and observation consumption, making RL a natural framework for optimizing end-to-end search behavior. Beyond Gold Supervision for Search Agents Many RL-based search agents have demonstrated strong performance, but they remain dependent on gold supervision, such as ground-truth answers, to construct reliable reward signals. This dependence creates a scalability bottleneck in domains where high-quality annotations are scarce, expensive, or rapidly evolving. To reduce the cost of annotation, several recent methods have explored alternatives to gold supervision by leveraging alternative supervision signals, including intrinsic signals and self-generated pseudo-labels. RLIF (Zhao et al., 2025), for example, derives reward signals from the model’s internal confidence to guide training without external labels. Rubric-based LLM judges (Liu et al., 2023), which we refer to as Constitutional Judges, can also be used to provide reward signals by scoring model outputs according to a predefined set of evaluation criteria. Test-time RL methods such as TTRL (Zuo et al., 2025) instead use agreement among multiple sampled rollouts as a proxy signal for updating policies on unlabeled instances. While these approaches reduce reliance on human annotation, their training signals are not specifically aligned with the central objective of search: rewarding trajectories that acquire sufficient and necessary external evidence to satisfy the original information need. Extending Cycle-Consistency to Search Trajectories Cycle-consistency has been widely used as a learning principle that encourages transformations to preserve essential information, notably in unsupervised machine translation (He et al., 2016; Lample et al., 2017; Han et al., 2021) and image-to-image translation (Zhu et al., 2017; Liu et al., 2017; Huang et al., 2018). Its central intuition is that a good transformation should retain enough information for the original input to be recoverable through an inverse mapping. Related reconstruction-based objectives have also been explored in retrieval-augmented pretraining, where retrieved documents serve as latent variables for reconstructing targets (Guu et al., 2020). In the search setting, we consider cycle-consistency over agentic search trajectories, treating a trajectory as an information-preserving encoding of the source question. A central challenge in this setting is information leakage: lexical overlap between the question and intermediate search queries (or the final response) can enable trivial reconstruction without reflecting genuine search quality. We address this issue through information bottlenecks, so that successful reconstruction must rely on retrieved observations together with the structural scaffold rather than surface-form redundancy.
3
Method
In this section, we formally introduce Cycle-Consistent Search (CCS), a framework designed to train search agents without gold supervision. We first establish the theoretical connection between search trajectory optimization and cycle-consistency. We then derive an information-theoretic objective and introduce structural constraints that prevent information leakage. Finally, we detail the optimization procedure using Group Relative Policy Optimization (GRPO). A schematic overview of CCS is shown in Figure 2. 3
𝐸(𝑞)
𝐸(𝑞) ො
Policy Model (𝜋𝜃 )
Search Env
𝜏1
𝜏2
𝜏3
…
𝜏𝐺−1
𝜏𝐺
𝑟1
𝑟2
𝑟3
…
𝑟𝐺−1
𝑟𝐺
Reconstructed Question (𝑞): ො What is the population of the architect’s birthplace city who designed the Burj Khalifa?
Reconstructor 𝑃𝜙
𝑆𝑖𝑚(𝐸 𝑞 , 𝐸(𝑞)) ො
Action 𝑎𝑡 Observation 𝑜𝑡
Action (𝑎1 ): Who is the architect of the [LOC1]? Observation (𝑜1 ): …Adrian Smith is the chief architect who designed the Burj Khalifa… Action (𝑎2 ): Where was [PER1] born? Observation (𝑜2 ): …Adrian Smith was born in Chicago, United States… Action (𝑎3 ): What is the population of [LOC2]? Observation (𝑜3 ): …the population of Chicago is approximately 2.7 million…
𝜏ǁ1
𝜓 = 𝜓𝑚𝑎𝑠𝑘 ∘ 𝜓𝑡𝑟𝑎𝑐𝑒
Action (𝑎1 ): Who is the architect of the Burj Khalifa? 𝜏1 Observation (𝑜1 ): …Adrian Smith is the chief architect who designed the Burj Khalifa… Action (𝑎2 ): Where was Adrian Smith born? Observation (𝑜2 ): …Adrian Smith was born in Chicago, United States… Action (𝑎3 ): What is the population of Chicago? Observation (𝑜3 ): …the population of Chicago is approximately 2.7 million… Action (𝑎4 ): …Adrian Smith, the architect who designed the Burj Khalifa, was born in Chicago, which has a population of approximately 2.7M…
Question (𝑞): What is the population of the city where the architect who designed the Burj Khalifa was born?
Figure 2 Overview of the Cycle-Consistent Search (CCS) framework. The agent πθ generates search trajectories τ to
address a question q. To ensure the search process is driven by information gain rather than lexical copying, τ is passed through a strategic information bottleneck (ψ) that removes the aT and masks entities within actions (highlighted in grey). A frozen reconstructor Pϕ then attempts to recover the original question from τ̃ . The agent is optimized using GRPO based on the semantic similarity between q and q̂, encouraging trajectories that capture sufficient evidence.
3.1
Cycle-Consistency as Information Conservation
Our approach is grounded in the principle of cycle-consistency, widely utilized in unsupervised machine translation (He et al., 2016; Lample et al., 2017; Han et al., 2021) and unpaired image-to-image translation (Zhu et al., 2017; Liu et al., 2017; Huang et al., 2018). Let X and Y denote two distinct domains. The fundamental assumption in these settings is that an ideal mapping F : X → Y preserves the essential semantic content of the input, such that there exists an inverse mapping G : Y → X satisfying x ≈ G(F (x)) for any x ∈ X . This implies that the transformation is information-preserving: the latent information required to reconstruct x is conserved throughout the transformation. We transpose this principle to agentic search. A user’s question q ∈ Q represents a semantic intent, and a search trajectory τ ∈ T represents a dynamic expansion of that intent into a sequence of actions and observations. Under the assumption of information conservation, an optimal search trajectory τ ∗ acts as a lossless encoding of the question q. Consequently, if τ contains the necessary and sufficient evidence to resolve q, it should be possible to reconstruct q solely from τ . Formally, this establishes a cycle q → τ → q̂, where the closeness between q and q̂ serves as a proxy for the quality of the intermediate representation τ , independent of external supervision.
3.2
Problem Formulation and Objective
We model the search process as a Partially Observable Markov Decision Process (POMDP). The agent, parameterized by θ, is defined as a policy πθ (· | q, ht−1 ), where ht−1 is the history of past actions and observations. At each intermediate step t < T , the agent issues an action (search query) at and receives an observation (search results) ot from the search environment. After T − 1 search steps, the agent produces a final response aT . The full trajectory is τ = (a1 , o1 , . . . , aT −1 , oT −1 , aT ). Under the cycle-consistency view in Section 3.1, the training objective can be formulated as maximizing the mutual information (MI) between the original question q and the generated trajectory τ . Here, the mutual information I(Q; Tθ ) is defined as: I(Q; Tθ ) = H(Q) − H(Q | Tθ ). (1) Since the entropy of the question distribution H(Q) is constant with respect to θ, maximizing MI is equivalent to minimizing the conditional entropy H(Q | Tθ ), which corresponds to maximizing the expected log-likelihood of the question given the trajectory: J (θ) = Eq∼D,τ ∼πθ (·|q) [log P (q | τ )] .
(2)
Directly modeling the true posterior P (q | τ ) is intractable. We therefore employ a variational surrogate using 4
a fixed, pre-trained reconstructor Pϕ (q | ·), which serves as a tractable approximation to the inverse mapping G. However, naively maximizing log Pϕ (q | τ ) can lead to degenerate solutions in which the agent encodes q into the action sequence through trivial lexical copying (e.g., by repeating the question in a search query or partially restating it in the final response for context), allowing the reconstructor to recover q without relying on the search results. To mitigate this issue, we introduce a bottleneck transformation ψ : T → T̃ and optimize the following objective: θ∗ = argmax Eq∼D, τ ∼πθ (·|q) log Pϕ q | ψ(τ ) . (3) θ
3.3
Strategic Information Bottlenecks
The transformation function ψ imposes structural constraints on the trajectory so that it retains the search scaffold while reducing lexical shortcuts that would otherwise allow trivial reconstruction of the source question. We define ψ as the composition of two operations: final response exclusion and entity masking: ψ = ψmask ◦ ψtrace . Final response exclusion. We first address leakage through the final response. In standard QA interactions, the final response aT often paraphrases q to provide context. Including it in the reconstruction input can bypass the evaluation of the search process. Therefore, we define the trace operator ψtrace which truncates the trajectory to exclude the final generation step, preserving only the investigative actions and observations: ψtrace (τ ) = (a1 , o1 , . . . , aT −1 , oT −1 ). Entity masking on search actions. We next mitigate leakage through search queries. Raw search queries at typically exhibit high lexical overlap with q. To prevent the reconstructor from recovering q from these surface-level cues even when the underlying search process is uninformative, we apply a masking operator ψmask that identifies named entities within each search query at (e.g., “Burj Khalifa”) and replaces them with generic typed tags (e.g., [LOC]), yielding a masked action ãt . The observations ot remain unmasked. This operation substantially reduces the direct lexical channel from actions to the question, forcing reconstruction to resolve masked content primarily from evidence in ot rather than from copied words in at . The full bottleneck transformation produces a processed trajectory: ψ(τ ) = τ̃ = (ã1 , o1 , . . . , ãT −1 , oT −1 ). Consequently, to recover the original question from τ̃ , the reconstructor must leverage the information in the observations together with the structural intent expressed by the masked queries (e.g., τ̃1 in Figure 2). Trajectories with an incorrect or insufficient search scaffold, or with irrelevant or insufficient observations, will fail to preserve the relational structure or informational content required to reconstruct the original question, resulting in low reconstruction fidelity and thus low reward. By tying the agent’s incentive to both the correctness of the search scaffold and the informativeness of the resulting observations, CCS makes the reward sensitive to the quality of the search trajectory. In this way, the cycle-consistency objective penalizes trajectories that are incorrect, incomplete, or noisy, while favoring those that preserve the relational structure of the question and gather the information required for faithful reconstruction.
3.4
Optimization via Group Relative Policy Optimization
The objective in Section 3.2 frames search-agent training as maximizing the reconstructability of the original question from the bottlenecked trajectory. This naturally defines a trajectory-level reward: trajectories that better preserve the information needed to reconstruct q receive higher rewards. We optimize the search agent with reinforcement learning using our cycle-consistency reward. Specifically, we adopt Group Relative Policy Optimization (GRPO) (Shao et al., 2024), which has proven effective for search-agent optimization (Jin et al., 2025a) and avoids the need for a separate value-function critic by using group-based relative advantages. The RL objective is to optimize πθ by maximizing expected returns over questions q ∼ D: max Eq∼D,τ ∼πθ (·|q) r(τ ) . πθ
5
(4)
For each question q, the agent samples a group of G trajectories {τ1 , τ2 , . . . , τG } from the current policy πθold . For each τi , we compute a reconstructed question q̂i ∼ Pϕ (· | ψ(τi )) and define the reward as the semantic similarity between q and q̂i : r(τi ) = Sim E(q), E(q̂i ) , (5) where E(·) is a sentence embedding function and Sim(·, ·) denotes cosine similarity. We then compute an advantage Ai by normalizing rewards within the group: Ai =
r(τi ) − µ({r(τj )}G j=1 ) , G σ({r(τj )}j=1 ) + ϵ
(6)
where µ and σ denote the mean and standard deviation of group rewards. The GRPO objective is then: " G π (τ |q) π (τ |q) 1 X θ i θ i LGRP O (θ) = Eq∼D min Ai , clip , 1 − ϵ, 1 + ϵ Ai G i=1 πθold (τi |q) πθold (τi |q) (7) # −βDKL (πθ ∥ πref ) . Here, observations are treated as environment outputs and πθ (τi | q) refers to the likelihood of the agentgenerated tokens/actions (Jin et al., 2025a). This objective increases the likelihood of trajectories that yield higher reconstruction fidelity relative to the group average, thereby encouraging search behaviors that better preserve the structure and information needed to recover the original question.
4
Experiments
We empirically evaluate Cycle-Consistent Search (CCS) across a diverse range of knowledge-intensive tasks. The experimental design is structured to assess the framework’s efficacy in both multi-hop and general question-answering scenarios against established baselines.
4.1
Datasets and Models
To ensure a comprehensive assessment, we categorize the evaluation benchmarks into two groups based on search complexity, following Jin et al. (2025a). For multi-hop reasoning, which necessitates iterative information retrieval and synthesis, we employ HotpotQA (Yang et al., 2018), 2WikiMQA (Ho et al., 2020), MuSiQue (Trivedi et al., 2022), and Bamboogle (Press et al., 2023). For general question answering, we utilize Natural Questions (NQ) (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017), and PopQA (Mallen et al., 2023). We select policy models from the Qwen family (Yang et al., 2025) to evaluate performance across different parameter scales and both base and instruct variants: Qwen2.5-7B-Instruct, Qwen3-4B-Instruct-2507, and Qwen3-32B. All evaluations are conducted using Gemini 2.5 Flash (Comanici et al., 2025) as the evaluator model to ensure consistent assessment of answer accuracy (see Section B.2).
4.2
Baselines
We compare CCS against a diverse set of baselines spanning model inference-only methods, methods trained with gold supervision, and methods that do not rely on gold supervision. Model Inference: We include Direct inference, zero-shot Chain-of-Thought (CoT) (Wei et al., 2022), Interleaved
Retrieval CoT (IRCoT) (Trivedi et al., 2023), Retrieval-Augmented Generation (RAG) (Lewis et al., 2020), and Search-o1 (Li et al., 2025), which represent prompting-based or retrieval-based approaches without additional training. With Gold Supervision: We include Supervised Fine-Tuning (SFT), which trains the model to generate ground-
truth answers, and Search-R1 (Jin et al., 2025a), which optimizes search agents with reinforcement learning using rewards derived from ground-truth answers.
6
Table 1 Main results. The best and second-best performances are shown in bold and underline, respectively. † /⋆ denotes
in-domain/out-of-domain datasets. Methods
General QA NQ
†
TriviaQA
∗
Multi-Hop QA PopQA
∗
HotpotQA
†
Avg.
2wiki∗
Musique∗
Bamboogle∗
0.091 0.128 0.129 0.135 0.197
0.120 0.384 0.304 0.304 0.536
0.191 0.275 0.440 0.465 0.486
Qwen2.5-7B-Instruct Model Inference
Direct Inference CoT IRCoT RAG Search-o1
0.344 0.430 0.608 0.645 0.654
0.111 0.139 0.761 0.805 0.588
0.144 0.203 0.515 0.535 0.462
0.279 0.325 0.486 0.501 0.486
0.248 0.318 0.279 0.331 0.481
SFT Search-R1 (EM)
0.366 0.693
0.587 0.872
0.220 0.627
0.312 0.587
0.617
0.244
0.087
0.224 0.568
0.295 0.601
TTRL CJ RLIF CCS
0.695 0.704 0.672
0.500 0.865 0.846
0.593 0.633 0.586
0.500 0.545 0.536
0.455 0.560 0.444 0.567
0.173 0.209 0.190 0.239
0.540 0.544 0.408
0.608
0.544 0.580 0.526
0.606
0.253 0.260 0.203 0.264 0.335
0.101 0.114 0.131 0.126 0.150
0.256 0.408 0.264 0.312 0.384
0.204 0.261 0.423 0.461 0.364 0.649
W. Gold Supervision
W.O. Gold Supervision
0.712
0.880
0.639
0.598
0.267
Qwen3-4B-Instruct-2507 Model Inference
Direct Inference CoT IRCoT RAG Search-o1
0.349 0.404 0.613 0.688 0.541
SFT Search-R1 (EM)
0.760
0.921
TTRL CJ RLIF CCS
0.740 0.709 0.686 0.744
W. Gold Supervision
0.312
0.091 0.142 0.770 0.803 0.489
0.119 0.204 0.525 0.564 0.234
0.260 0.294 0.457 0.472 0.413
0.474
0.196 0.671
0.638
0.668
0.277
0.078
0.120 0.608
0.902 0.893 0.879 0.902
0.630 0.677 0.630
0.540 0.539 0.566 0.625
0.488 0.495 0.514 0.606
0.018 0.189 0.221 0.262
0.482 0.552 0.520
0.632
0.543 0.579 0.574 0.636
W.O. Gold Supervision
0.275
0.683
0.275
0.247
Qwen3-32B Model Inference
Direct Inference CoT IRCoT RAG Search-o1
0.485 0.488 0.522 0.714 0.722
0.308 0.322 0.697 0.838 0.881
0.277 0.282 0.443 0.593 0.608
0.368 0.364 0.411 0.560 0.579
0.340 0.338 0.223 0.358 0.629
0.144 0.144 0.113 0.173 0.247
0.224 0.336 0.312 0.400 0.584
0.307 0.325 0.389 0.519 0.607
SFT Search-R1 (EM)
0.454 0.681
0.670 0.880
0.262 0.578
0.362 0.648
0.314 0.736
0.128 0.278
0.742
0.288
0.354 0.649
0.640
0.870 0.556 0.887
0.645
0.616
0.523 0.665 0.568
0.739
0.301
0.552 0.718
0.221 0.284
0.540 0.731 0.528 0.712
0.583 0.624 0.582
W. Gold Supervision
W.O. Gold Supervision
TTRL CJ RLIF CCS
0.733
0.714
0.733
0.896
0.601 0.624
0.666
7
0.653
0.238
0.662
Table 2 Ablation studies on the components of CCS. The best and second-best performances are shown in bold and
underline, respectively. Methods
Actions + Observations + Final Response Actions + Observations Observations Masked Actions + Observations
General QA
Multi-Hop QA
NQ†
TriviaQA∗
PopQA∗
HotpotQA†
2wiki∗
Musique∗
Bamboogle∗
0.700 0.679 0.691
0.852 0.852 0.866
0.618 0.598 0.610
0.543 0.526 0.589
0.476 0.474
0.202 0.201 0.225
0.536 0.488 0.520
0.712
0.880
0.639
0.598
0.588
0.567
0.239
0.608
Avg.
0.561 0.545 0.584
0.606
Without Gold Supervision: We compare against RLIF (Zhao et al., 2025), which derives reward signals from
the model’s internal confidence; Constitutional Judge (CJ) (Liu et al., 2023), which derives reward signals from a rubric-based LLM judge using predefined evaluation criteria; and TTRL (Zuo et al., 2025), which uses agreement among multiple sampled rollouts as a proxy reward signal.
4.3
Setup
Our experimental protocol is informed by the configuration used in Search-R1 (Jin et al., 2025a). We utilize the same search engine interface for all methods employing tool use. For the RAG baseline, we employ E5 (Wang et al., 2022) as the dense retriever, indexing a 2018 Wikipedia dump (Karpukhin et al., 2020b) as the knowledge source. The training corpus consists of a mixture of the NQ and HotpotQA training sets. Agents are trained for 300 steps with a global batch size of 512. Please refer to Section B for more details.
4.4
Results
We present the main results in Table 1. Across all three models, CCS achieves the best average performance among methods that do not rely on gold supervision, consistently outperforming both model inference baselines and prior gold-free training methods. In particular, CCS improves over the strongest competing non-gold baseline by 4.5%, 9.8%, and 6.1% on Qwen2.5-7B-Instruct, Qwen3-4B-Instruct-2507, and Qwen3-32B, respectively. These results indicate that cycle-consistency provides an effective training signal for search-agent optimization without access to ground-truth answers. Notably, CCS is also competitive with gold-supervised training. On Qwen2.5-7B-Instruct and Qwen3-32B, CCS achieves the best overall average performance in the table, surpassing Search-R1 by 0.5 and 1.3 points, respectively. On Qwen3-4B-Instruct-2507, Search-R1 achieves the highest average score, while CCS also delivers strong overall performance. At the dataset level, the strongest method varies across benchmarks, but CCS remains consistently strong across both general QA and multi-hop QA, yielding the most robust overall performance among gold-free methods. Overall, these results show that CCS substantially narrows—and in some settings reverses—the gap between gold-free and gold-supervised search-agent training.
5
Analysis
5.1
Ablation Study
We conduct an ablation study to evaluate the role of the proposed information bottlenecks. As shown in Table 2, variants that expose the reconstructor to unfiltered lexical cues—either by including the final response or by using unmasked actions—perform worse than the CCS bottleneck design (Masked Actions + Observations), with average scores of 0.561 and 0.545, respectively. This suggests that both the final response and raw search queries introduce shortcut signals that weaken the intended training objective. Using observations alone yields an average score of 0.584, which remains below the 0.606 achieved by the CCS bottleneck design. This performance gap indicates that masked actions contribute more than simply removing lexical leakage: they preserve useful structural intent that helps the reconstructor interpret the observations within a specific investigative context. Overall, these results show that the proposed bottlenecks are effective not only because they suppress surface-level shortcuts, but also because they retain the structural signals needed for faithful reconstruction.
8
Low Reward (Information Void) Question: Which pianist born in 1954 composed Wenji? Action (𝑎1 ): Which pianist born in 1954 composed Wenji → [MISC1]? Observation (𝑜1 ): Liu Wenjin (May 1937 – 27 June 2013) was a Chinese composer of classical Chinese music best known for his erhu piece Sanmenxia Changxiangqu … Low Reward (Shallow Depth) Question: In what county is the village on the Isis River that is 75 miles from the Australian city with a population of approximately 225,000? Action (𝑎1 ): Australian → [MISC1] city with population of approximately 225,000 Observation (𝑜1 ): Hobart is the capital and most populous city of the Australian island … with a population of approximately 225,000 … Action (𝑎2 ): village on the Isis River → [LOC1] Observation (𝑜2 ): Maitland is a rural locality, which is located on the Isis River … Action (𝑎3 ): distance between Isis River → [LOC1] and Hobart → [LOC2] Observation (𝑜3 ): 442.4km from Hobart to …
High Reward Question: Which scientist lived longer, George Adomian or Edward Keonjian? Action (𝑎1 ): Which scientist lived longer, Goerge Adomian → [PER1] or Edward Keonjian → [PER2]? Observation (𝑜1 ): Edward Keonjian lived from 1909 to 1999 … Action (𝑎2 ): George Adomian → [PER1] lifespan Observation (𝑜2 ): George Adomian lived from 21 March 1922 to 17 June 1996 …
Figure 3 Qualitative analysis of search trajectories. CCS assigns low rewards to trajectories exhibiting (a) Information Void, where the retrieved observations fail to satisfy key entity constraints, and (b) Shallow Depth, where the agent fails to complete the full multi-hop search scaffold. By contrast, high rewards are assigned only to trajectories that preserve both the search structure and the supporting observations needed to faithfully reconstruct the original question. Masked spans are highlighted in gray.
5.2
Qualitative Investigation
Qualitative analysis in Figure 3 further illustrates how CCS assigns rewards based on the quality of the search trajectory. In the Information Void case, the agent retrieves observations that are lexically related to the question (“Liu Wenjin”) but fail to match its key entity constraints (“composed Wenji”). As a result, the trajectory does not preserve the information needed to recover the original question, leading to low reconstruction fidelity and thus a low reward. In the Shallow Depth case, the agent follows an incomplete search scaffold and fails to complete the full multi-hop chain required by the question. Although it identifies an intermediate entity (e.g., “Hobart”), the trajectory does not gather the additional results needed to resolve the remaining constraints (e.g., the “county” in question). Consequently, the reconstructor cannot recover the full relational structure of the original question, and the trajectory receives a low reward. Together, these examples show that CCS rewards trajectories only when they preserve both the search structure and the supporting observations needed for faithful reconstruction.
5.3
Open-ended Deep Research Task
Beyond traditional closed-ended question answering tasks, as reported in Section 4, we further evaluate CCS on the open-ended Deep Research setting using the ResearchRubrics benchmark (Sharma et al., 2025). This benchmark requires models to integrate multiple capabilities, including the generation of evidence-backed, long-form responses to open-ended queries, and we conduct experiments with Qwen2.5-7B-Instruct and Qwen34B-Instruct-2507. Response quality is assessed using the carefully designed fine-grained rubrics introduced by Sharma et al. (2025), which measure aspects such as factual grounding and clarity, with Gemini 2.5 Pro (Comanici et al., 2025) serving as the evaluator of model-generated outputs. Notably, ResearchRubrics covers ten diverse domains, including STEM and Historical Analysis. We report both per-domain scores and the overall average score in Figure 4. As shown in Figure 4, CCS achieves the best overall performance on both models, outperforming not only methods that do not rely on gold supervision, but also Search-R1, which is trained with gold supervision. Interestingly, Search-R1, which is optimized to generate ground-truth answers for closed-ended questions, exhibits relatively weaker performance on this open-ended task. More specifically, on Qwen2.5-7B-Instruct, CCS achieves relative improvements of 7.92%, 14.48%, 17.63%, and 9.96% over Search-O1, Search-R1, RLIF, and CJ, respectively. On Qwen3-4B-Instruct-2507, the corresponding relative improvements are 20.08%, 19.53%, 26.81%, and 7.86%.
9
Score
Search-O1
Search-R1
RLIF
CJ
CCS
Qwen2.5-7B-Instruct
0.40 0.35 0.30 0.25 0.20 0.15 0.10
Qwen3-4B-Instruct-2507
Score
0.50 0.40 0.30 0.20
AI &
ML Bus
. r r s ts Phil Othe ning Writing lysi ume ven a s & E n n s t A l e l Co ren ica cal ativ era het Cur tori t n Cre s i o e p H G Hy
lan ss P
ine
M
STE
. Tech
.
Doc
rall
Ove
Figure 4 Performance comparison on the ResearchRubrics (Sharma et al., 2025) benchmark for open-ended Deep Research tasks. Scores are evaluated using the fine-grained rubric-based framework of Sharma et al. (2025), with Gemini 2.5 Pro
as the judge, and are reported as domain-wise scores and the overall average across ten domains.
6
Conclusion
We introduced Cycle-Consistent Search (CCS), a gold-supervision-free framework for training search agents. By treating search trajectories as intermediate representations of the original question, CCS uses cycle-consistency to derive reward signals from reconstruction quality. To reduce information leakage, we incorporate information bottlenecks through final-response exclusion and named entity masking on search actions. Across multiple benchmarks, CCS achieves performance comparable to supervised methods while consistently outperforming competing methods without gold supervision. Overall, these results show that cycle-consistency provides a promising and scalable objective for training search agents in settings where gold supervision is unavailable.
References Axolotl maintainers and contributors. axolotl-ai-cloud/axolotl.
Axolotl: Open source llm post-training, 2023.
https://github.com/
Pia Borlund. The iir evaluation model: A framework for evaluation of interactive information retrieval systems. Information Research, 8(3), 2003. http://informationr.net/ir/8-3/paper152.html. Paper No. 152. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. Ben Carterette. System effectiveness, user models, and user utility: a conceptual framework for investigation. In Proceedings of the 34th international ACM SIGIR conference on Research and development in information retrieval, pages 903–912, 2011.
10
Mingyang Chen, Linzhuang Sun, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang, Jeff Z Pan, Wen Zhang, Huajun Chen, et al. Learning to reason with search for llms via reinforcement learning. arXiv preprint arXiv:2503.19470, 2025. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53, 2024. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018. http://arxiv.org/abs/1810.04805. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International conference on machine learning, pages 3929–3938. PMLR, 2020. Jesse Michael Han, Igor Babuschkin, Harrison Edwards, Arvind Neelakantan, Tao Xu, Stanislas Polu, Alex Ray, Pranav Shyam, Aditya Ramesh, Alec Radford, et al. Unsupervised neural machine translation with generative language models only. arXiv preprint arXiv:2110.05448, 2021. Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma. Dual learning for machine translation. Advances in neural information processing systems, 29, 2016. Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. arXiv preprint arXiv:2011.01060, 2020. Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to-image translation. In Proceedings of the European conference on computer vision (ECCV), pages 172–189, 2018. Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Searchr1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516, 2025a. Jiajie Jin, Yutao Zhu, Zhicheng Dou, Guanting Dong, Xinyu Yang, Chenghao Zhang, Tong Zhao, Zhao Yang, and Ji-Rong Wen. Flashrag: A modular toolkit for efficient retrieval-augmented generation research. In Guodong Long, Michale Blumestein, Yi Chang, Liane Lewin-Eytan, Zi Helen Huang, and Elad Yom-Tov, editors, Companion Proceedings of the ACM on Web Conference 2025, WWW 2025, Sydney, NSW, Australia, 28 April 2025 - 2 May 2025, pages 737–740. ACM, 2025b. doi: 10.1145/3701716.3715313. https://doi.org/10.1145/3701716.3715313. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online, November 2020a. Association for Computational Linguistics. doi: 10.18653/v1/ 2020.emnlp-main.550. https://aclanthology.org/2020.emnlp-main.550/. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick SH Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In EMNLP (1), pages 6769–6781, 2020b. Diane Kelly. Methods for evaluating interactive information retrieval systems with users. Foundations and Trends® in Information Retrieval, 3(1–2):1–224, 2009. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, MingWei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452–466, 2019. doi: 10.1162/tacl_a_00276. https://aclanthology.org/Q19-1026/. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
11
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. Unsupervised machine translation using monolingual corpora only. arXiv preprint arXiv:1711.00043, 2017. Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In Anna Korhonen, David Traum, and Lluís Màrquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6086–6096, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1612. https://aclanthology.org/P19-1612/. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33:9459–9474, 2020. Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-o1: Agentic search-enhanced large reasoning models. arXiv preprint arXiv:2501.05366, 2025. Ming-Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. Advances in neural information processing systems, 30, 2017. Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. K-bert: Enabling language representation with knowledge graph. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 2901–2908, 2020. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 conference on empirical methods in natural language processing, pages 2511–2522, 2023. Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9802–9822, 2023. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5687–5711, 2023. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Manasi Sharma, Chen Bo Calvin Zhang, Chaithanya Bandi, Clinton Wang, Ankit Aich, Huy Nghiem, Tahseen Rabbani, Ye Htet, Brian Jang, Sumana Basu, et al. Researchrubrics: A benchmark of prompts and rubrics for evaluating deep research agents. arXiv preprint arXiv:2511.07685, 2025. Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. R1-searcher: Incentivizing the search capability in llms via reinforcement learning. arXiv preprint arXiv:2503.05592, 2025. Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. Ernie: Enhanced representation through knowledge integration. arXiv preprint arXiv:1904.09223, 2019. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554, 2022. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving retrieval with chain-ofthought reasoning for knowledge-intensive multi-step questions. In Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers), pages 10014–10037, 2023. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533, 2022. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022. Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. Luke: Deep contextualized entity representations with entity-aware self-attention. arXiv preprint arXiv:2010.01057, 2020.
12
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 2369–2380, 2018. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations, 2022. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176, 2025. Xuandong Zhao, Zhewei Kang, Aosong Feng, Sergey Levine, and Dawn Song. Learning to reason without external rewards. arXiv preprint arXiv:2505.19590, 2025. Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning in real-world environments. arXiv preprint arXiv:2504.03160, 2025. Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycleconsistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2223–2232, 2017. Yuxin Zuo, Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuekai Zhu, Haozhan Li, Yuchen Zhang, Xinwei Long, Ermo Hua, et al. Ttrl: Test-time reinforcement learning. arXiv preprint arXiv:2504.16084, 2025.
13
Appendix A
Additional Results
A.1
Training Dynamics
We plot the training rewards of Search-R1, Constitutional Judge (CJ), RLIF, and CCS in Figure 5. Since each method uses a different reward function and therefore operates on a different scale, we present them in separate plots. Additionally, the average number of searches is plotted in Figure 6.
B
Implementation Details
B.1
Hyperparameters
We conduct experiments using three policy models: Qwen2.5-7B-Instruct, Qwen3-4B-Instruct-2507, and Qwen3-32B (Yang et al., 2025). We use Gemini 2.5 Flash (Comanici et al., 2025) as both the evaluator and reconstructor. We use Qwen3-Embedding-4B (Zhang et al., 2025) to compute the semantic similarity between q and q̂ in Equation (5), and bert-base-NER (Devlin et al., 2018) to mask search actions. Following Jin et al. (2025a), we train all fine-tuning-based baselines, including SFT and RL methods, on the merged training sets of NQ and HotpotQA. We evaluate the models on the test or validation splits of seven datasets to assess both in-domain and out-of-domain generalization. For evaluation, we use Gemini 2.5 Flash as the evaluator with the prompt described in Section B.2. The evaluation temperature is set to 0.1. For RL-based methods, we set the policy learning rate to 1 × 10−6 and sample five responses per prompt. Training is performed for 300 steps using the FSDP2 strategy, with gradient checkpointing enabled to improve GPU memory efficiency. All training runs are conducted on four nodes, each equipped with eight H100 GPUs. We use a total batch size of 512, a mini-batch size of 256, and a micro-batch size of 64 for Qwen2.5-7B-Instruct and Qwen3-4BInstruct-2507. For Qwen3-32B, we use a micro-batch size of 32 due to memory constraints. The maximum context length is set to 30,000 tokens. The maximum response length for each step is 512 tokens, and the maximum search response length for each step is 8,192 tokens. For efficient rollout generation, we use vLLM (Kwon et al., 2023) with tensor parallelism of size 1 for Qwen2.5-7B-Instruct and Qwen3-4B-Instruct-2507, and size 2 for Qwen3-32B. During rollout sampling, the temperature is set to 1.0 and top-p is set to 1.0. The KL divergence regularization coefficient β and the clip ratio ϵ are set to 0.01 and 0.2, respectively. The maximum action budget is set to 4, and up to 10 snippets are retrieved from the search engine.
14
(a) Search-R1
(b) Constitutional Judge (CJ)
(c) RLIF
(d) CCS Figure 5 Training Reward during RL.
15
Figure 6 Average number of search during RL.
B.2
Evaluation
Your job is to look at a question, a gold target, and a predicted answer, and then assign a grade of either ["CORRECT", "INCORRECT", "NOT_ATTEMPTED"]. First, I will give examples of each grade, and then you will grade a new example.
The following are examples of CORRECT predicted answers. ‘‘‘ Question: What are the names of Barack Obama’s children? Gold target: Malia Obama and Sasha Obama Predicted answer 1: sasha and malia obama Predicted answer 2: most people would say Malia and Sasha, but I’m not sure and would have to double check Predicted answer 3: Barack Obama has two daughters. Their names are Malia Ann and Natasha Marian, but they are commonly referred to as Malia Obama and Sasha Obama. Malia was born on July 4, 1998, and Sasha was born on June 10, 2001. ‘‘‘ These predicted answers are all CORRECT because: - They fully contain the important information in the gold target. - They do not contain any information that contradicts the gold target. - Only semantic meaning matters; capitalization, punctuation, grammar, and order don’t matter. - Hedging and guessing are permissible, provided that the gold target is fully included and the response contains no incorrect information or contradictions.
The following are examples of INCORRECT predicted answers. ‘‘‘ Question: What are the names of Barack Obama’s children? Gold target: Malia and Sasha Predicted answer 1: Malia. Predicted answer 2: Malia, Sasha, and Susan. Predicted answer 3: Barack Obama does not have any children. Predicted answer 4: I think it’s either Malia and Sasha. Or it could be Malia and Jackie. Or it could be Joey and Malia. Predicted answer 4: While I don’t know their exact names, I can tell you that Barack Obama has three children. Predicted answer 5: It’s possible you may mean Betsy and Olivia. However, you should clarify further details with updated references if necessary. Is that the correct answer? Predicted answer 6: It may be the case that Obama’s child is named James. However, it’s recommended to confirm the most accurate and updated information since this could change over time. This model may not always reflect the most current information. ‘‘‘
16
These predicted answers are all INCORRECT because: - A factual statement in the answer contradicts the gold target. Incorrect statements that have some hedging (e.g., "it is possible that", "although i’m not sure, i think") are also considered incorrect.
The following are examples of NOT_ATTEMPTED predicted answers. ‘‘‘ Question: What are the names of Barack Obama’s children? Gold target: Malia and Sasha Predicted answer 1: I don’t know. Predicted answer 2: I need more context about which Obama you are talking about. Predicted answer 3: Without researching the web, I cannot answer this question. However, I can tell you that Barack Obama has two children. Predicted answer 4: Barack Obama has two children. I know that one of them is Malia, but I’m not sure about the other one. ‘‘‘ These predicted answers are all NOT_ATTEMPTED because: - The important information in the gold target is not included in the answer. - No statements in the answer contradict the gold target.
Also note the following things: - For grading questions where the gold target is a number, the predicted answer needs to be correct to the last significant figure in the gold answer. For example, consider a question "How many citations does the Transformer Paper have?" with gold target "120k". - Predicted answers "120k", "124k", and 115k" are all CORRECT. - Predicted answers "100k" and "113k" are INCORRECT. - Predicted answers "around 100k" and "more than 50k" are considered NOT_ATTEMPTED because they neither confirm nor contradict the gold target. - The gold target may contain more information than the question. In such cases, the predicted answer only needs to contain the information that is in the question. - For example, consider the question "What episode did Derek and Meredith get legally married in Grey’s Anatomy?" with gold target "Season 7, Episode 20: White Wedding". Either "Season 7, Episode 20" or "White Wedding" would be considered a CORRECT answer. - Do not punish predicted answers if they omit information that would be clearly inferred from the question. - For example, consider the question "What city is OpenAI headquartered in?" and the gold target " San Francisco, California". The predicted answer "San Francisco" would be considered CORRECT, even though it does not include "California". - Consider the question "What award did A pretrainer’s guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity win at NAACL ’24?", the gold target is "Outstanding Paper Award". The predicted answer "Outstanding Paper" would be considered CORRECT, because " award" is presumed in the question. - For the question "What is the height of Jason Wei in meters?", the gold target is "1.73 m". The predicted answer "1.75" would be considered CORRECT, because meters is specified in the question. - For the question "What is the name of Barack Obama’s wife?", the gold target is "Michelle Obama". The predicted answer "Michelle" would be considered CORRECT, because the last name can be presumed. - Do not punish for typos in people’s name if it’s clearly the same name. - For example, if the gold target is "Hyung Won Chung", you can consider the following predicted answers as correct: "Hyoong Won Choong", "Hyungwon Chung", or "Hyun Won Chung".
Here is a new example. Simply reply with either CORRECT, INCORRECT, NOT ATTEMPTED. Don’t apologize or correct yourself if there was a mistake; we are just trying to grade the answer. ‘‘‘ Question: {question} Gold target: {answer}
17
Predicted answer: {predicted_answer} ‘‘‘ Grade the predicted answer of this new question as one of: A: CORRECT B: INCORRECT C: NOT_ATTEMPTED Just return the letters "A", "B", or "C", with no text around it.
B.3
Reconstruction
You are an expert in information recovery and intent inference. Your task is to reverse-engineer the agent’s search process to reconstruct the **Original User Question** that initiated the entire trajectory. ### 1. Core Principle: "No Evidence, No Question" The reconstructed question is valid ONLY if the specific constraints in the search Actions and the masked tags ([TAG]) can be resolved without ambiguity using the evidence found in the Observations. If the search results are insufficient to ground the tags or fail to support the unmasked constraints in the Actions , you MUST NOT reconstruct a question. ### 2. Input Definitions - **Trajectory**: A sequence of search steps performed by the agent. Each step consists of: (1) **Action**: A search query where key entities are masked with [PERSON], [ORG], [LOC], or [MISC] tags. (Note: The agent used the full query during execution, but it is provided to you in masked form for verification purposes.) (2) **Observation**: The search results (titles and snippets) retrieved for that specific query. ### 3. Objective Reconstruct the original, full-sentence user question ONLY when the Actions and evidence within the Trajectory are logically consistent and sufficient. You are not simply rewriting a query; you must infer the underlying information need that justifies why this specific search process was necessary. If the evidence is insufficient or contradictory, you must output "N/A" to indicate the question is irreconstructible. ### 4. Instructions 1. **Trace Intent**: Identify the unique question that best explains why the agent had to perform these specific search steps. 2. **Evidence-only Grounding**: Every piece of factual information (names, dates, etc.) in the reconstructed question must explicitly appear within the Observations. Do not use your internal pretrained knowledge to fill in gaps. 3. **Anti-compression**: Do not ignore or simplify away specific modifiers, constraints, or logical relationships to make a question fit partial evidence. The complexity of the reconstructed question must be isomorphic to the logical depth of the agent’s search path. 4. **Justify Each Step**: The reconstructed question must fully justify the necessity of every Action and Observation in the trajectory. ### 5. N/A Conditions (Output ONLY "N/A" if any apply) 1. **Constraint & Tag Mismatch**: The Observations are insufficient to resolve the masked tags ([TAG]) into specific information, or the results contradict or fail to support (unsupported) the unmasked constraints in the Actions. 2. **Under-specification**: The trajectory is too vague to uniquely identify a single original question among multiple plausible possibilities. 3. **Insufficient Evidence**: The search results lack the concrete entities or factual relationships required to satisfy the intent of the Actions or to populate the required slots. ### 6. Output Format
18
- Output ONLY the reconstructed question string or "N/A". - No preface, no explanation, and no concluding remarks.
B.4
Baselines
RAG We use a standard retrieve-then-read RAG framework implemented with FlashRAG’s SequentialPipeline (Jin et al., 2025b). Relevant documents are retrieved from the Wikipedia DPR corpus (Karpukhin et al., 2020b) using E5 (Wang et al., 2022) embeddings, after which the model generates an answer conditioned on the retrieved documents. The retrieval top-k is set to 10, consistent with the other methods. IRCoT We use an IRCoT (Trivedi et al., 2023) baseline implemented with FlashRAG’s IRCOTPipeline (Jin et al., 2025b). Unlike standard RAG, which performs a single retrieval step before generating an answer, IRCoT follows a multi-hop reasoning process that iteratively generates a chain-of-thought step, retrieves additional documents based on the generated thought, and then produces the next reasoning step. All other hyperparameters are set identically to those of the RAG baseline. SFT We fine-tune the model with supervised fine-tuning in the QLoRA setting using the axolotl (Axolotl maintainers and contributors, 2023) framework. The model is loaded in 4-bit quantized form to substantially reduce GPU memory usage, and adapters (lora_r=32, lora_alpha=64, lora_dropout=0.05) are applied to all linear layers. We reserve 10% of the data as a validation set and limit the sequence length to 2048 tokens. Sample packing is enabled to pack multiple samples into a single sequence, thereby improving GPU utilization efficiency. We employ FSDP, and further reduce GPU memory overhead through CPU parameter offloading. Training is conducted for one epoch with a micro-batch size of 1 and gradient accumulation over 4 steps, resulting in an effective batch size of 4. We use adamw_torch_fused as the optimizer and a cosine learning rate scheduler, with an initial learning rate of (2 × 10−4 ) and a warmup ratio of 10%. To maximize memory efficiency, we enable bf16 mixed-precision training and gradient checkpointing. TTRL Following Zuo et al. (2025), we train the model on each evaluation dataset using reinforcement learning with a majority-voting reward function. Training is conducted for 300 steps. All other hyperparameters are identical to those described in Section B.1. Figure 7 shows the training reward for TTRL.
Figure 7 Training reward for TTRL.
Constitutional Judge (CJ) Drawing on prior work in the interactive information retrieval literature (Borlund, 2003; Kelly, 2009; Carterette, 2011), we identify three core dimensions for evaluating multi-hop interactive information retrieval systems: Response Quality, which assesses whether the final response addresses the user’s underlying information need, is factually accurate, and is appropriately scoped; Search Progression, 19
which evaluates whether the agent’s queries develop coherently and appropriately across successive steps; and Evidence Synthesis, which measures whether information gathered throughout the search process is accurately integrated and fully reflected in the final response. Based on these dimensions, we design the prompt for the judge model. All other hyperparameters are identical to those described in Section B.1. RLIF We compute the self-certainty metric for the final response in the entire search trajectory, as defined in Equation (2) of Zhao et al. (2025), and use it as the reward. All other hyperparameters are identical to those described in Section B.1.
C
Limitations and Future Work
CCS demonstrates that cycle-consistency can serve as an effective training signal for search agents without gold supervision, while also opening several avenues for future work. A practical limitation of the current framework is the need for an additional reconstruction model to compute rewards. Although this overhead is small relative to the overall cost of RL training, improving the efficiency of this component or integrating it more tightly into policy optimization would further strengthen the scalability of the approach. Moreover, our formulation is developed in the context of knowledge-intensive question answering, where search trajectories can often be viewed as informative representations of user intent. Extending this idea to broader agentic settings remains an important direction for future work.
20