arXiv CSopen access

$π$-Play: Multi-Agent Self-Play via Privileged Self-Distillation without External Data

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

machine learning, deep learning, neural networks

arXiv:2604.14054v1 [cs.LG] 15 Apr 2026

π-Play: Multi-Agent Self-Play via Privileged Self-Distillation without External Data Yaocheng Zhang∗ 1,2 , Yuanheng Zhu† 1,4 , Wenyue Chong1,2 , Songjun Tu1,4 , Qichao Zhang1,4 , Jiajun Chai3 , Xiaohan Wang3 , Wei Lin3 , Guojun Yin3 , Dongbin Zhao1,4 1 Institute of Automation, Chinese Academy of Sciences 2 School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences 3 Meituan 4 School of Artificial Intelligence, University of Chinese Academy of Sciences {zhangyaocheng2023,yuanheng.zhu}@ia.ac.cn

Abstract Deep search agents have emerged as a promising paradigm for addressing complex information-seeking tasks, but their training remains challenging due to sparse rewards, weak credit assignment, and limited labeled data. Self-play offers a scalable route to reduce data dependence, but conventional self-play optimizes students only through sparse outcome rewards, leading to low learning efficiency. In this work, we observe that self-play naturally produces a question construction path (QCP) during task generation, an intermediate artifact that captures the reverse solution process. This reveals a new source of privileged information for self-distillation: self-play can itself provide high-quality privileged context for the teacher model in a low-cost and scalable manner, without relying on human feedback or curated privileged information. Leveraging this insight, we propose Privileged Information Self-Play (π-Play), a multi-agent self-evolution framework. In π-Play, an examiner generates tasks together with their QCPs, and a teacher model leverages QCP as privileged context to densely supervise a student via self-distillation. This design transforms conventional sparse-reward self-play into a dense-feedback self-evolution loop. Extensive experiments show that data-free πPlay surpasses fully supervised search agents and improves evolutionary efficiency by 2–3× over conventional self-play.

Introduction

Deep search agents leverage the reasoning capabilities of large language models (LLMs) and external search engines to perform multi-turn retrieval and analysis for complex questions, emerging as a promising paradigm for information acquisition [28, 9, 49]. Recent advances have shown that reinforcement learning (RL) can substantially improve both reasoning and search behaviors, enabling LLM agents to tackle increasingly challenging information-seeking tasks [28, 9]. However, training strong search agents at scale remains fundamentally bottlenecked by data [45, 13, 37, 17, 6]. Supervised pipelines rely on labeled data and costly expert trajectories, while outcome-supervised RL often suffers from sparse rewards and poor credit assignment, especially in multi-turn search scenarios [13, 46, 33, 36, 47, 43]. A promising direction for alleviating data dependence is self-evolution through self-play. In existing self-play frameworks, models of the same scale alternately play the roles of examiner and student: the examiner autonomously constructs training tasks, while the student learns by solving those tasks ∗ Work primarily done during an internship at Meituan. † Corresponding author

Preprint.

QA合成路径的示意图以及合成过程

Question Construction Path (QCP) Question

𝑹𝒂𝒏𝒅𝒐𝒎 𝑫𝒐𝒄𝒖𝒎𝒆𝒏𝒕

... Alan Turing was an English mathematician, computer scientist …

𝑫𝒐𝒄𝒖𝒎𝒆𝒏𝒕 𝟏

𝑫𝒐𝒄𝒖𝒎𝒆𝒏𝒕 𝟐

𝑫𝒐𝒄𝒖𝒎𝒆𝒏𝒕 𝒏

... Born in London, Turing was raised in southern England ....

... The 2012 Summer Olympics, also known as London 2012, were held from 27 July to 12 August 2012 in London ...

... He graduated from King's College, Cambridge, and in 1938, earned a doctorate degree from Princeton University ...

Query 𝟏

Examiner

Privileged Information (Question Construction Path)

Query 𝟐

Teacher

Which mathematician, born in the European city that hosted the 2012 Summer Olympics, later studied at Cambridge and earned a Princeton PhD in 1938?

Answer

Query 𝒏

𝝅𝑻𝝍 (·｜𝒒, 𝒄, y<𝒕 )

Alan Turing

Self-Distillation 𝔻𝐃𝐢𝐬𝐭𝐢𝐥𝐥 (𝝅𝑺𝜽 |𝐬𝐭𝐨𝐩𝐠𝐫𝐚𝐝[𝝅𝑻𝝍 ])

𝝅𝑺𝜽 (·｜𝒒, y<𝒕 )

Student

Figure 1: Overview of QCP-guided self-distillation in π-Play. The examiner is equipped with search tools and interacts with the search engine to obtain factual information, ensuring the correctness of both the synthesized QA pairs and their construction paths c. The teacher policy πψT leverages QCP as additional context to provide token-level supervision to the student policy πθS along the student’s rollout y, by minimizing the per-token reverse KL divergence DDistill πθS ∥ stopgrad[πψT ] (Eq. 9). [45, 11, 20, 23, 4]. This paradigm allows the model to bootstrap its own curriculum without relying on manually collected training data, offering a promising route toward scalable and autonomous improvement. Despite its strengths, self-play still suffers from a critical limitation. The student is typically optimized only through sparse outcome rewards, which makes learning inefficient for multiturn search tasks [45]. Notably, self-play produces more than just the final training question-answer (QA) pair (q, o⋆ ). As shown in Fig. 1, self-play also naturally produces a high-quality yet previously overlooked intermediate artifact, the question construction path (QCP), denoted by c, which captures the multi-turn interaction process by which the examiner constructs the question through iterative search. Hence, the examiner’s output is more accurately represented as a triplet (q, c, o⋆ ), where the QCP records a reverse solution process from the answer back to the question. Rather than being merely an intermediate artifact, the QCP serves as a form of intrinsic privileged information for dense supervision. Unfortunately, existing self-play methods largely ignore this signal, since they cannot directly use it for supervised fine-tuning (SFT) [45, 11, 20, 5], and thus fail to exploit it as a valuable source of dense supervision.

Avg Performance Iter1

Accuracy

Another line of self-evolution, self-distillation, addresses the credit assignment problem by employing high-quality privileged information [12, 29, 48, 41, 24]. Unlike on-policy distillation[1, 8, 44, 21], which relies on a larger external teacher, self-distillation uses a teacher model of the same scale as the student and augments it with privileged information to provide token-level dense supervision for the student. Common sources of such privileged information include expert demonstrations [24, 29], external (human) feedback [12, 35], and prior knowledge [41, 26]. Prior studies have shown that such privileged supervision can significantly enhance learning efficiency. However, obtaining high-quality privileged information is often nontrivial. In several prior works, privileged information is typically constructed with the help of human experts or stronger models [29, 48, 41]. Furthermore, self-distillation typically relies on training data consisting of QA pairs during optimization. This dependence on both high-quality privileged information and curated training data makes self-distillation difficult to scale efficiently.

Iter2

Iter3

3× Efficient

-Play

Self-Play 0

100 120 140

Training Step

Figure 2: π-Play outperforms self-play methods (Dr.Zero [45]) across seven QA benchmarks with Qwen3-4B-Instruct2507. A single iteration of π-Play achieves gains that match or even exceed those of three iterations of self-play, demonstrating its superior evolutionary efficiency.

This work is motivated by a key observation: self-play naturally produces intrinsic privileged information that can be exploited for self-distillation. Since QCP records how the question is 2

Self-Play

𝝅-Play (Ours)

Sparse reward for student

Employ Question Construction Path to generate dense supervision

Ignore the Question Construction Path

No need for external labeled data and privileged information

tion ruc nst n Co h io t t s Pa Self-distillation Que Soft Update Guidance

Difficulty Feedback Update with Difficulty

Update with Sparse Outcome Reward

Self-Distillation

Update with Difficulty

Rely on labeled data pairs

Dif

fi c ult

Rely on high-quality privileged information

Priv

ed ileg

Inf

Update with Outcome Reward & Teacher Guidance

n atio orm

Self-distillation Guidance

yF eed bac k

Self-Play

𝝅-Play

Soft Update

Expert

Examiner

QA Pair

Teacher

Student

Performance

Evolutionary Efficiency

Figure 3: Comparison of π-Play with other self-evolution frameworks. All models (examiner, teacher, and student) in π-Play are initialized from the same base LLM and function as search agents. π-Play uses alternating optimization to evolve multiple agents in a closed loop. Compared to self-play, it overcomes the sparse-reward problem of the student and enables the student to be optimized under the joint effect of outcome rewards and teacher guidance. constructed from factual evidence, it provides privileged context that can help a same-scale teacher to generate more accurate rollouts than a student conditioned only on the question. Based on this insight, we propose Privileged Information Self-Play (π-Play), a multi-agent self-evolution framework in which an examiner first generates training tuples (q, c, o⋆ ), and a teacher model conditioned on the construction path c then provides token-level distillation signals to guide the student (Fig. 1). Through this mechanism, π-Play improves search and reasoning capability while transforming sparse-reward self-play into a dense-feedback self-evolution loop. Through multiple efficient training iterations guided by the teacher, π-Play surpasses supervised baselines and conventional self-play baselines and exhibits stronger evolutionary efficiency (Fig.2). In summary, our main contributions are as follows: ◦ We reveal a new source of privileged information for self-distillation. QCP is tightly aligned with the training question and approximates a reverse solution path, making it effective privileged context for enhancing the teacher model. ◦ We show that self-play naturally generates QCPs during task construction in a low-cost and scalable manner. This enables QCP-guided self-distillation without relying on human feedback or curated privileged information. ◦ We propose Privileged Information Self-Play (π-Play), a novel multi-agent self-evolution framework with dense feedback, in which QCP bridges the examiner, teacher, and student, enabling their efficient co-evolution. ◦ Extensive experiments demonstrate that π-Play improves final performance and achieves 2–3× higher evolutionary efficiency than conventional self-play.

Related Work

2.1

Deep Search Agents

Deep search agents leverage the reasoning capabilities of large language models and external search engines to perform multi-turn retrieval and analysis for complex questions, emerging as a promising paradigm for information acquisition. Recent work has leveraged RL to further enhance both reasoning and access to up-to-date knowledge, enabling LLMs to tackle complex tasks more effectively [28, 9]. Some agentic RL works, including Search-R1 [13], R1-Searcher [30], DeepResearcher [49], and ZeroSearch [31], further enhance question-answering capabilities but remain constrained by limited training data. To scale agentic RL, some pipelines [37, 17, 6] employ offline question3

synthesis strategies, yet they do not explicitly couple task generation with the evolving capability of the solver. In contrast, self-play enables search agents to jointly generate and solve tasks without human annotations, reducing reliance on manual supervision and allowing agentic RL to scale to broader scenarios [11, 20, 45]. 2.2

Self-play for LLMs

Self-play enables LLMs to autonomously improve their reasoning and problem-solving capabilities by iteratively generating tasks and learning from their own experiences [18, 11, 20, 19, 45]. Early approaches leverage the LLM as both generator and evaluator, refining its policy without human supervision [23, 4]. For instance, self-rewarding LLMs employ iterative training loops where the model judges its own outputs to construct preference data for optimization [42]. More recent frameworks, such as R-Zero [11], SSP [20], and Dr.Zero [45], typically involve only two roles: an examiner, which generates QA pairs, and a student, which is optimized via outcome-based RL on these self-generated tasks. These methods typically rely on sparse outcome rewards and do not exploit the construction paths produced during question generation. Such coarse-grained feedback makes it difficult to distinguish effective from ineffective tool-use behaviors, leading to inefficient credit assignment and slow policy improvement. In contrast, π-Play incorporates these construction paths into student optimization as token-level supervision. 2.3

Privileged Information Self-distillation for LLMs

Self-distillation is a training paradigm that enables a student model to improve by learning from its own generated outputs. In this process, a teacher model evaluates students’ rollouts and provides supervision signals at the token level to guide students. Unlike on-policy distillation [1, 8, 44, 21], self-distillation does not rely on a larger external teacher. Instead, the teacher typically shares the same architecture and scale as the student, but is augmented with privileged information to provide reliable supervision [12, 29, 48, 41, 24]. Such dense supervision has been shown to effectively enhance the student model’s learning efficiency [12, 29, 41, 24]. However, obtaining high-quality privileged information is often nontrivial. In several prior works, privileged information is typically constructed with the help of human experts or stronger models [29, 48, 41], including expert demonstrations [24, 29] and prior knowledge [41, 26], which limits the scalability of self-distillation. In π-Play , the privileged signal provided to the teacher model is derived from the task-construction process in self-play, thereby avoiding dependence on externally provided privileged information.

Methods

3.1

Self-Play with Privileged Information

As shown in Fig. 3, we employ a self-evolution framework based on efficient multi-agent collaboration, in which all models function as search agents capable of leveraging external knowledge. Let y denote a full response (or rollout), and o(y) denote the final answer extracted from y; for brevity, we write oi := o(yi ). Under this notation, The examiner πϕE , the teacher πψT , and the student πθS are each optimized according to their respective objectives: Optimization Objective ◦ Examiner (Sec. 3.2): Learn to generate diverse and challenging questions. max E(q,c,o⋆ )∼πϕE (·),{yk }nk=1 ∼πθS (·|q) [rd (o⋆ , {ok }nk=1 )], ϕ

(1)

◦ Teacher (Sec. 3.4): Incorporate privileged information to generate more accurate rollouts while avoiding an excessive discrepancy between the teacher and the student. T max E(q,c,o⋆ )∼πE (·),y∼πT (·|q,c) [I(o = o⋆ ) − βDKL πψ (· | q, c) ∥ stopgrad[πθS (· | q)])], ψ

(2)

◦ Student (Sec. 3.3): Learns efficiently through a combination of outcome reward and teacher guidance, thereby enhancing its evolutionary efficiency. T max E(q,c,o⋆ )∼πE (·),y∼πS (·|q) [I(o = o⋆ ) − λDDistill πθS (· | q) ∥ stopgrad[πψ (· | q, c)])], (3) ϕ θ θ {z } | {z } | Teacher guidance

Learning from outcome reward

where rd denotes the difficulty reward and I is the indicator function. To generate questions of moderate difficulty (i.e., suitable for the student’s current capability), the examiner’s reward rd is defined over the distribution of predicted answers. If all predictions (i.e., {ok }nk=1 ) are correct, the question is considered trivial, whereas if none are correct, the question is likely too difficult for the student. To jointly optimize the examiner, teacher, and student, we adopt an alternating optimization loop that couples question generation, teacher guidance, and student improvement into a unified self-evolution process. As the student becomes stronger, it drives the examiner to generate increasingly challenging questions. Meanwhile, the student’s updated behavior is softly propagated to the teacher, allowing the teacher to track the student while remaining more stable. This iterative process establishes a continuously evolving curriculum. All three agents are initialized from the same base LLM and rely exclusively on the search tool to access external knowledge. Following prior work [45], we strictly adhere to a training data-free setting, avoiding any demonstrations, questions, or annotated answers from external sources or human experts. 3.2

Examiner Training

To utilize student feedback for training the examiner, we employ a difficulty reward function rd that encourages both verifiability (the task must be solvable) and difficulty (the task must not be trivial) [45]. Specifically, we leverage the student’s success rate on the generated questions as a proxy for these properties. Let k denote the number of correct solutions out of n sampled attempts; we penalize cases where the student either fails completely (k = 0) or succeeds trivially (k = n), thereby encouraging the examiner to generate questions of moderate difficulty. The reward is defined as: rd (o⋆ , {oi }ni=1 ) = I(0 < k < n)

n X n−k I(oi = o⋆ ), + rf , with k = n−1 i=1

(4)

The difficulty reward is maximized when exactly one solution is correct and decays linearly as the number of correct predictions increases. Here, rf denotes a format reward that encourages the examiner to interleave reasoning with search during task generation, so that the synthesized questions are both factually grounded and accompanied by an informative construction path. These paths serve as privileged information, enabling the teacher model to provide guidance to the student model. Following prior work, we employ hop-grouped relative policy optimization [45] to train the examiner. Specifically, we estimate advantages by grouping structurally similar questions within a batch. Generated QA pairs are clustered according to their cross-hop complexity, measured by the number of hops h ∈ H. Intuitively, questions with fewer hops are typically simpler, whereas higher-hop questions demand extensive search and multi-turn reasoning. This hop-specific normalization of returns produces low-variance advantage estimates while avoiding the computational cost of sampling multiple candidate questions per prompt. " JExaminer (ϕ) = E{(qi ,ci ,o⋆ )∼πE (·),{yi,k }n i

S N k=1 ∼πθ (·|qi )}i=1

# 1 X X E E log πϕ Ai,h − βDKL (πϕ ∥πref ) , N i∈I h∈H

(5)

where N denotes the batch size and β controls the strength of the KL regularization. The advantage of each generated QA triplet (qi , ci , o⋆i ) is obtained by hop-wise reward normalization: rid − Ej∈Ih [rjd ] Ai,h = q Varj∈Ih [rjd ] + δ

(6)

where Ih is the set of questions in the hop group h, and δ is a small constant for numerical stability. 3.3

Student Training

For student training, we sample training tuples (q, c, o⋆ ) from the examiner policy πϕE . The student policy πθS then generates candidate rollouts for each question and is optimized using a hybrid training objective. Importantly, the construction path c is only visible to the teacher model, ensuring that the teacher can produce reliable guidance while the student learns to solve the task without direct access 5

Algorithm 1 The Evolution Process of π-Play Require: Examiner model πϕE , Teacher model πψT , Student model πθS , Base LLM πref E 1: Initialize πϕ , πψT , and πθS from the Base LLM πref

2: for iteration j ← 1 to M do 3: # Training Examiner Model 4: for step k ← 1 to W do E 5: Sample N QA triplets {(qi , ci , o⋆i )}N i=1 from the examiner policy πϕ 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18:

Update πϕE (·) with Eq. (5) end for # Generate Training Data For Student Model E D Generate D = {(qi , ci , o⋆i )}N i=1 from the examiner policy πϕ (·)

# Training Student&Teacher Model for step k ← 1 to W do Sample a batch Db from D for each QA triplets (q, c, o⋆ ) in Db do S Sample G rollouts {yi }G i=1 from the student policy πθ (·|q) end for Update πθS with Eq. (7) by teacher guidance; Then Soft Update πψT with Eq. (10) 19: end for 20: end for 21: return Student model πθS

to privileged information. Formally, the student objective is defined as follows: " Jπ-play (θ) = E(q,c,o⋆ )∼πE ,{yi }G ϕ

S i=1 ∼πθ (·|q)

|yi |

1 XX T Li,t − β DKL (πθS ∥πref ) − λDDistill πθS ∥ πψ ) G i=1 t=1 | {z } | {z } Teacher guidance

# ,

Learning from outcome reward

(7) !

Li,t = min

πθS (yi,t | q, yi,<t ) πθS (yi,t | q, yi,<t ) Ai , clip , 1 − ϵ, 1 + ϵ Ai S πθold (yi,t | q, yi,<t ) πθSold (yi,t | q, yi,<t )

! ,

(8)

r e −Ej∈G [r e ]

where Ai = iVarj∈G [re ]j . The overall objective consists of two complementary components. The first j term corresponds to group relative policy optimization (GRPO), which improves the student policy using outcome rewards derived from answer correctness (i.e., re (q, oi ) = I(oi = o⋆ )). By computing advantages from group statistics, GRPO reinforces successful rollouts while penalizing failed ones. The second term is a self-distillation objective, where the student policy is aligned with the teacher policy πψT along the student’s rollout. The distillation loss is defined as follows: DDistill πθS ∥ πψT ) =

|yi | 1 X KL(πθS (· | q, yi,<t )∥stopgrad(πψT (· | q, c, yi,<t ))) |yi | t=1

(9)

The full formulation of the distillation loss is provided in Appendix C. Since the teacher has access to privileged information in the form of the QCP, it can provide reliable token-level guidance. This dense supervision complements sparse outcome rewards through a favorable bias-variance trade-off [27, 7]: outcome rewards are unbiased but high-variance, whereas teacher guidance may introduce modest bias while substantially reducing variance. Their combination enables more efficient credit assignment and faster policy improvement. 3.4

Teacher Updating

Although Eq. (2) defines the ideal teacher objective, directly optimizing it would introduce additional computational overhead. In practice, to provide stable teacher supervision while allowing the teacher 6

Table 1: The main results of π-Play. Bold value indicates the top-performing result, while underline value denote the second-best. Through efficient teacher guidance, π-Play consistently outperforms both supervised and self-play search agents. General QA NQ

TriviaQA

Multi-Hop QA

PopQA

HotpotQA

2WikiMQA

MuSiQue

Bamboogle

Total

1.6 10.5 10.8 7.8 11.2

8.8 37.6 29.6 31.2 35.2

68.4 239.7 246.6 245.2 250.0

6.3 12.9 10.2 14.4 13.4

27.2 50.0 35.2 36.8 44.0

166.6 266.0 249.9 266.3 277.2

7.4 9.8 37.8 9.9 13.1 12.4

28.8 37.6 48.0 32.0 40.0 40.0

149.1 242.8 248.8 274.9 280.3

Qwen3-4B ReAct Search-R1 SQLM* Dr.Zero π-Play

7.2 35.2 39.9 40.4 38.1

16.6 57.3 60.4 61.2 58.4

10.3 39.5 42.1 44.3 42.3

11.2 30.7 33.0 31.0 32.3

12.7 28.9 30.8 29.3 32.5

Qwen3-4B-Instruct-2507 ReAct Search-R1 SQLM* Dr.Zero π-Play

18.4 39.5 39.0 41.3 41.8

45.5 62.6 62.2 61.6 63.2

28.4 41.3 41.9 44.0 44.0

22.8 34.9 34.1 38.4 38.5

ReAct Search-R1 ToolForge SQLM* Dr.Zero π-Play

15.9 35.7 30.3 38.9 42.1 43.0

37.3 59.9 16.8 60.5 62.5 64.6

14.9 40.0 35.2 42.7 45.8 47.2

22.3 32.0 33.1 36.6 38.9

18.0 24.8 27.3 29.8 32.3 Qwen3-8B 22.5 27.8 31.7 34.8 34.2

to co-evolve with the student, we approximate this objective by updating the teacher parameters as an exponential moving average (EMA) of the student parameters [12, 24]: ψ ← (1 − τ )ψ + τ θ, τ ∈ (0, 1), (10) where τ controls the teacher update rate. This update keeps the teacher relatively stable while enabling it to gradually track the student’s improvements over training. In all experiments, we set τ = 0.05. 3.5

The Co-Evolution Procedure of π-Play

In summary, we present π-Play, a data-free self-evolution framework that jointly optimizes the examiner, teacher, and student models (Fig. 3). In each iteration, the examiner generates QA pairs together with QCP, and is trained via student-derived difficulty feedback to produce challenging yet solvable questions. The student improves its search and reasoning abilities through training on QA pairs generated by examiner, while the teacher leverages the QCP as additional context to provide token-level guidance to the student. After each update, the teacher is softly aligned with the student, enabling both models to co-evolve. This alternating optimization loop forms a symbiotic feedback cycle in which stronger students drive the examiner toward more challenging questions, while teacher guidance accelerates student improvement. The training process is summarized in Algorithm 1.

Experiment

4.1

Setup

Datasets & Models. We conduct experiments on three models from the Qwen-3 series [38]. The base models are Qwen3-4B, Qwen3-4B-Instruct-2507, and Qwen3-8B. More details on experimental setups can be found in Appendix A. We evaluate π-Play primarily on three one-hop benchmarks NQ [16], TriviaQA [14], PopQA [22], as well as four multi-hop QA benchmarks, including HotpotQA [39], 2WikiMQA [10], MuSiQue [32], and Bamboogle [25]. 7

Table 2: Learning dynamics of π-Play with increasing iterations. General QA

Multi-Hop QA

Methods

TriviaQA

PopQA

HotpotQA

2WikiMQA

MuSiQue

Bamboogle

Total

Qwen3-4B Dr.Zero Iter1 π-Play

36.1 38.6

58.7 59.4

42.8 43.5

28.6 32.2

25.9 31.4

6.2 10.7

21.6 32.8

219.9 248.6

Iter2

Dr.Zero π-Play

40.6 38.1

60.5 58.4

44.4 42.3

30.8 32.3

29.1 32.5

8.0 11.2

28.8 35.2

242.2 250.0

Iter3

Dr.Zero π-Play

40.4 41.1

61.2 62.0

44.3 44.8

31.0 32.5

29.3 31.5

7.8 9.0

31.2 28.8

245.2 249.7

Qwen3-4B-Instruct-2507 Dr.Zero 39.8 Iter1 π-Play 40.7

61.7 62.1

42.6 42.9

35.4 36.7

27.4 30.1

11.7 13.2

36.0 39.2

254.6 264.9

Iter2

Dr.Zero π-Play

39.3 41.2

61.7 62.5

43.2 43.5

38.2 37.8

28.4 32.5

14.2 14.2

38.4 40.0

263.4 271.7

Iter3

Dr.Zero π-Play

41.3 41.8

61.6 63.2

44.0 44.0

38.4 38.5

29.8 32.3

14.4 13.4

36.8 44.0

266.3 277.2

Qwen3-8B Dr.Zero Iter1 π-Play

36.7 38.2

59.5 59.7

42.9 43.4

32.1 34.1

26.9 33.2

9.5 10.9

18.4 35.2

226.0 254.7

Iter2

Dr.Zero π-Play

36.7 42.9

60.0 63.2

43.2 46.4

36.6 37.7

35.4 34.7

12.7 12.4

38.4 36.0

263.0 273.3

Iter3

Dr.Zero π-Play

42.1 43.0

62.5 64.6

45.8 47.2

36.6 38.9

34.8 34.2

13.1 12.4

40.0 40.0

274.9 280.3

Baseline & Evaluation. To demonstrate the efficacy of π-Play, we compare it against a variety of baseline search agents: (1) training-free: ReAct [40]; (2) supervised RL: Search-R1 [13] and ToolForge [2], and (3) self-play: Dr.Zero [45] and SQLM* [3, 45]. All models are evaluated using exact match scores with identical search engine (E5-base [34]) and corpus settings (English Wikipedia dump [15]), using the checkpoint from their best-performing iteration (step). 4.2

Main Results

We first analyze the main evaluation results as reported in Table 1 and derive several key observations from them: (1) Strong Overall Performance: π-Play achieves substantial improvements in overall performance over base LLM (e.g., ReAct), demonstrating strong robustness and generalization across diverse task types and model scales. The results further demonstrate the effectiveness and superiority of the multi-agent self-evolution framework for search agents. (2) π-Play consistently surpasses supervised RL methods. π-Play delivers strong performance without using any training data. In terms of average performance, it surpasses the Search-R1 by 6.2%, 5.2%, and 14.5% on Qwen3-4B, Qwen3-4B-Instruct-2507, and Qwen3-8B, respectively. This self-evolution framework demonstrates greater performance gains when instantiated with a stronger base LLM. (3) π-Play consistently surpasses self-play methods. π-Play outperforms the self-play methods (SQLM* and Dr.Zero) across multiple model scales, benefiting from the additional guidance provided by the teacher to the student. Notably, the performance gains are even more substantial on multi-hop benchmarks, where complex multi-step reasoning is required. We attribute this advantage to the teacher model’s token-level credit assignment, which provides finer-grained and more effective supervision for long-horizon reasoning. 4.3

Training Dynamics

To better understand the self-evolving dynamics of π-Play, we analyze model performance across training iterations, with detailed comparisons with Dr.Zero summarized in Table 2 and Fig. 2. These results lead to several key observations: (1) Across all three iterations, the student model shows a steady upward performance trend, consistently outperforming Dr.Zero after every iteration. This 8

Student Reward

Examiner Reward

Iteration 2

Examiner Entropy

Iteration 3

0.6

0.3

0.4

0.2

0.1

Entropy

Reward

Iteration 1

Student Entropy

100

Step

150

Figure 4: Iterative reward and entropy dynamics of the examiner and student in π-Play with Qwen3-4B-Instruct-2507. Both reward and entropy reach a converged state by Iteration 3. Table 3: Ablation study on the question construction path (QCP). π-Play w/o Distillation denotes the variant of π-Play without teacher-guided distillation loss. Variants of the form π-Play w/ [Privileged Info] replace QCP with different forms of privileged information for self-distillation. General QA

π-Play w/o Distillation π-Play w/ GT π-Play w/ GT+Hop π-Play w/ Partial QCP π-Play w/ QCP (Ours)

Multi-Hop QA

TriviaQA

PopQA

HotpotQA

2WikiMQA

MuSiQue

Bamboogle

Total

41.2 42.0 42.1 41.6 41.8

62.3 62.0 61.9 62.1 63.2

43.6 44.5 42.9 43.3 44.0

37.7 37.7 37.7 38.3 38.5

29.0 29.4 29.1 31.0 32.3

13.3 13.3 14.0 13.6 13.4

38.4 35.2 39.2 43.2 44.0

265.5 264.1 266.9 273.1 277.2

highlights the effectiveness of the examiner–teacher–student interplay and suggests that collaborative self-evolution is more effective than standard self-play by enabling more efficient information sharing among agents. (2) After the first training iteration, the student in π-Play achieves substantial gains in both search and reasoning abilities. In terms of performance, it already matches or even surpasses the converged performance of Dr.Zero after three iterations, demonstrating the superior evolutionary efficiency. (3) After the second iteration, the improvement trends begin to vary across model sizes. While Qwen3-4B-Instruct-2507 and Qwen3-8B continue to show modest gains, Qwen3-4B drops slightly from 250 to 249.7, suggesting that performance has started to plateau. Beyond this point, further iterations bring only marginal or no additional improvements across model sizes. In summary, these dynamics validate the design of π-Play, demonstrating that the introduction of the privileged teacher brings significant benefits.

Further Analysis

In this section, we analyze the importance of the QCP and the co-evolution of the examiner. Further experiments, including extensive ablation studies, training cost, search behavior analysis (search count and query redundancy), and case analyses, are provided in Appendix B. 5.1

Ablation Study on the Question Construction Path

As shown in Table 3, we further evaluate the effectiveness of QCP as privileged information by comparing it with several alternative forms of teacher-side privileged context, including the groundtruth answer alone (π-Play w/ GT), the ground-truth answer combined with the question hop count (π-Play w/ GT+HOP), and a partial QCP obtained by randomly truncating half of the original QCP (π-Play w/ Partial QCP). Among these variants, using GT yields the worst performance, which is nearly comparable to the variant without a teacher model (π-Play w/o Distillation). This is because GT contains only the final answer and provides little information about the underlying logic used to construct the question, making it difficult for the teacher to provide effective guidance to the student. In contrast, the full QCP achieves the best performance, while Partial QCP performs second best. 9

Moreover, Partial QCP still substantially outperforms both GT and GT+HOP, further highlighting the effectiveness of QCP as privileged information. 5.2

Evolution of Question Difficulty

To understand the co-evolutionary dynamic of the examiner, we examined how the tasks it generated changed across iterations. After each of the training iterations, we sampled 2000 questions from its policy, creating three distinct evaluation sets: Dstep 50 , Dstep 100 , and Dstep 150 . The results show that the examiner generates progressively more challenging questions as training proceeds. This is evidenced by the performance of a fixed solver on these evolving question sets: for instance, the static student (Step 50) drops from 57.1 on Dstep 50 to 45.1 on Dstep 150 . This suggests that the examiner successfully increases task difficulty over the course of training. Table 4: Co-evolutionary dynamic of the Examiner Performance of Evaluated Model

Dstep 50 Dstep 100 Dstep 150

Base Model

Student(step 50)

Student(step 100)

27.5 27.0 22.9

57.1 51.9 45.1

58.4 57.3 53.3

Conclusion

We propose π-Play, a novel self-evolution framework that bridges the gap between self-play and selfdistillation for search agents. Our key insight is that the QCP generated during self-play constitutes an intrinsic form of privileged information that can be transformed into dense token-level supervision through self-distillation. This design transforms sparse-reward self-play into a dense-feedback selfevolution paradigm. Extensive experiments show that π-Play improves both final task performance and evolutionary efficiency compared with conventional self-play. In the current work, we focus on search tasks as a testbed to validate the feasibility of π-Play. An important direction for future work is to extend this self-evolution framework to broader domains, including math and code agents.

References [1] Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In The Twelfth International Conference on Learning Representations, pages 1–18, Vienna, Austria, 2024. OpenReview.net. [2] Hao Chen, Zhexin Hu, Jiajun Chai, Haocheng Yang, Hang He, Xiaohan Wang, Wei Lin, Luhang Wang, Guojun Yin, and Zhuofeng zhao. Toolforge: A data synthesis pipeline for multi-hop search without real-world apis. arXiv preprint arXiv:2512.16149, 2025. [3] Lili Chen, Mihir Prabhudesai, Katerina Fragkiadaki, Hao Liu, and Deepak Pathak. Selfquestioning language models. arXiv preprint arXiv:2508.03682, 2025. [4] Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335, 2024. [5] Yuqian Fu, Tinghong Chen, Jiajun Chai, Xihuai Wang, Songjun Tu, Guojun Yin, Wei Lin, Qichao Zhang, Yuanheng Zhu, and Dongbin Zhao. Srft: A single-stage method with supervised and reinforcement fine-tuning for reasoning. arXiv preprint arXiv:2506.19767, 2025. [6] Jiaxuan Gao, Wei Fu, Minyang Xie, Shusheng Xu, Chuyi He, Zhiyu Mei, Banghua Zhu, and Yi Wu. Beyond ten turns: Unlocking long-horizon agentic search with large-scale asynchronous rl. arXiv preprint arXiv:2508.07976, 2025. 10

[7] Shixiang Gu, Timothy Lillicrap, Zoubin Ghahramani, Richard E Turner, and Sergey Levine. Q-prop: Sample-efficient policy gradient with an off-policy critic. arXiv preprint arXiv:1611.02247, 2016. [8] Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. MiniLLM: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations, pages 1–24, Vienna, Austria, 2024. OpenReview.net. [9] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. [10] Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609–6625, Barcelona, Spain (Online), 2020. International Committee on Computational Linguistics. [11] Chengsong Huang, Wenhao Yu, Xiaoyang Wang, Hongming Zhang, Zongxia Li, Ruosen Li, Jiaxin Huang, Haitao Mi, and Dong Yu. R-zero: Self-evolving reasoning llm from zero data. arXiv preprint arXiv:2508.05004, 2026. [12] Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, and Andreas Krause. Reinforcement learning via self-distillation. arXiv preprint arXiv:2601.20802, 2026. [13] Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan O Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training LLMs to reason and leverage search engines with reinforcement learning. In Second Conference on Language Modeling, 2025. [14] Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1601–1611, Vancouver, Canada, July 2017. Association for Computational Linguistics. [15] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick SH Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6769–6781, Online, 2020. Association for Computational Linguistics. [16] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452–466, 2019. [17] Kuan Li, Zhongwang Zhang, Huifeng Yin, Liwen Zhang, Litu Ou, Jialong Wu, Wenbiao Yin, Baixuan Li, Zhengwei Tao, Xinyu Wang, Weizhou Shen, Junkai Zhang, Dingchu Zhang, Xixi Wu, Yong Jiang, Ming Yan, Pengjun Xie, Fei Huang, and Jingren Zhou. Websailor: Navigating super-human reasoning for web agent. arXiv preprint arXiv:2507.02592, 2025. [18] Bo Liu, Leon Guertler, Simon Yu, Zichen Liu, Penghui Qi, Daniel Balcells, Mickel Liu, Cheston Tan, Weiyan Shi, Min Lin, Wee Sun Lee, and Natasha Jaques. Spiral: Self-play on zero-sum games incentivizes reasoning via multi-agent multi-turn reinforcement learning. arXiv preprint arXiv:2506.24119, 2026. [19] Bo Liu, Chuanyang Jin, Seungone Kim, Weizhe Yuan, Wenting Zhao, Ilia Kulikov, Xian Li, Sainbayar Sukhbaatar, Jack Lanchantin, and Jason Weston. Spice: Self-play in corpus environments improves reasoning. arXiv preprint arXiv:2510.24684, 2025. [20] Hongliang Lu, Yuhang Wen, Pengyu Cheng, Ruijin Ding, Jiaqi Guo, Haotian Xu, Chutian Wang, Haonan Chen, Xiaoxi Jiang, and Guanjun Jiang. Search self-play: Pushing the frontier of agent capability without supervision. arXiv preprint arXiv:2510.18821, 2025. 11

[21] Kevin Lu and Thinking Machines Lab. On-policy distillation. Thinking Machines Lab: Connectionism, 2025. [22] Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, pages 9802–9822, Toronto, Canada, 2023. Association for Computational Linguistics. [23] OpenAI OpenAI, Matthias Plappert, Raul Sampedro, Tao Xu, Ilge Akkaya, Vineet Kosaraju, Peter Welinder, Ruben D’Sa, Arthur Petron, Henrique P. d. O. Pinto, Alex Paino, Hyeonwoo Noh, Lilian Weng, Qiming Yuan, Casey Chu, and Wojciech Zaremba. Asymmetric self-play for automatic goal discovery in robotic manipulation. arXiv preprint arXiv:2101.04882, 2021. [24] Emiliano Penaloza, Dheeraj Vattikonda, Nicolas Gontier, Alexandre Lacoste, Laurent Charlin, and Massimo Caccia. Privileged information distillation for language models. arXiv preprint arXiv:2602.04942, 2026. [25] Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5687–5711, Singapore, 2023. Association for Computational Linguistics. [26] Hejian Sang, Yuanda Xu, Zhengze Zhou, Ran He, Zhipeng Wang, and Jiachen Sun. Crisp: Compressed reasoning via iterative self-policy distillation. arXiv preprint arXiv:2603.05433, 2026. [27] John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. Highdimensional continuous control using generalized advantage estimation. In The Fourth International Conference on Learning Representations, pages 1–14, San Juan, Puerto Rico, 2016. OpenReview.net. [28] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. [29] Idan Shenfeld, Mehul Damani, Jonas Hübotter, and Pulkit Agrawal. Self-distillation enables continual learning. arXiv preprint arXiv:2601.19897, 2026. [30] Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. R1-searcher: Incentivizing the search capability in llms via reinforcement learning. arXiv preprint arXiv:2503.05592, 2025. [31] Hao Sun, Zile Qiao, Jiayan Guo, Xuanbo Fan, Yingyan Hou, Yong Jiang, Pengjun Xie, Yan Zhang, Fei Huang, and Jingren Zhou. Zerosearch: Incentivize the search capability of llms without searching. arXiv preprint arXiv:2505.04588, 2025. [32] Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554, 2022. [33] Songjun Tu, Chengdong Xu, Qichao Zhang, Yaocheng Zhang, Xiangyuan Lan, Linjing Li, and Dongbin Zhao. Dynamic dual-granularity skill bank for agentic rl. arXiv preprint arXiv:2603.28716, 2026. [34] Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533, 2022. [35] Yinjie Wang, Xuyang Chen, Xiaolong Jin, Mengdi Wang, and Ling Yang. Openclaw-rl: Train any agent simply by talking. arXiv preprint arXiv:2603.10165, 2026. 12

[36] Ziliang Wang, Xuhui Zheng, Kang An, Cijun Ouyang, Jialu Cai, Yuhang Wang, and Yichao Wu. Stepsearch: Igniting llms search ability via step-wise proximal policy optimization. arXiv preprint arXiv:2505.15107, 2025. [37] Jialong Wu, Baixuan Li, Runnan Fang, Wenbiao Yin, Liwen Zhang, Zhenglin Wang, Zhengwei Tao, Ding-Chu Zhang, Zekun Xi, Xiangru Tang, Yong Jiang, Pengjun Xie, Fei Huang, and Jingren Zhou. Webdancer: Towards autonomous information seeking agency. In The Thirtyninth Annual Conference on Neural Information Processing Systems, pages 1–29, RSan Diego, USA, 2025. Curran Associates Inc. [38] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. [39] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380, Brussels, Belgium, 2018. Association for Computational Linguistics. [40] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, pages 1–33, Kigali, Rwanda, 2023. OpenReview.net. [41] Tianzhu Ye, Li Dong, Xun Wu, Shaohan Huang, and Furu Wei. On-policy context distillation for language models. arXiv preprint arXiv:2602.12275, 2026. [42] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason E Weston. Self-rewarding language models. In Proceedings of the 41st International Conference on Machine Learning, pages 57905–57923, Vienna, Austria, 2024. PMLR. [43] Chuhuai Yue, Chengqi Dong, Yinan Gao, Hang He, Jiajun Chai, Guojun Yin, and Wei Lin. Promoting efficient reasoning with verifiable stepwise reward. arXiv preprint arXiv:2508.10293, 2025. [44] Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model? In The Thirty-ninth Annual Conference on Neural Information Processing Systems, pages 1–36, RSan Diego, USA, 2025. Curran Associates Inc. [45] Zhenrui Yue, Kartikeya Upasani, Xianjun Yang, Suyu Ge, Shaoliang Nie, Yuning Mao, Zhe Liu, and Dong Wang. Dr. zero: Self-evolving search agents without training data. arXiv preprint arXiv:2601.07055, 2026. [46] Yaocheng Zhang, Haohuan Huang, Zijun Song, Yuanheng Zhu, Qichao Zhang, Zijie Zhao, and Dongbin Zhao. Criticsearch: Fine-grained credit assignment for search agents via a retrospective critic. arXiv preprint arXiv:2511.12159, 2025. [47] Yaocheng Zhang, Yuanheng Zhu, Yuqian Fu, Songjun Tu, and Dongbin Zhao. Offline goalconditioned reinforcement learning with elastic-subgoal diffused policy learning. In Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems, page 2336–2344, Richland, SC, 2025. International Foundation for Autonomous Agents and Multiagent Systems. [48] Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models. arXiv preprint arXiv:2601.18734, 2026. 13

[49] Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. DeepResearcher: Scaling deep research via reinforcement learning in real-world environments. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, page 414–431, Suzhou, China, 2025. Association for Computational Linguistics.

Implementation

In our experiments, we implement π-Play through alternating optimization over the examiner and the student-teacher modules. Following prior self-play work, the examiner generates 1-, 2-, 3-, and 4-hop questions with a default ratio of 4:3:2:1. In each iteration (i.e., iter1, iter2, and iter3), we first train the examiner for 50 steps, then use it to generate QA data from the corresponding prompts, and subsequently train the student on the synthesized data for another 50 steps. Meanwhile, the teacher is soft-updated at every student training step. Consistent with prior self-play settings, we run only three iterations in total, yielding 150 total training steps for each model, which is substantially fewer than baselines such as Search-R1. Throughout training, we adopt a decayed λ schedule to gradually weaken teacher guidance, allowing the student to progressively move toward regions with higher EM reward. For Qwen3-4B and Qwen3-8B, we set λ to 0.03, 0.003, and 0.002 across the three iterations, respectively. For Qwen3-4B-Instruct-2507, we set λ to 0.1, 0.03, and 0.03. The format reward rf for the examiner is set in the same way as the proposer’s format reward in Dr.Zero [45]. Full hyperparameter details are reported in Table 5, Table 6, and Table 7.

Table 5: Examiner hyperparameter settings. Steps Optimizer Optimizer Momentum Warmup Ratio Weight Decay Learning Rate Max Gradient Norm Group size Reward size KL-Div Total Train Batch Size LR Scheduler Precision (WA) Max Turn in Rollout Max Sequence Length

50 AdamW β1 , β2 = 0.9, 0.999 0.03 0.01 5e-7, 1e-6 0.1, 1.0 1 5 0 256 Constant with Warmup BF16-mixed 5 4096

Table 6: Student hyperparameter settings. Steps Optimizer Optimizer Momentum Warmup Ratio Weight Decay Learning Rate Max Gradient Norm Group size in GRPO KL-Div in GRPO ϵ in GRPO Total Train Batch Size LR Scheduler Precision (WA) Max Turn in Rollout Max Sequence Length

50 AdamW β1 , β2 = 0.9, 0.999 0.03 0.01 1e-6 0.1, 1.0 5 0, 0.001 0.2 256 Constant with Warmup BF16-mixed 5 8192

Table 7: Teacher hyperparameter settings. Precision (WA) Soft Update Weight τ Max Sequence Length

Further Analysis

B.1

Ablation Study

BF16-mixed 0.05 8192

For the distillation coefficient λ in Eq. 7, we adopt a decaying schedule, as described in Appendix A. Accordingly, we conduct an ablation study on Qwen3-4B-Instruct-2507 to verify whether this decaying strategy is superior to using a fixed λ across three iterations. As shown in Table 8, compared with a constant λ, the decaying schedule not only maintains a higher evolution speed in the early stages, but also converges to better final performance in the later stages of training. Table 8: Ablation study of λ. Bold value indicates the top-performing result, while underline value denotes the second-best. General QA λ

TriviaQA

Multi-Hop QA

PopQA

HotpotQA

2WikiMQA

MuSiQue

Bamboogle

Total

12.8 11.8 9.85 12.4 10.6 11.1 13.2

40.8 41.6 40.0 35.2 32.0 31.2 39.2

263.6 263.8 257.9 252.2 251.8 248.9 264.9

13.8 12.8 13.6 12.8 13.0 13.8 14.2

40.0 35.2 38.4 36.8 36.8 36.0 40.0

264.4 261.9 264.1 264.2 263.0 258.4 271.7

13.4 14.0 14.2 12.0 13.8 13.3 13.4

40.0 36.0 37.6 36.0 38.4 38.4 44.0

264.8 261.3 268.4 265.6 267.1 265.5 277.2

Iter 1 0.100 0.050 0.030 0.010 0.005 0.000 Decay

40.1 39.8 40.1 38.5 40.1 40.8 40.7

61.8 62.1 61.1 61.6 61.1 61.3 62.1

42.5 42.2 42.7 42.5 42.0 42.3 42.9

35.5 34.9 33.7 35.4 34.9 34.3 36.7

30.1 31.4 30.4 26.6 31.1 27.9 30.1 Iter 2

0.100 0.050 0.030 0.010 0.005 0.000 Decay

40.1 40.6 40.9 41.0 41.2 41.2 41.2

61.5 62.2 61.7 62.3 62.6 61.6 62.5

42.5 43.6 43.2 43.7 44.0 43.4 43.5

36.0 36.2 37.4 37.2 37.3 36.3 37.8

30.5 31.3 28.9 30.4 28.1 26.1 32.5 Iter 3

0.100 0.050 0.030 0.010 0.005 0.000 Decay

40.8 41.0 42.5 42.5 42.4 41.2 41.8

61.7 61.3 62.0 62.1 62.4 62.3 63.2

42.3 42.8 44.3 44.5 43.9 43.6 44.0

35.6 35.7 38.0 36.5 37.6 37.7 38.5

31.0 30.5 29.8 32.0 28.6 29.0 32.3

Moreover, we perform an ablation study on the top-K parameter in Eq. 11. The study is conducted on Qwen3-4B-Instruct-2507, with K selected from 25, 50, 100, 200. As shown in Table 9, the results reveal that larger values of K do not necessarily yield better performance. Instead, the optimal performance is achieved when K is around 50. 16

Table 9: Ablation study of K. Bold value indicates the top-performing result General QA Top K

TriviaQA

Multi-Hop QA

PopQA

HotpotQA

2WikiMQA

MuSiQue

Bamboogle

Total

12.9 12.6 13.2 11.5

35.2 38.4 39.2 36.8

255.8 259.8 264.9 255.4

11.4 14.0 14.2 12.9

36.0 36.8 40.0 43.2

259.0 264.6 271.7 268.3

13.2 13.8 13.4 12.8

39.2 36.8 44.0 36.8

269.0 268.5 277.2 267.1

Iter 1 200 100 50 25

37.8 39.4 40.7 40.0

61.7 61.8 62.1 61.2

37.1 36.8 36.7 35.1

42.1 42.4 42.9 41.0

29.0 28.4 30.1 29.8 Iter 2

200 100 50 25

41.4 41.9 41.2 40.5

61.7 62.2 62.5 61.9

43.2 44.4 43.5 43.0

35.5 36.8 37.8 36.9

29.8 28.5 32.5 29.9 Iter 3

200 100 50 25

B.2

42.3 41.9 41.8 43.1

62.1 62.4 63.2 62.7

44.6 44.8 44.0 42.9

37.3 37.6 38.5 36.7

30.3 31.2 32.3 32.1

Training Cost

Table 10 reports the average per-step training time of different methods. Across various search agents, we observe that although our approach introduces an additional teacher model compared with Dr.Zero, it does not significantly increase the per-step training time. While the teacher model performs an extra forward pass along the student’s rollout to compute token probabilities, π-Play introduces almost no additional training time compared with Dr.Zero. We attribute this to the fact that Dr.Zero lacks fine-grained feedback to penalize ineffective tool-use behaviors, causing the student model to generate redundant search queries (Fig. 5), thereby increasing training time. This effect occurs both during the computation of difficulty rewards in the examiner and during the student model’s rollout. The reported training times were measured on a node equipped with 8 NVIDIA H20 GPUs, and we follow the same training configurations described in Appendix A. Table 10: The training time. We analyze the per-step training time (in seconds) for each iteration under Qwen3-4B-Instruct-2507. Although the teacher model requires an additional forward pass along the student’s rollout to compute token probabilities, π-Play introduces only a small per-step training overhead of 4.3% relative to Dr.Zero.

Examiner Student (Rollout) Teacher (Forward) Student (Updating) Total Time

B.3

Iter1 Dr.Zero π-Play

Iter2 Dr.Zero π-Play

Iter3 Dr.Zero π-Play

Average Dr.Zero π-Play

311.6 88.9 94.0 494.5

407.8 146.0 116.2 670.0

623.4 176.8 123.2 923.4

447.6 137.2 111.1 695.9

311.6 81.1 21.8 94.1 508.6

526.3 159.9 31.8 123.0 841.0

529.9 149.8 30.6 117.8 828.1

455.9 130.3 28.1 111.6 725.9

Search Behavior Analysis

To further investigate the effect of QCP guided self-distillation on the student’s search behavior, we quantitatively analyze two metrics, search count and query redundancy, across seven QA benchmarks. 17

Question: Are Northwestern University and Middlebury College both private schools? Ground Truth: Yes Dr.Zero-4B

𝝅-Play-4B Targets the required information

Targets the required information Are Northwestern University and Middlebury College both private schools

Are Northwestern University and Middlebury College both private schools

(No key information)

# Inefficient Query Unchanged.

Redundant

# Change Query

Focused search

Are Northwestern University and Middlebury College private institutions

Is Northwestern University a private school

(No key information)

… the only private university in Big Ten Conference …

Focused search

Is Northwestern University a private university

Is Middlebury College a private school

… the only private university in Big Ten Conference …

… private university …

Focused search Ans: Yes

Is Middlebury College a private college … private university …

Ans: Yes

Figure 5: Side-by-side trajectories of Dr.Zero (left) and π-Play (right) on the same question. Each trajectory shows multi-turn interactions with the search engine (actions, responses, and final answer). Although both Dr.Zero and π-Play answer the query correctly, our method (right) uses fewer queries and reaches a logically structured answer with minimal redundancy.

As shown in Table 11, compared with conventional self-play methods such as SQLM* and Dr.Zero, the search agent trained with π-Play not only achieves higher answer accuracy, but also performs fewer search actions. These results indicate that π-Play learns more efficient search behaviors, which we attribute to the token-level supervision provided by QCP guided self-distillation. The same pattern is reflected in query redundancy: with fine-grained supervision, π-Play produces more effective queries with lower redundancy. Table 11: Quantitative analysis of search count and query redundancy. We report the average number of search actions and the average query redundancy across seven QA benchmarks (NQ, TriviaQA, PopQA, HotpotQA, 2WikiMQA, MuSiQue, and Bamboogle) using Qwen3-4B-Instruct2507. Compared with SQLM* and Dr.Zero, π-Play demonstrates fewer search actions and lower query redundancy.

SQLM* Dr.Zero π-Play

Avg Accuracy ↑

Search Count ↓

Query Redundancy ↓

35.7 38.0 39.6

2.3 2.4 1.9

0.43 0.48 0.37

B.4

Case Study

We present QCP examples generated by the examiner during the π-Play training process. The QCPs for different hop settings are presented in Fig. 6, Fig. 7, and Fig. 8. All cases are obtained from the model trained on Qwen3-4B-Instruct-2507. In particular, we further compare the responses of the student model generated by the baseline Dr.Zero and our π-Play method on the same question, as shown in Fig. 5. Each response contains multi-turn interactions between the model and the external search engine. Random Document Now, generate a question and its answer with n = 1 hops starting from the following source document: (Title: "FC Barcelona Rugby League")\nwidely speculated, but purely a hypothesis, that this branch of the successful FC Barcelona club will become members of the Rugby Football League and possibly Super League should the venture prove a success in Catalonia, as they were now playing more competitive rugby, on a more regular basis. Their first ever game was against CR Sant Cugat on 13 May 2010, a game which ended in a disappointing 30-20 loss. They earned their first win of the competition with a 44-30 win at home to GEiEG to give them a hope of qualifying for the finals. And, despite GEiEG winning ...

Examiner: Generate Task <question>Which club played its first ever rugby game against CR Sant Cugat on 13 May 2010?</question> <answer>FC Barcelona Rugby League</answer>

Figure 6: QCP example with hop = 1 provided by the examiner.

Random Document Now, generate a question and its answer with n = 2 hops starting from the following source document: (Title: "Luna 19") Luna 19 (a.k.a. Lunik 19) (E-8-LS series), was an unmanned space mission of the Luna program. "Luna 19" extended the systematic study of lunar gravitational fields and location of mascons (mass concentrations). It also studied the lunar radiation environment, the gamma-active lunar surface, and the solar wind. Photographic coverage via a television system was also obtained. "Luna 19" was the first of the "advanced" lunar orbiters whose design was based upon the same Ye-8class bus used for the lunar rovers and the sample collectors. For these orbiters, designated Ye-8LS, the basic "lander stage" was topped off by a ...

Examiner: Query 1 <tool_call>{"name": "search", "arguments": {"query_list": ["Luna 19 mission spacecraft class design"]}}</tool_call>

Tool Response (Title: "Luna 19") ... Luna 19 was the first of the "advanced" lunar orbiters whose design was based upon the same Ye-8-class bus used for the lunar rovers and the sample collectors ...

Examiner: Query 2 <tool_call>{"name": "search", "arguments": {"query_list": ["Ye-8LS spacecraft used for lunar rovers and sample collectors"]}}</tool_call>

Tool Response (Title: "Luna 21") ... Luna 21 (Ye-8 series) was an unmanned space mission, and its spacecraft, of the Luna program, also called Lunik 21, in 1973. The spacecraft landed on the Moon and deployed the second Soviet lunar rover, "Lunokhod 2". ...

Examiner: Generate Task <question>Which country operated the lunar rover Lunokhod 2, which was deployed by the Ye-8 series spacecraft used in the Luna 19 mission</question> <answer>Soviet Union</answer>

Figure 7: QCP example with hop = 2 provided by the examiner

Random Document Now, generate a question and its answer with n = 3 hops starting from the following source document: (Title: "Chris Charsley")\nseason, including the test match against Darwen which won them promotion to the First Division. He also played for Aston Villa as a guest in 1886. Charsley had a brief spell with West Bromwich Albion, whom he joined in August 1891. He made his only appearance for Albion in a 2-1 defeat to Preston North End in November of the same year, before re-joining Small Heath the following month. On 25 February 1893, he became the first Birmingham City player-and the only player who had played for the club in its Small Heath Alliance days-to represent ...

Examiner: Query 1 <tool_call>{"name": "search", "arguments": {"query_list": ["Chris Charsley Birmingham City first player Small Heath Alliance"]}}</tool_call>

Tool Response (Title: "History of Birmingham City F.C. (1875-1965)") ... Goalkeeper Chris Charsley, a policeman who played as an amateur and went on to serve as chief constable of Coventry, became the first Small Heath player to be capped by England. ...

Examiner: Query 2 <tool_call>{"name": "search", "arguments": {"query_list": ["Chris Charsley England international debut date"]}}</tool_call>

Tool Response (Title: "1892-93 Small Heath F.C. season") ... Chris Charsley became the first Small Heath player to play for England, and only the second, after Caesar Jenkyns for Wales, to represent his country at full international level, when he took the field against Ireland on 27 February at Aston Villa\'s ground in Perry Barr, Birmingham. ...

Examiner: Query 3 <tool_call>{"name": "search", "arguments": {"query_list": ["Chris Charsley England international match opponent 27 February 1893"]}}</tool_call>

Tool Response (Title: "1892-93 in English football") ... Chris Charsley of Small Heath, who later went on to be Chief Constable of Coventry, made his solitary England appearance in goal. ...

Examiner: Generate Task <question>Which team did Chris Charsley play against in his only England international match?</question> <answer>Ireland</answer>

Figure 8: QCP example with hop = 3 provided by the examiner.

Distillation Loss

To save GPU memory, we adopt top-K distillation following [12], where the top-K set is defined with respect to the student distribution: T DDistill πθS ∥ πψ )=

|yi | 1 X T KL(πθS (· | q, yi,<t )∥stopgrad(πψ (· | q, c, yi,<t ))) |yi | t=1

ŷi,t ∈πθ

|yi |

 X πθS (ŷi,t | q, yi,<t ) 1 X S πθ (ŷi,t | q, yi,<t ) · log   T   |yi | stopgrad(πψ (ŷi,t | q, c, yi,<t )) t=1 ŷi,t ∈top (π S )   K θ     |yi | =  S X X πθ (ŷi,t | q, yi,<t )  + 1 S πθ (ŷi,t | q, yi,<t ) · log   T stopgrad(π (ŷ | q, c, y ))   |yi | t=1 i,t i,<t ψ S \top (π S ) ŷi,t ∈πθ   K θ | {z } 

tail



 |yi | X πθS (ŷi,t | q, yi,<t ) 1 X S   πθ (ŷi,t | q, yi,<t ) · log T  |yi |  stopgrad(πψ (ŷi,t | q, c, yi,<t ))   t=1 ŷi,t ∈top (π S ) K θ     P S ≈  (ŷ | q, y ) π 1− S i,t i,<t P ŷt ∈topK (πθ ) θ + 1− S  P · log S ) πθ (ŷi,t | q, yi,<t )   ŷt ∈topK (πθ T   stopgrad 1− S ) πψ (ŷi,t | q, c, yi,<t ) ŷt ∈topK (πθ   | {z } tail

(11)

Rather than computing the full KL divergence over the entire vocabulary, we split the distillation loss into two components: the exact contribution from the student’s top-K tokens and a tail term corresponding to all remaining tokens. The tail is further approximated by collapsing the nontop-K probability mass into a single residual term. This strategy avoids storing two full copies of vocabulary logits: one for the student and one for the teacher, thereby greatly reducing memory usage. Empirically, this approximation has negligible impact on performance, since most tokens of the vocabulary are not informative at a given time [12]. Further analysis of the choice of K can be found in Appendix B.

Prompts

We provide the system prompts for all models in Section D.1, and the user prompts for the examiner, teacher, and student models in Sections D.2, D.3, and D.4, respectively. D.1

System Prompts System Prompt for the Examiner, Teacher and Student You are a helpful assistant. # Tools You may call one or more functions to assist with the user query. You are provided with function signatures within <tools></tools> XML tags: <tools> {"type": "function", "function": {"name": "search", "description": "Searches the web for relevant information based on the given query.", "parameters": {"type": "object", "properties": {" query_list": {"type": "array", "description": "A list of fully-formed semantic queries. The tool will return search results for each query.", "enum": null}}, "required": ["query_list"]}, "strict ": false}} </tools> For each function call, return a json object with function name and arguments within <tool_call></ tool_call> XML tags:

<tool_call> {"name": <function-name>, "arguments": <args-json-object>} </tool_call>

Figure 9: System prompt for the examiner, teacher and student in π-play. They use the same system prompt. D.2

Examiner Prompts

User Prompt for the Examiner You are an expert in question generation. Craft one challenging, deterministic question and its single, unambiguous answer based on the provided source document. The logical path must start from the document and require exactly n hops (i.e., n-1 searches) to reach the final answer. ### Definitions 1. Hop: A node in the reasoning chain. Hop 1 is the starting entity found in the document. Hop n is the final answer. ### Inputs 1. n: the exact number of hops in the reasoning chain (requiring n-1 searches). 2. Source document: the full source text. ### Process & Tools 1. Analyze the Document and Select the Starting Point - Read and analyze the source document. - Select a specific entity, event or detail explicitly mentioned in the text. This entity becomes Hop 1 (the initial clue). 2. Design the Chain Forwards - From Hop 1 to Hop 2: Identify a factual attribute or relation of Hop 1 that is NOT in the text but can be found via search. The result is Hop 2. - Iterate: Continue connecting the current Hop i to the next Hop i+1 using deterministic, verifiable relation found via search. - Stop at Hop n: Continue this process until you have exactly n hops. Hop n must be a single, canonical final answer. 3. Reasoning & Search Protocol - Always reason inside `<think> ... </think>` when you plan connections or receive new information. - For each hop transition that requires external information, issue search query using `< tool_call> ... </tool_call>`. - Search results will be provided between `<tool_response> ... </tool_response>` by the system. 4. Output Format - Emit a numbered sequence of EXACTLY n-1 search steps. For each search i (1 to n-1), produce: `<think> Reasoning step i: Identify Hop i in document/search results, formulate query to reach Hop i+1 </think>` `<tool_call> Query to search Hop i+1 </tool_call>` `[Wait for search results in <tool_response> from system]` - After completing all searches and arriving at Hop n, output the question and final answer: `<think> Final reasoning step: Confirm the chain is complete with Hop n and formulate the question </think>` `<question> A challenging question that provides Hop 1 (the initial clue) and asks for the final answer (Hop n) </question>` `<answer> The single, concise final answer (Hop n) </answer>` ### Examples 1. Example template for Hop n = 1, i.e. no search: `<think> [Explain how Hop 1 is selected from the source document and how the question is formulated] </think>` `<question> [Question based solely on the text entity Hop 1] </question>` `<answer> [Answer (Hop 1)] </answer>` 2. Example template for Hop n = 3, i.e. 2 searches: `<think> [Reasoning step 1: Find Hop 1 in the source document, formulate the query to reach Hop 2] </think>` `<tool_call> [Search query to find Hop 2 based on Hop 1] </tool_call>` `[Wait for search results in <tool_response> from system]` `<think> [Reasoning step 2: Reason on search results to identify Hop 2 and write the next query to find Hop 3] </think>` `<tool_call> [Search query to find Hop 3 based on Hop 2] </tool_call>` `[Wait for search results in <tool_response> from system]` `<think> [Final reasoning step: Confirm Hop 3 in search results and formulate the question starting from Hop 1] </think>` `<question> [Question starting with Hop 1, requiring the solver to find Hop 2 to eventually reach the Answer (Hop 3)] </question>` `<answer> [Answer (Hop 3)] </answer>` ### Critical Rules

1. Start in Document: Hop 1 must be explicitly present in the source text. Every subsequent hop must be supported by the corresponding search results. 2. Search is mandatory for n > 1: Each link between hops beyond Hop 1 must use the search engine. 3. Exact search count: Emit exactly (n-1) `<tool_call>` entries, no more, no fewer. 4. No spoilers: The question must mention only Hop 1; do not include or hint at intermediate hops. 5. Clarity: The question is self-contained; the answer is concise and direct (no extra commentary, formatting or explanation). 6. Chain integrity: Each hop must depend strictly on the previous hop. No hop should be skippable or derivable without its immediate predecessor. Now, generate a question and its answer with n = {hops} hops starting from the following source document: {document}

Figure 10: Initial instructions for the examiner in π-play. Our prompt for examiner is developed based on Dr.Zero [45] D.3

Teacher Prompts User Prompt for the Teacher (Qwen3-4B-Instruct-2507) You are a helpful assistant. You will be given privileged information about the reverse solution process of the question (i.e., construction process of the question). Please pretend not to know the source document used to construct the question and the ground-truth in the reverse solution process, and answer the question again with reference to the construction process of the question. ### General Rules 1. Precise Searching: When generating search queries, ensure they are specific, semantically complete, and directly target the key information you are missing. 2. Context Retention: Remember prior conversations and search results, maintaining logical consistency across multiple rounds of searching. 3. Termination Judgment: When information are sufficient to determine the answer, immediately stop searching and output the answer. 4. Reference to Privileged Information: When outputting search actions and answers, refer to the question construction process in privileged information, but do not directly output the groundtruth in the first turn or directly use extra information from the source document. 5. Search Boundary Constraint: When searching, do not introduce information that is not contained in the question or search results, even if it exists in the source document. ### Example (Only showcase the logical style, please do not directly imitate the specific content ) #### User This is the reverse solution process of the question (i.e., the process of generating the question) given by teacher: =========================================================================== Now, generate a question and its answer with n = 3 hops starting from the following source document: (Title: "Steven Febey") selected in the national draft. He fell just short of Robert Flower's club record of 272 games and as of 2007 only four people have played more games for the Demons After the 2002 season, Febey went to Bali with his teammates and was caught up in the bombings which devastated the island. Standing beside former Demons player Steven Armstrong and David Robbins, Febey was about to enter the Sari Club when the car bomb exploded...[omitted due to page size] Reasoning step 1: Identify Hop 1 in the document, formulate the query to reach Hop 2 <tool_call>\n{"name": "search", "arguments": {"query_list": ["other players who played in both 1988 and 2000 AFL grand finals for Melbourne Demons"]}}\n</tool_call><|im_end|> ..... Hop 2 in the document and search results is not explicitly mentioned, so we need to further refine the search. The document mentions David Neitz, the captain of Melbourne, and players like Shane Woewodin and Greg Healy. However, the search results point us to the 1988 and 2000 AFL Grand Finals, indicating that we need to look for a player specifically from these finals who also played for Melbourne. The final hop, or Hop 3, must be a well-known player from the Melbourne Demons who played in both 1988 and 2000 grand finals, which can be Robert Harvey based on historical knowledge. Reasoning step 2: Identify Hop 2 in the search results and reformulate to find Hop 3 <tool_call>\n{"name": "search", "arguments": {"query_list": ["Robert Harvey Melbourne Demons 1988 2000 grand finals"]}}\n</tool_call><|im_end|> ..... The search results confirm that Robert Harvey played for the Melbourne Demons in both 1988 and 2000, making him the well-known player who fits the criteria. Therefore, Hop 3 is "Robert Harvey". The question is formulated to start with the explicit entity in the document, "Steven Febey", and lead to the single, unambiguous answer, "Robert Harvey". <question> Who is the only other active Melbourne Demons player to have played in both the 1988 and 2000 AFL Grand Finals, alongside Steven Febey? </question>

<answer> Robert Harvey </answer> =========================================================================== Now, answer this question again by yourself, pretending you do not know the ground-truth answer, the source document. You need to start answering this question from scratch. Most importantly, pretending you do not know the ground-truth answer, the source document, and do not mention the source document when thinking. If you find you lack some knowledge, you can call a search engine by <tool_call> query </tool_call> and it will return the top searched results between < tool_response> and </tool_response>. You can search as many times as your want. If you find no further external knowledge needed, you can directly provide the answer inside <answer> and </ answer>, without detailed illustrations. For example, <answer> Beijing </answer>. Question: Who wrote a number of articles, profiles, and interviews of musicians between 2004 and 2006? #### Assistant (turn 1) <tool_call>\n{"name": "search", "arguments": {"query_list": ["1988 AFL Grand Finals players"]}}\n </tool_call> <tool_response> ...... </tool_response> #### Assistant (turn 2) <tool_call>\n{"name": "search", "arguments": {"query_list": ["2000 AFL Grand Finals players"]}}\n </tool_call> <tool_response> ...... </tool_response> #### Assistant (turn 3) I find that Robert Harvey and Steven Febey are the active Melbourne Demons players who have played in both the 1988 and 2000 AFL Grand Finals. <answer> Robert Harvey </answer> ### Notice Most importantly, pretending you do not know the ground-truth answer, the extra information in the source document, and do not mention the source document when thinking. ###Begin This is the reverse solution process question) given by teacher:

of the question (i.e., the process of generating the

=========================================================================== {demonstration} =========================================================================== Now, answer this question again by yourself, pretending you do not know the ground-truth answer, the source document. You need to start answering this question from scratch. Most importantly, pretending you do not know the ground-truth answer, the source document, and do not mention the source document when thinking. If you find you lack some knowledge, you can call a search engine by <tool_call> query </tool_call> and it will return the top searched results between < tool_response> and </tool_response>. You can search as many times as your want. If you find no further external knowledge needed, you can directly provide the answer inside <answer> and </ answer>, without detailed illustrations. For example, <answer> Beijing </answer>. Question: { question}

User Prompt for the Teacher (Qwen3-4B / Qwen3-8B) You are a helpful assistant. You will be given privileged information about the reverse solution process of the question (i.e., construction process of the question). Please pretend not to know the source document used to construct the question and the ground-truth in the reverse solution process, and answer the question again with reference to the construction process of the question. ### General Rules Precise Searching: When generating search queries, ensure they are specific, semantically complete , and directly target the key information you are missing. Context Retention: Remember prior conversations and search results, maintaining logical consistency across multiple rounds of searching. Termination Judgment: When information are sufficient to determine the answer, immediately stop searching and output the answer. Reference to Privileged Information: When outputting search actions and answers, refer to the question construction process in privileged information, but do not directly output the groundtruth in the first turn or directly use extra information from the source document. Search Boundary Constraint: When searching, do not introduce information that is not contained in the question or search results, even if it exists in the source document. ### Example (Only showcase the logical style, please do not directly imitate the specific content ) #### User This is the reverse solution process of the question (i.e., the process of generating the question) given by teacher: ===========================================================================

Now, generate a question and its answer with n = 3 hops starting from the following source document: (Title: "Steven Febey") selected in the national draft. He fell just short of Robert Flower's club record of 272 games and as of 2007 only four people have played more games for the Demons After the 2002 season, Febey went to Bali with his teammates and was caught up in the bombings which devastated the island. Standing beside former Demons player Steven Armstrong and David Robbins, Febey was about to enter the Sari Club when the car bomb exploded...[omitted due to page size] Reasoning step 1: Identify Hop 1 in the document, formulate the query to reach Hop 2 <tool_call>\n{"name": "search", "arguments": {"query_list": ["other players who played in both 1988 and 2000 AFL grand finals for Melbourne Demons"]}}\n</tool_call><|im_end|> ..... Hop 2 in the document and search results is not explicitly mentioned, so we need to further refine the search. The document mentions David Neitz, the captain of Melbourne, and players like Shane Woewodin and Greg Healy. However, the search results point us to the 1988 and 2000 AFL Grand Finals, indicating that we need to look for a player specifically from these finals who also played for Melbourne. The final hop, or Hop 3, must be a well-known player from the Melbourne Demons who played in both 1988 and 2000 grand finals, which can be Robert Harvey based on historical knowledge. Reasoning step 2: Identify Hop 2 in the search results and reformulate to find Hop 3 <tool_call>\n{"name": "search", "arguments": {"query_list": ["Robert Harvey Melbourne Demons 1988 2000 grand finals"]}}\n</tool_call><|im_end|> ..... The search results confirm that Robert Harvey played for the Melbourne Demons in both 1988 and 2000, making him the well-known player who fits the criteria. Therefore, Hop 3 is "Robert Harvey". The question is formulated to start with the explicit entity in the document, "Steven Febey", and lead to the single, unambiguous answer, "Robert Harvey". <question> Who is the only other active Melbourne Demons player to have played in both the 1988 and 2000 AFL Grand Finals, alongside Steven Febey? </question> <answer> Robert Harvey </answer> =========================================================================== Now, answer this question again by yourself, pretending you do not know the ground-truth answer, the source document. You need to start answering this question from scratch. Most importantly, pretending you do not know the ground-truth answer, the source document, and do not mention the source document when thinking. You must conduct reasoning inside <think> and </think> first every time you get new information. After reasoning, If you find you lack some knowledge, you can call a search engine by <tool_call> query </tool_call> and it will return the top searched results between <tool_response> and </ tool_response>. You can search as many times as your want. If you find no further external knowledge needed, you can directly provide the answer inside < answer> and </answer>, without detailed illustrations. Question: Who wrote a number of articles, profiles, and interviews of musicians between 2004 and 2006? #### Assistant (turn 1) <think>\nTo answer this question, I need to identify the players who played for the Melbourne Demons in both 1988 and 2000 AFL Grand Finals. The question mentions Steven Febey, and we need to find another player who played in both finals. First, I will look for information about the 1988 and 2000 AFL Grand Finals and the players who participated in them. / I will search for "1988 and 2000 AFL Grand Finals players" to find the relevant information.\n</ think> <tool_call\n{"name": "search", "arguments": {"query_list": ["1988 AFL Grand Finals players"]}}\n</ tool_call> <tool_response> ...... </tool_response> #### Assistant (turn 2) <think>\nI find the information about the 1988 AFL Grand Finals players. Next, I will search for "1988 and 2000 AFL Grand Finals players" to find the relevant information.\n</think> <tool_call>\n{"name": "search", "arguments": {"query_list": ["2000 AFL Grand Finals players"]}}\n </tool_call> <tool_response> ...... </tool_response> #### Assistant (turn 3) <think>\n \n</think> I find that Robert Harvey and Steven Febey are the active Melbourne Demons players who have played in both the 1988 and 2000 AFL Grand Finals. <answer> Robert Harvey </answer> ### Notice

Most importantly, pretending you do not know the ground-truth answer, the extra information in the source document, and do not mention the source document when thinking. ###Begin This is the reverse solution process question) given by teacher:

of the question (i.e., the process of generating the

=========================================================================== {demonstration} =========================================================================== Now, answer this question again by yourself, pretending you do not know the ground-truth answer, the source document. You need to start answering this question from scratch. Most importantly, pretending you do not know the ground-truth answer, the source document, and do not mention the source document when thinking. You must conduct reasoning inside <think> and </think> first every time you get new information. After reasoning, if you find you lack some knowledge, you can call a search engine by <tool_call> query </tool_call> and it will return the top searched results between <tool_response> and </tool_response>. You can search as many times as your want. If you find no further external knowledge needed, you can directly provide the answer inside <answer> and </answer>, without detailed illustrations. For example, <answer> Beijing </answer>. Question: { question}

Figure 11: Initial instructions for the teacher in π-play D.4

Student Prompts User Prompt for the Student (Qwen3-4B-Instruct-2507) Answer the given question. If you find you lack some knowledge, you can call a search engine by < tool_call> query </tool_call> and it will return the top searched results between <tool_response> and </tool_response>. You can search as many times as your want. If you find no further external knowledge needed, you can directly provide the answer inside <answer> and </answer>, without detailed illustrations. For example, <answer> Beijing </answer>. Question: {question}

User Prompt for the Student (Qwen3-4B / Qwen3-8B) Answer the given question. You must conduct reasoning inside <think> and </think> first every time you get new information. After reasoning, if you find you lack some knowledge, you can call a search engine by <tool_call> query </tool_call> and it will return the top searched results between <tool_response> and </tool_response>. You can search as many times as your want. If you find no further external knowledge needed, you can directly provide the answer inside <answer> and </answer>, without detailed illustrations. For example, <answer> Beijing </answer>. Question: { question}

Figure 12: Initial instructions for the student in π-play

Record · ID 14012 · SHA-256 af150ccf441b404b
Conceptio Open Knowledge Archive — every document is proof-bundled with source, license, and retrieval metadata.