arXiv CSopen access

DiPO: Disentangled Perplexity Policy Optimization for Fine-grained Exploration-Exploitation Trade-Off

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

machine learning, deep learning, neural networks

arXiv:2604.13902v1 [cs.LG] 15 Apr 2026

DiPO: Disentangled Perplexity Policy Optimization for Fine-grained Exploration-Exploitation Trade-Off Xiaofan Li1,2 , Ming Yang2†, Zhiyuan Ma2 , Shichao Ma2 , Jintao Du2 , Yu Cheng2 , Weiqiang Wang2 , Zhizhong Zhang1 , Xin Tan1 , Yanyun Qu4 , Lizhuang Ma1 , Yuan Xie1,3‡ 1 East China Normal University, 2 Ant Group 3 Shanghai Innovation Institute, 4 Xiamen University [email protected], {lzma, yxie}@cs.ecnu.edu.cn

Abstract Reinforcement Learning with Verifiable Rewards (RLVR) has catalyzed significant advances in the reasoning capabilities of Large Language Models (LLMs). However, effectively managing the exploration and exploitation trade-off remains a critical challenge. In this paper, we fully analyze the exploration and exploitation dilemma of extremely hard and easy samples during the training and propose a new fine-grained trade-off mechanism. Concretely, we introduce a perplexity space disentangling strategy that divides the sample space into distinct exploration (high perplexity) and exploitation (low perplexity) subspaces, thereby mining fine-grained samples requiring exploration-exploitation trade-off. Subsequently, we propose a bidirectional reward allocation mechanism with a minimum impact on verification rewards to implement perplexity-guided exploration and exploitation, enabling more stable policy optimization. Finally, we have evaluated our method on two mainstream tasks: mathematical reasoning and function calling, and experimental results demonstrate the superiority of the proposed method, confirming its effectiveness in enhancing LLM performance by fine-grained exploration-exploitation trade-off.

Introduction

Exploration-Exploitation Trade-Off (EETO) Auer et al. [2002] represents a core challenge in RL for LLM post-training, where error samples stand to benefit from further exploration, while correct samples are suitable for exploitation. In this paper, we propose the two dilemmas of EETO in GRPO-based methods. First dilemma: the degradation of the advantage of extreme samples. As illustrated in Figure 1 (a), a large proportions of sample groups fall into either the hard group (with uniform zero rewards) or the easy group (with uniform one rewards) during training. These extreme groups yield a zero advantage, leading to the lack of training gradients and thus exposes the GRPO paradigm to an dilemma of ETTO, where the hard and easy groups lack signals for exploration and exploitation, respectively. Second dilemma: ineffective exploration and exploitation. As shown in Figure 1 (b), the distribution of perplexity (PPL) reflects the exploratory (high PPL) or exploitative (low PPL) tendency of samples, where some error samples exhibit an exploitative tendency, while some correct samples show an exploratory one. This ineffective exploration and exploitation severely decrease the stability and plasticity of RL. Recent approaches have investigated the EETO by introducing PPL reward shaping. DACE Li et al. [2025] encourages all samples either to explore or to exploit based on intra-group accuracy. However, this coarse-grained partitioning would hinder policy optimization during early training stages when † Equan Contribution ‡ Corresponding Author

Preprint.

High PPL EH EL

(a) Proportion of Extreme Samples

Low PPL

Frenquency

Proportion(%)

Steps

PPL

(b) Distribution of PPL

Correct samples

with high PPL

(CH)

Error samples

with high PPL

(EH)

Encourage

exploitation

Encourage

exploration

Correct samples

with low PPL

(CL)

Error samples

with low PPL

(EL)

Correct

Error

Figure 1: (a) The proportion of Easy/Normal/Hard groups in each step during the DAPO training. (b) The PPL distribution of correct and error samples in the validation set at 300th steps of DAPO training. (c) Illustration of four samples after PSD fine-grained partitioning. most samples belong to the hard group. Meanwhile, CDE Dai et al. [2025] designs multiple weighting mechanisms to employ exploration rewards at appropriate times, yet it does not explicitly address exploitation and introduces more hyper-parameters. Moreover, both CDE and DACE directly use perplexity as a reward bias, which may introduce additional uncertainty. In this paper, we propose a novel method Disentangled Perplexity Policy Optimization (DiPO) to solve the two dilemmas and achieve a fine-grained EETO. To enable fine-grained sample mining for exploratory and exploitative purposes, we propose the Perplexity Space Disentangling (PSD) strategy, that integrates statistical probability of correctness (verification reward) and PPL. In Figure 1 (b), error samples typically exhibit higher PPL, whereas correct samples generally maintain lower PPL. PSD infers the optimal threshold τ ∗ by advantage judgment and minimizing classification errors, thus capturing the inherent correlation between PPL and sample correctness. Subsequently, as shown in Figure 1 (c), the entire sample space is partitioned into four fine-grained quadrants: correct samples with high PPL (CH), correct samples with low PPL (CL), error samples with high PPL (EH), and error samples with low PPL (EL). While the CL and EH samples naturally align with effective exploitation and exploration; The critical of EETO lies in encouraging exploitation for CH samples and exploration for EL samples. In addition, the uncertainty of PPL distribution results in a significant discrepancy between the PPL reward and the verification reward. To achieve stable RL training for EETO, we abandon the direct adoption of PPL for reward shaping and proposes a Bidirectional Reward Reallocation (BRR) mechanism. Specifically, on the one hand, to avoid interfering with the policy optimization guided by the verification reward, BRR is only performed on the zero-gradient easy and hard groups. On the other hand, because the verification reward variance of the easy and hard groups is zero, we introduce a maximum-PPL reward reallocation strategy, where the rewards corresponding to the maximum PPL samples in the easy and hard groups are set to 0 and 1, respectively, ensuring the variance of the reallocated reward distribution is close to zero. The proposed BRR incorporates the PPL signal while minimizing the perturbation to the original reward distribution, thus realizing more stable training. To summarize, the main contributions of this paper are: • We propose a perplexity space disentanglement strategy that partitions the PPL space based on perplexity and correctness distributions, enabling a fine-grained EETO. • We design a bidirectional reward reallocation mechanism that stabilizes training by reallocating rewards with minimal perturbation to the verification reward distribution.” • For mathematical reasoning and function calling tasks, our method achieves superior results confirming its effectiveness in reasoning enhancement for LLMs.

Preliminaries and Definitions

2.1

RLVR Algorithms

Group Relative Policy Optimization (GRPO) introduces a concise advantage calculation method. For each query q and its ground-truth answer a, given the verifiable reward calculation function: R(oi , a) = 1

if is_equivalent(a, oi ) else 2

(1)

push

Verification Rewards Verification Rewards . . .

Easy groups . . .

. . .

Easy group in ErS Hard group in EiS

Correct

. . .

Reward exploration

Error Penalize exploration

. . .

Reward Reallocation

. . .

Perplexity

. . .

Bidirectional Reward Reallocation . . .

. . .

Perplexity Space Disentangling

PPL Queue

Input Batch

Perplexity

. . .

Hard group Easy group

in EiS in ErS

. . .

Exploitation Space

. . .

Exploration Space

. . .

Hard groups Reallocated Rewards Reallocated Rewards

Figure 2: Illustration of DiPO, consisting of three modules: PPL Queue, Perplexity Space Disentangling (PSD), and Bidirectional Reward Reallocation (BRR). Specifically, the PPL Queue caches PPL items; PSD is used for fine-grained sample partition; and then BRR performs reward allocation. where {oi }G i=1 is generated by rollout policy πθold (o|q) through G-sampling. The estimated advantage i

Ât under R is then computed as: i

Ât (R) =

R(oi , a) − mean({R(oi , a)}G i=1 ) . std({R(oi , a)}N ) i=1

(2)

When G-sampling rewards are all 0 or 1, all advantages come to 0, indicating the sampled group consists solely of hard groups and easy groups. Dynamic Sampling Policy Optimization (DAPO) introduces new improvements based on GRPO The maximum optimization objective under R is as follows: i

JDAPO (θ, R) = E(q,a)∼D,{oi }G i=1 ∼πθ

|o | G X X

(o|q) PG old

i i=1 |o | i=1 t=1

# i i min sit Ât (R), clip sit , 1 − ϵlow , 1 + ϵhigh Ât (R) , s.t. 0 < |{R(oi , a)|R(oi , a) = 1)}| < G

(3) π (oi |q,oi ) sit = πθθ (ot i |q,o<ti ) is the ratio of importance sampling. In this paper, we use advanced t <t

where old DAPO as our baseline without overlong reward shaping, as in some cases, overlong reward shaping would damage the model’s performance. 2.2

Perplexity

Perplexity (shorted as PPL) in Large Language Models is a metric that quantifies how confidently a probability model predicts a sample. Mathematically, given a model πθ , a question q, and a response oi = (oi1 , oi2 , ..., oiT ) generated by πθ , the perplexity is given by: ! T 1X i i p = exp − log πθ (ot |q, o<t ) (4) T t=1 For a response, a low PPL indicates that the trajectory tends to exploitation, while higher PPL indicates that the trajectory tends to exploration Dai et al. [2025], Sun et al. [2025].

Methodology

3.1

Overview

As illustrated in Figure 2, the proposed DiPO mainly contains two modules: 1) Perplexity Space Disentangling (PSD); 2) Bidirectional reward reallocation (BRR). PSD calculates conditional probability distribution based on the PPL Queue cached with PPL-reward pairs, and then determines the optimal threshold by the advantage judgment and minimizing classification errors. We term higher and lower PPL region exploration space (ErS) and exploitation space (EiS), respectively. Through 3

PPL disentanglement, the ineffective samples for exploration and exploitation, i.e., hard groups in EiS and Easy groups in ErS are selected for policy optimization. Subsequently, BRR is implemented for these samples; for the hard group in EiS, the reward for maximum PPL sample is set to 1, thus encouraging exploration; the easy group in ErS, the reward for maximum PPL sample is set to zero, thus penalizing exploration. BRR incorporates the PPL signal and minimizes perturbations to the verification reward, enabling more stable training. 3.2

Perplexity Space Disentangling

PSD calculates the optimal threshold τ ∗ for dividing EiS and ErS by introducing the conditional probability distribution Pr(R|P ) where P and R represent PPL and reward. For ETTO, error samples stand to benefit from further exploration, while correct samples are suitable for exploitation. Thus the optimal EiS and ErS should have the highest relevance to sample correctness, where samples falls in EiS are considered more likely to be correct, whereas those in ErS are generally deemed error. Online Statistical Estimation. Since the PPL distribution induced by the policy changes dynamically during training, direct estimation of conditional probabilities may suffer from an unfavorable variancebias trade-off. To stabilize the estimation, we maintain a PPL queue Q that caches samples from the most recent two batches: Q = {(pi , r i )}M i=1 , where pi ∈ [1, +∞) denotes the perplexity (PPL) of the i-th sample and r i ∈ {0, 1} denotes its corresponding validation reward. For a given threshold τ , we estimate the empirical conditional probabilities of the reward given whether the PPL is below or above τ as PM

i=1 I(pi < τ, ri = 1) , PM i=1 I(pi < τ ) PM I(pi > τ, ri = 1) c = 1 | P > τ ) = i=1 Pr(R , PM i=1 I(pi > τ )

c = 1 | P < τ) = Pr(R

c = 0 | P < τ) Pr(R c = 0 | P > τ) Pr(R

i=1 I(pi < τ, ri = 0) , PM i=1 I(pi < τ ) PM I(pi > τ, ri = 0) = i=1 . PM i=1 I(pi > τ )

(5)

where I(·) denotes the indicator function. To quantify the statistical uncertainty of each estimate, we compute 95% confidence intervals using the Normal (Wald) approximation Wald [2004]. For a generic empirical probability estimate p̂ = m/n, p where m denotes the number of successes among n samples, the standard error is SE(p̂) = p̂(1 − p̂)/n, and the corresponding 95% confidence interval is [ p̂ − 1.96 · SE(p̂), p̂ + 1.96 · SE(p̂) ] .

(6)

These estimated probabilities represent the sample likelihood of being correct or error in the Exploitation Space (EiS, P < τ ) and Exploration Space (ErS, P > τ ) demarcated by the threshold τ. Advantage Judgment. During the early RL stage of pre-trained models, the correlation between PPL and correctness may not be positively correlated (see Appendix E.5), where directly activating PSD would hinder optimization. To this end, we introduce a advantage judgment mechanism based on the results of online statistical estimation. Specifically, for a given threshold τ , we define two advantage functions that characterize the separation between correct samples and error samples in divided PPL regions: ∆EiS (τ ) = Pr(R = 1 | P < τ ) − Pr(R = 0 | P < τ ), ∆ErS (τ ) = Pr(R = 0 | P > τ ) − Pr(R = 1 | P > τ ),

(7)

where ∆EiS (τ ) measures the correctness advantage in the EiS, and ∆ErS (τ ) measures the error advantage in the ErS. To reduce the impact of sampling randomness and ensure robust judgment, we conservatively evaluate these gaps using the boundary values of the 95% confidence intervals derived in the online estimation stage (Equation (6)). Concretely, the two judgment functions are as follows: ∆EiS (τ ) = L1 − U2

and

∆ErS (τ ) = L4 − U3 ,

(8)

where [L1 , U1 ] and [L2 , U2 ] denote the 95% confidence intervals of Pr(R = 1 | P < τ ) and Pr(R = 0 | P < τ ), respectively, [L3 , U3 ], [L4 , U4 ] correspond to Pr(R = 1 | P > τ ) and 4

Pr(R = 0 | P > τ ). Considering the PPL space to be meaningfully correlated with correctness at threshold τ only when both conditions ∆EiS (τ ) > 0 and ∆ErS (τ ) > 0 hold. Minimizing Classification Errors. In the middle and later RL stages, a wide range of thresholds may satisfy Advantage Judgment, necessitating the further selection of an optimal value. We formalize a classification task using PPL as the criterion: responses with a PPL below τ are classified as correct, while those above are deemed error. By minimizing classification errors (Err(τ ) in Figure 2) to find the optimal threshold: X 1 τ ∗ = arg min |r i − I(pi < τ )| τ |Q| (9) (ri ,pi )∈Q

s.t. ∆EiS (τ ) > 0, ∆ErS (τ ) > 0 where I(·) is the indicator function. Through advantage judgment and minimizing classification errors, the disentangled PPL spaces are highly correlated with correctness. Consequently, the hard groups in EiS and the easy groups in ErS would be the important samples for achieving the EETO. Algorithm details are illustrated in Algorithm 1. 3.3

Bidirectional Reward Reallocation

To implement a stable exploration-exploitation optimization and minimize the impact to verification rewards, BRR does not directly employ PPL as the basis for reward shaping, but introduces the maximum-PPL reward reallocation strategy. BRR thus aims to drive updates toward the exploratory (high-entropy) direction for the hard groups in EiS, and toward the exploitative (low-entropy) direction for the easy groups in ErS.. Theorem 1 (Entropy Increase with Maximum-PPL Reward). Let πt be the policy at step t. Given a query q and a group of outputs {oi } ∼ πt (· | q), if the output with the maximum PPL among {oi } is assigned a reward, then the average entropy of the updated policy πt+1 (· | q) increases. Reward Reallocation for Hard Groups. As shown in Figure 2, we define a hard group as q̂, {ôi }, {r̂ i }, {p̂i }, here {r̂ i } = {0}. If mean({p̂i }) < τ ∗ means that this hard group is located in EiS, the goal of the reward reallocation is to update it towards the high entropy direction. Concretely, the index m̂ corresponding to the maximum PPL in {ôi } is identified: m̂ = arg max1≤i≤G p̂i . Subsequently, the reallocated reward {r̂ ir } is constructed by setting r̂ m r = 1 while preserving all other original values, such that r̂ ir = r̂ i for all i ̸= m̂. Noted that if τ ∗ does not exist, or mean({p̂i }) > τ ∗, reallocated reward remains unchanged, i.e., {r̂ ir } = {r̂ i } = {0}. Theorem 2 (Entropy decrease with Maximum-PPL penalty). Let πt be the policy at step t. Given a query q and a group of outputs {oi } ∼ πt (· | q), if the output with the maximum PPL among {oi } is assigned a penalty, then the average entropy of the updated policy πt+1 (· | q) decreases. Reward Reallocation for Easy Groups. As shown in Figure 2, reward reallocation for the easy group is the opposite of that for the hard group. A easy group is defined as q̄, {ōi }, {r̄ i }, {p̄i }, here {r̄ i } = {1}. If mean({p̄i }) > τ ∗ means that this easy group is located in ErS, the goal of the reward reallocation is to update it towards the low entropy direction. Concretely, the index m̄ corresponding to the maximum PPL in {ōi } is identified: m̄ = arg max1≤i≤G p̄i . Subsequently, the reallocated reward {r̄ ir } is constructed by setting r̄ m r = 0 while preserving all other original values, such that r̄ ir = r̄ i for all i ̸= m̄. Noted that if τ ∗ does not exist, or mean({p̄i }) < τ ∗, reallocated reward remains unchanged, i.e., {r̄ ir } = {r̄ i } = {1}. Finally, we abstracted the peocess of BRR into a formalized reward function Rr , defined as follows:  i i {r̂ r } if {r } = {0}, i i Rr ({r }) = {r̄ r } if {r i } = {1}, (10)  {0} other, where BRR only performs reward reallocation for the easy groups and hard groups, and sets all rewards of the normal groups to zero, thus ensuring that the reallocated rewards (denoted as Rr ) and verification rewards (denoted as R) are orthogonal. Algorithm details are illustrated in Algorithm 2. Policy Optimization. R and Rr are orthogonal, which means that the samples with gradients under the two rewards are orthogonal. Thus, we use a hyper-parameter α to control the loss weight of Rr . 5

Table 1: Comparison of mathematical reasoning in ACC/mean@8 on 6 mathematics benchmarks. The best and second-best results are respectively marked in bold and underlined. Method

AIME24

AIME25

MATH

AMC

OLY

MIN

AVG

48.48 85.83 86.43 86.78 85.93

31.48 60.24 61.90 62.95 62.35

25.52 53.06 53.88 54.53 52.25

24.82 44.39 44.34 44.53 43.11

23.94 48.92 49.43 50.01 49.08

87.00

63.70

54.09

44.76

50.55

66.78 89.08 89.43 89.58 89.35

39.46 69.28 69.12 69.87 68.07

33.30 56.20 56.90 57.21 57.11

66.78 48.62 48.02 47.56 47.75

37.30 53.24 53.23 53.90 53.37

89.55

71.23

57.73

47.75

54.79

Qwen3-4B-Base Base model GRPO DAPO DAPO w/ EL CDE

9.58 26.67 26.25 26.67 26.67

3.75 23.33 23.75 24.58 24.17

DiPO (ours)

29.17

24.58

Qwen3-8B-Base Base model GRPO DAPO DAPO w/ EL CDE

8.33 31.67 30.08 33.75 31.67

9.17 24.58 25.83 25.42 26.25

DiPO (ours)

35.00

27.50

Qwen2.5-7B Base model GRPO DAPO DAPO w/ EL CDE

7.08 20.42 20.42 20.00 20.00

2.08 15.42 16.67 14.58 15.00

41.53 79.15 79.08 79.85 79.00

22.74 58.43 59.94 58.73 55.87

19.28 42.42 42.70 43.05 42.94

17.19 36.95 37.55 39.65 35.94

18.32 42.13 42.73 42.64 41.46

DiPO (ours)

22.92

16.67

80.35

60.09

43.72

37.59

43.56

The maximum optimization objective is as follows: JDiP O (θ) = JDAP O (θ, R) + α × JDAP O (θ, Rr ),

(11)

where JDAP O is the DAPO optimization objective in Eq. (3).

Experiment

4.1

Experiment Setup

To comprehensively evaluate the effectiveness of our proposed DiPO, we conduct experiments on two downstream tasks of LLMs: mathematical reasoning and function calling. Detailed configurations are provided below. Mathematical Reasoning. We evaluate DiPO on the mathematical reasoning task using the DAPO17K Yu et al. [2025] dataset for training. Model performance is assessed on five challenging benchmarks: AIME24, AIME25, AMC23 (denoted as AMC), MATH500 Hendrycks et al. [2021] (denoted as MATH) the OE_TO_mat_en_COMP subset from OlympiadBench He et al. [2024](denoted as OLY), and Minerva (denoted as MIN). These evaluations are designed to comprehensively examine the method’s effectiveness across diverse problem difficulties and formats. We perform a comparative analysis against several strong baselines: 1. GRPO Shao et al. [2024], 2. DAPO Yu et al. [2025], 3. DAPO enhanced with Entropy Loss (DAPO w/ EL) Williams [1992], and 4. CDE Dai et al. [2025] with PPL reward shaping. All mathematical reasoning experiments are implemented using the VERL Sheng et al. [2025] framework. To ensure a rigorous comparison, we employ three different pretrained base models: Qwen3-4B-Base, Qwen3-8B-Base Yang et al. [2025a], and Qwen2.5-7B Yang et al. [2025b]. All models utilize consistent training configurations and prompts, as detailed in Appendix D.1. Function Calling. We establish our experimental setup using the publicly available ToolRL Qian et al. [2025] framework as the primary baseline. Furthermore, we implement ToolRL augmented with DAPO (ToolRL+DAPO) for comparison. Since ToolRL does not utilize a binary verification reward, certain modifications were made when implementing TooRL+DiPO. 6

Table 2: Comparison of function calling in acc on BFCLv3 benchmark. The best and second-best results are respectively marked in bold and underlined. Method

Non-Live Acc

Live Acc

Base model SFT400 SFT400+PPO SFT400+GRPO PPO, Cold Start ToolRL+GRPO ToolRL+DAPO

42.52 69.29 78.29 76.21 82.42 81.58 82.19

53.96 41.40 58.76 64.15 67.78 73.78 69.43

ToolRL+DiPO

83.42

73.06

Base model SFT400 SFT400+PPO SFT400+GRPO PPO, Cold Start ToolRL+GRPO ToolRL+DAPO

66.02 69.29 83.90 80.69 79.33 86.17 87.10

53.51 41.40 51.84 46.51 63.17 74.90 76.31

ToolRL+DiPO

86.21

76.83

Multi Turn Acc

Relevance Detection

Irrelevance Detection

Overall

1.00 0.00 5.12 1.75 4.88 3.75 8.00

44.44 94.44 100.00 94.44 100.00 100.00 81.25

82.49 60.14 48.40 58.63 18.09 56.44 57.60

33.04 34.08 45.80 46.42 51.15 52.98 53.21

8.62

100.00

54.16

55.03

4.25 0.00 0.25 0.25 0.38 18.12 19.75

76.47 94.44 100.00 100.00 88.89 83.33 87.50

62.66 8.11 29.66 14.19 52.92 76.68 67.25

41.97 34.08 42.02 39.25 46.68 58.38 61.06

24.50

87.50

69.57

62.51

Qwen2.5-3B-Instruct

Qwen2.5-7B-Instruct

We conduct comprehensive testing on the BFCLv3 benchmark Patil et al. [2025], which provides a diverse set of function calling scenarios for thorough evaluation. Consistent with the TooRL setup, we use the same model architectures (Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct) and maintain identical training configurations and datasets across all compared methods, details in see Appendix D.2. 4.2

Comparison Results of Mathematical Reasoning

Table 1 shows the comprehensive comparison results of mathematical reasoning in ACC/mean@8 across 6 benchmarks evaluated on three base models, our proposed DiPO demonstrates consistent and superior performance enhancements over other reinforcement learning algorithms. Overall, DiPO achieves the highest average score (AVG) across all three model scales, with 50.55% for Qwen3-4BBase, 54.79% for Qwen3-8B-Base, and 43.56% for Qwen2.5-7B, surpassing all compared baselines. Specifically, on the more challenging AIME benchmarks, DiPO attains the best results in most cases, such as 29.17% on AIME24 and 24.58% on AIME25 with Qwen3-4B-Base, and notably 35.00% on AIME24 and 27.50% on AIME25 with Qwen3-8B-Base, indicating its strong capability in handling complex mathematical reasoning tasks. Furthermore, the performance gains are particularly pronounced in larger models like Qwen3-8B-Base, where DiPO outperforms the second-best method by a clear margin in AVG (54.79% vs. 53.90%), highlighting its scalability and effectiveness in leveraging model capacity. Notably, although DAPO with EL achieves highly competitive results, it demonstrates pronounced sensitivity to its configuration coefficients, as further examined in the Appendix E.3. 4.3

Comparison Results of Function Calling

Building upon the mathematical reasoning evaluation, we further assess the performance of DiPO on the function calling task. As presented in the Table 2, we use TooRL as the baseline and replicated TooRL+DAPO for comparison. The results demonstrate that DiPO delivers superior overall performance, achieving the highest Overall acc of 55.03% and 62.51% on the Qwen2.5-3BInstruct and Qwen2.5-7B-Instruct models, respectively. Notably, DiPO demonstrates exceptional capability in handling complex, multi-round interactions, as evidenced by its leading performance in Multi-Turn Acc. It achieves scores of 8.62% and 24.50% for the 3B and 7B models respectively, surpassing the second-best method (ToolRL+DAPO, with 8.00% and 19.75%) by 0.62 and 4.75 7

Table 3: The impact of hyperparameter α on performance. The best results are marked in bold. α

AIME24

AIME25

α = 0.0 α = 0.1 α = 1.0

26.25 29.17 25.83

23.75 24.58 24.17

MATH

AMC

OLY

MIN

AVG

61.90 63.70 62.05

53.88 54.09 54.01

44.34 44.76 45.50

49.43 50.55 49.66

69.12 71.23 70.03

56.90 57.73 56.71

48.02 47.79 48.07

53.23 54.79 53.36

Qwen3-4B-Base 86.43 86.93 86.38 Qwen3-8B-Base α = 0.0 α = 0.1 α = 1.0

30.08 35.00 32.08

25.83 27.50 24.58

89.43 89.55 88.68

percentage points. This comprehensive improvement over strong function calling baselines, validates the effectiveness and general applicability of the DiPO not only in mathematical reasoning. 4.4

Hyperparameter Analysis

The orthogonal design between the reallocated rewards and validation rewards enables precise, independent adjustment of DiPO’s influence via the hyperparameter α. Table 3 presents the performance variations of Qwen3-4B-Base and Qwen3-8B-Base models under different values of α. For both model scales, the performance tends to reach the optimal level when α is set to 0.1: 4B model achieves the highest average (AVG) performance of 50.55%, with significant improvements compared to α = 0.0 and α = 1.0; similarly, 8B-model attains the best AVG of 54.79% at α = 0.1. When α = 0.0, DiPO’s contribution is zero, leading to the lowest AVG performance for both models (49.43% and 53.23%). 4.5

Ablation Experiment and Discussion

Table 4 presents ablation results of the contributions of PSD and BRR on Qwen3-4B-Base and Qwen38B-Base on six mathematical reasoning benchmarks. PPL reward refers to using PPL directly as rewards: positive PPL for hard groups, and negative PPL for easy groups. Overall, the combination of PSD and BRR achieves the best performance for both 4B and 8B models, verifying their effectiveness. Detailed discussions are as follows. Discussion 1. Fine-grained exploration and exploitation make RL more effective. In Table 4, without PSD, all hard samples are indiscriminately driven toward high PPL, whereas easy samples are steered toward low PPL. PSD improves this by disentangling the space into ErS (high PPL) and EiS (low PPL), allowing for fine-grained optimization: easy groups within ErS are encouraged toward EiS, while hard groups within EiS are directed toward ErS. The results confirm the critical role of PSD in model enhancement. When using the PPL reward, PSD yields improvements of 2.39 and 1.19 points on the 4B and 8B models, respectively; with the BRR reward, the corresponding gains are even larger at 2.99 and 3.88 points. Significant improvements indicate that invalid exploration and exploitation are needed for optimization, while excessive encouragement of exploration and exploitation is detrimental. Discussion 2. Reward shaping should not cause significant changes to the intrinsic validation rewards. In Table 4, we compare our BRR against PPL reward. The experimental results confirm that BRR is superior to the PPL reward method. Concretely, with using PSD, BRR compared to PPL reward brings an improvement of 0.93 and 2.23 respectively on the 4B and 8B models, while using PSD and PPL reward is even worse than the baseline. Science the reward variance of easy and hard groups are both zeros, directly using PPL as a reward causes drastic shifts in the reward distribution, potentially destabilizing the training process. In contrast, BRR introduces the reallocated rewards maintain a distribution very similar to the original validation rewards, which allows BRR to leverage PPL signals for enhancement with the minimum impact on the validation reward landscape. 4.6

Quantitative Results

To further verify the properties of DiPO, we conducted a quantitative analysis of some data during the training process of DAPO and DiPO. 8

Table 4: Ablation study results on Qwen3-4B-Base and Qwen-8B-Base, showing ACC/mean@8 on six mathematical reasoning benchmarks. The best results are marked in bold. PSD

BRR PPL reward

AIME24

AIME25

MATH

AMC

OLY

MIN

AVG

86.43 85.10 84.60 86.35 86.93

61.90 59.19 61.45 62.35 63.70

53.88 49.37 49.09 54.00 54.09

44.34 43.06 41.82 44.21 44.76

49.43 47.23 47.55 49.62 50.55

69.12 65.36 64.76 68.83 71.23

56.90 53.62 53.45 56.65 57.73

48.02 46.23 45.63 46.30 47.42

53.23 51.37 50.86 52.56 54.79

Qwen3-4B-Base ✗ ✗ ✗ ✓ ✓

✗ ✓ ✗ ✗ ✓

✗ ✗ ✓ ✓ ✗

26.25 25.83 24.58 27.50 29.17

✗ ✗ ✗ ✓ ✓

✗ ✓ ✗ ✗ ✓

✗ ✗ ✓ ✓ ✗

30.08 30.08 27.92 30.83 35.00

23.75 20.83 23.75 23.33 24.58

Qwen3-8B-Base 25.83 24.58 25.42 24.58 27.50

89.43 88.35 88.00 88.15 89.55

Exploration-exploitation trade-off. Figure 3 shows the PPL distribution of the Qwen3-8B-Base after training on DAPO-17K with DAPO and DiPO. The results demonstrate that under the DAPO training, the PPL distribution of correct and incorrect samples exhibits a significant overlap. This phenomenon directly causes a large number of hard samples to lose exploration capability, thereby limiting the plasticity of RL. In contrast, DiPO shows superior distribution characteristics: error samples are more located in the high-PPL region, while correct samples are still concentrated in the low-PPL region, which achieves a balanced trade-off between exploration and exploitation in RL.

30 20

30 20 10

10 01.00

1.02

1.04

1.06

PPL

1.08

1.10

0 1.00

30 25 DiPO (Raw) DiPO (Smoothed) DAPO (Raw) DAPO (Smoothed)

20 15 10

1.02

1.04

PPL

1.06

1.08

1.10

200

400

Step

600

AIME25

Max: 35.00

Max: 30.83

ACC/mean@8 (%)

Frequency

Correct Error

ACC/mean@8 (%)

Correct Error

Frequency

AIME24

DiPO

DAPO 60

Max: 27.50 Max: 25.83

25 20 DiPO (Raw) DiPO (Smoothed) DAPO (Raw) DAPO (Smoothed)

15 10 0

200

400

Step

600

Figure 3: PPL distribution comparison of correct Figure 4: ACC/mean@8 curves of DiPO and and error samples for Qwen3-8B-Base trained DAPO (raw and smoothed curves) on AIME24 and AIME25 with using Qwen3-8B-Base model. on DAPO-17K Dataset via DAPO and DiPO. Higher upper bound for later training. Figure 4 presents the test curves of DiPO and DAPO on two AIME benchmarks during the training process. The results show that at the initial stage of training, there was no significant difference in the performance of the two methods; in the later stages of training, however, the performance curve of DAPO exhibited a significant slowdown in growth, whereas DiPO maintained a sustained capacity for exploratory improvement. Combined with the PPL distribution characteristics in Figure 4, this phenomenon can be further explained: DiPO enables hard samples to conduct more sufficient exploration, while the training paradigm of DAPO leads to a conservative tendency in the exploration strategy of hard samples.

Conclusion

This paper explores the exploration-exploitation trade-off between in RL training. We analyze the two ETTO dilemmas faced by the extreme hard and easy groups, and proposed the Disentangled Perplexity Policy Optimization. First, we develop a novel Perplexity Space Disentangling method to identify which hard samples require encouragement for exploration and which easy samples need promotion for exploitation. Then, we design a Bidirectional Reward Reallocation mechanism that incorporates PPL-based exploration and exploitation signals while minimizing disruptions to the original verification reward distribution. Finally, extensive experiments on mathematical reasoning and function calling tasks demonstrate the comprehensive superiority of the proposed method. 9

References Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2):235–256, 2002. Liang Chen, Xueting Han, Qizhou Wang, Bo Han, Jing Bai, Hinrich Schutze, and Kam-Fai Wong. EEPO: exploration-enhanced policy optimization via sample-then-forget. arXiv preprint arXiv:2510.05837, 2025a. Zhipeng Chen, Xiaobo Qin, Youbin Wu, Yue Ling, Qinghao Ye, Wayne Xin Zhao, and Guang Shi. Pass@ k training for adaptively balancing exploration and exploitation of large reasoning models. arXiv preprint arXiv:2508.10751, 2025b. Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective. arXiv preprint arXiv:2506.14758, 2025. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/2110.14168, 2021. Runpeng Dai, Linfeng Song, Haolin Liu, Zhenwen Liang, Dian Yu, Haitao Mi, Zhaopeng Tu, Rui Liu, Tong Zheng, Hongtu Zhu, et al. Cde: Curiosity-driven exploration for efficient reinforcement learning in large language models. arXiv preprint arXiv:2509.09675, 2025. Jia Deng, Jie Chen, Zhipeng Chen, Daixuan Cheng, Fei Bai, Beichen Zhang, Yinqian Min, Yanzipeng Gao, Wayne Xin Zhao, and Ji-Rong Wen. From trial-and-error to improvement: A systematic analysis of llm exploration mechanisms in rlvr. arXiv preprint arXiv:2508.07534, 2025. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, et al. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783. Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3828–3850, 2024. Haoyang He, Zihua Rong, Kun Ji, Chenyang Li, Qing Huang, Chong Xia, Lan Yang, and Honggang Zhang. Rethinking reasoning quality in large language models through enhanced chain-of-thought via rl, 2025. URL https://arxiv. org/abs/2509.06024, 2025. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021. Ang Li, Zhihang Yuan, Yang Zhang, Shouda Liu, and Yisen Wang. Know when to explore: Difficultyaware certainty as a guide for llm reinforcement learning. arXiv preprint arXiv:2509.00125, 2025. Jia Liu, ChangYi He, YingQiao Lin, MingMin Yang, FeiYang Shen, and ShaoGuo Liu. Ettrl: Balancing exploration and exploitation in llm test-time reinforcement learning via entropy mechanism. arXiv preprint arXiv:2508.11356, 2025. Shishir G. Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning, 2025. Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tür, Gokhan Tur, and Heng Ji. Toolrl: Reward is all tool learning needs. arXiv preprint arXiv:2504.13958, 2025. 10

John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889–1897. PMLR, 2015. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pages 1279–1297, 2025. Yuda Song, Julia Kempe, and Remi Munos. Outcome-based exploration for llm reasoning. arXiv preprint arXiv:2509.06941, 2025. Yi Su, Dian Yu, Linfeng Song, Juntao Li, Haitao Mi, Zhaopeng Tu, Min Zhang, and Dong Yu. Crossing the reward bridge: Expanding rl with verifiable rewards across diverse domains. arXiv preprint arXiv:2503.23829, 2025. Yan Sun, Jia Guo, Stanley Kok, Zihao Wang, Zujie Wen, and Zhiqiang Zhang. Efficient reinforcement learning for large language models with intrinsic exploration. arXiv preprint arXiv:2511.00794, 2025. Abraham Wald. Sequential analysis. Courier Corporation, 2004. Christian Walder and Deep Karkhanis. Pass@ k policy optimization: Solving harder reinforcement learning problems. arXiv preprint arXiv:2505.15201, 2025. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229–256, 1992. Rihui Xin, Han Liu, Zecheng Wang, Yupeng Zhang, Dianbo Sui, Xiaolin Hu, and Bingning Wang. Surrogate signals from format and length: Reinforcement learning for solving mathematical problems without ground truth answers. arXiv preprint arXiv:2505.19439, 2025. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025a. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025b. URL https://arxiv.org/abs/2412.15115. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025. Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.05118, 2025. Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.18071, 2025. Yang Zhou, Sunzhu Li, Shunyu Liu, Wenkai Fang, Kongcheng Zhang, Jiale Zhao, Jingwen Yang, Yihe Zhou, Jianwei Lv, Tongya Zheng, et al. Breaking the exploration bottleneck: Rubric-scaffolded reinforcement learning for general llm reasoning. arXiv preprint arXiv:2508.16949, 2025. 11

Proof of Section 3.3

A.1

Mathematical proof

Proof. Given a language model πθ with parameters θ. Given a query q, sample G responses {oi }G i=1 . Each response oi is a token sequence (oi1 , . . . , oiT i ). The model’s probability for token oit given context (q, oi<t ) = πθ (y | q, oi<t ) is denoted as πti (oit ). The probability of other tokens in position of the t-th token is denoted as πti (y), and the logistic of y is denoted as lit (y), y ∈ V and V is the vocabulary of πθ . For convenience, we do not consider the clip operation, the loss function is as follows: i

|o | G 1 X 1 X i πti (oit ) , where L(θ) = − Â G i=1 |oi | t=1 t πold (oit )

exp(lti (oti )) . i y∈V exp(lt (y))

πti (oit ) = P

(12)

Optimization gradient descent is θ′ = θ − η∇θ L, with small learning rate η. Define the token-level entropy for response i at token t: X X Hti = − πθ (y | q, oi<t ) log πθ (y | q, oi<t ) = − πti (y) log πti (y). (13) y∈V

y∈V

The average token entropy over all responses and time steps is: i

|o | G 1 X 1 X i H . Havg = G i=1 |oi | t=1 t

(14)

The following will prove that high PPL rewards and high PPL penalties can respectively increase and decrease Havg . Logit Change. Consider a single token of response oit for training, the loss contribution is πti (oit ) . The gradient of the loss with respect to logit lti (y) is: −Âit πold (oi ) t

π i (y)

t ∂(−Âit πold (y) )

∂lti (y)

= Âit

πti (y) i (π (y) − 1y=oit ) ≈ Âit (π i (y) − 1y=oit ). πold (y)

(15)

Using gradient descent with learning rate η, the logit update is: ∆lti (y) = η Âit (1y=oit − πti (y)).

(16)

Probability Change. According the multivariate Taylor Expansion, for softmax, the first-order change in probability is: ∆πti (y) ≈

X ∂π i (y)

! X

t ∆li (z) = πti (y)(1z=y − πti (z))∆lti (z) = πti (y) ∂lti (z) t z∈V z∈V

∆lti (y) −

πti (z)∆lti (z)

z∈V

(17) where πti (y) = P

exp(lti (y)) . Substituting ∆πti (y). i z∈V exp(lt (z))

! ∆πti (y) = πti (y)

η Âit (1y=oi − πti (y)) −

πti (z)η Âit (1z=oit − πti (z))

z∈V

(18)

! = η Âit πti (y)

1y=oit − πti (y) − πti (oit ) +

πti (z)2

z∈V

Entropy Change. According the Taylor Expansion, the first-order change in entropy Hti is: X ∂H t X i ∆Hti ≈ ∆πti (y) = − ∆πti (y)(1 + log πti (y)), i ∂π (y) t y y∈V

 where Hit = −

 X

πti (y) log πti (y) .

y∈V

(19) 12

Substituting ∆πti (y): " ∆Hti = −η Âit

πti (y)

1y=oit − πti (y) − πti (oit ) +

y∈V

πti (z)2

(1 + log πti (y))

z∈V

" πti (oit ) log πti (oit ) −

= − η Âit

πti (y)2 log πti (y) − πti (oit )

y∈V

πti (y) log πti (y) +

y∈V

πti (z)2

z∈V

πti (y) log πti (y)

y∈V

= − η Âit πti (oit ) log πti (oit ) −

X y∈V

πti (y)2 Hti −

πti (y)2 log πti (y) + πti (oit )Hti

y∈V

(20) Average Entropy Change. Ignore the impact of cross-context updates, the first-order change in token entropy over all queries is: " ∆Havg = Eq∼D

|o | G X X 1 X 1 X −η Âit πti (oit ) log πti (oit ) − πti (y)2 log πti (y) + πti (oit )Hti − πti (y)2 Hti i G i=1 |o | t=1 y∈V

Let Bti = πti (oit ) log πti (oit ) + πti (oit )Hti and Fti = − y∈V πti (y)2 log πti (y) − # " G h i −η X i i i i Â B + Ft E ∆Havg = Eq∼D G i=1 t=1:|o | t t P

y∈V

(21) i 2 i π (y) H t: y∈V t

(22)

m The response with maximum PPL of {oi }G i=1 is denoted as o ,

• Rewarding maximum PPL Âm t =

√

1 G − 1, Âit = − √G−1 .

  G X √ 1 −η    Et=1:|oi | Bti + Fti + G − 1Et=1:|om | [(Btm + Ftm )] ∆Havg = Eq∼D  − √ G G − 1 i=1 

i̸=m

√ −η G − 1 = Eq∼D Et=1:|om | [Btm + Ftm ] − Ei̸=m Et=1:|oi | Bti + Fti G √ −η G − 1 = E [Btm ] + E [Ftm ] − Ei̸=m Bti − Ei̸=m Fti G

(23)

We assume the difference in Ft is negligible as it reflects global distribution statistics that remain statistically invariant across samples in a large dataset, thus E [Ftm ] − Ei̸=m Fti ≈ 0. √ η G−1 (24) ∆Havg ≈ Ei̸=m Bti − E [Btm ] G √ i √ 1 • Penalizing maximum PPL Âm . t = − G − 1, Ât = G−1 √ η G−1 ∆Havg ≈ E [Btm ] − Ei̸=m Bti G

(25)

When e−1−Ht < oit < 1, Bti increases with πti (oit ). Moreover, since {oi } is sampled by the π with probability, and the smaller Hti is, the higher the sampling probability, we can approximately assume i that e−1−Ht < oit < 1 is basically satisfied. The conclusion is that rewarding maximum PPL then ∆Havg > 0 (entropy increases) and penalize maximum PPL then ∆Havg < 0 (entropy decreases). Import Statement. Given that the parameter space of LLMs is vast and complex updating, strict mathematical proof is unrealizable. The aforementioned proof is based on multiple idealized assumptions and only provide an approximate estimate of the trend in entropy change. In actual training, the theoretical results need to be further verified through experiments. 13

A.2

Experimental verification

To further verify the feasibility of the theory in section 3.3, we designed a verification experiment. Specifically, we used the Qwen3-0.6B Yang et al. [2025a] model and employed DAPO-17K Yu et al. [2025] as the training set, discarding the verification reward. Instead, we utilize max-PPL reward and max-PPL penalty as training reward, respectively, and recorded the changes in model entropy. As illustrated in Figure 5, the trend of entropy update is consistent with the proof.

Maximum PPL Reward 12.0

7.5 5.0 2.5 100

11.5 300

350

200

300

step

0.8 0.6 0.4 0.2

0.050

Entropy

10.0

Maximum PPL Penality

400

450

400

500

0.025 300

100

200

step

350

300

400

450

400

500

Figure 5: Entropy curves of maximum-PPL reward and maximum-PPL penalty trained on Qwen30.6B model with using DAPO-17K.

Related Works

B.1

Reinforcement Learning for LLMs

Policy optimization algorithms have evolved significantly to address the challenges of RL for LLMs (LLM-RL). Trust Region Policy Optimization (TRPO) Schulman et al. [2015] introduced KL divergence constraints for stable policy updates, laying foundational ideas. Proximal Policy Optimization (PPO) Schulman et al. [2017] simplified TRPO with clip-based objectives, becoming the standard for LLM-RL. GRPO Shao et al. [2024] optimized PPO for mathematical reasoning by a novel advantage calculation strategy, boosting efficiency and removing value networks. DAPO Li et al. [2025] further scaled LLM-RL with decoupled clipping and dynamic sampling, while VAPO Yue et al. [2025] enhanced reasoning reliability via hierarchical advantage estimation. GSPO Zheng et al. [2025] extended GRPO’s grouping strategy to sequence-level optimization for mixture-of-experts models and long-form reasoning tasks. B.2

Reward Shaping

Recent works on reward shaping for LLM-RL have advanced across key directions. CrossDomainRLVR Su et al. [2025] expanded verifiable reward RL to diverse unstructured domains via generative scoring. PKPO Walder and Karkhanis [2025] and Pass@k Training Chen et al. [2025b] optimized pass@k performance to enhance sample diversity and collective utility, addressing exploration limitations. rl-without-gt Xin et al. [2025] introduced format-length surrogate signals to bypass ground truth dependence. RuscaRL Zhou et al. [2025] leveraged rubric scaffolding to break exploration bottlenecks, while DACE Li et al. [2025], CDE Dai et al. [2025] proposed difficulty-aware certainty and curiosity-driven signals for adaptive exploration. DRER He et al. [2025] focused on reasoning quality with fine-grained CoT rewards, and OBE Song et al. [2025] mitigated diversity collapse via outcome-based exploration bonuses. B.3

Entropy and Perplexity-Driven Exploration

Entropy and perplexity are key signals for RL of adaptive balance between exploration and exploitation. Some advantage-enhanced methods Cheng et al. [2025], Deng et al. [2025] revisited entropy/perplexity as a signal, augmenting the advantage function to promote deep reasoning chains and boost Pass@K. Liu et al. [2025] extended entropy mechanisms to test-time RL via ETMR and EAR, enhancing efficiency and diversity. Dai et al. [2025] integrated actor perplexity and critic value variance as curiosity bonuses to mitigate premature convergence. Li et al. [2025] leveraged difficulty-aware certainty to dynamically modulate exploration, rewarding or penalizing confidence 14

based on task complexity. Chen et al. [2025a] introduced adaptive unlearning in two-stage rollouts to break entropy collapse loops.

Algorithm Details

Algorithm 1 Perplexity Space Disentangling Require: Batch of queries {q} from D, Policy πθ , PPL Queue Q 1: for each query q do 2: // G-Sampling 3: Generate outputs {oi }G i=1 ∼ πθ (·|q) 4: Compute verifiable rewards {r i } ← R(oi , a) 5: Compute perplexity {pi } using Eq. (4) 6: Update Q with pairs {(pi , r i )} 7: end for c c 8: Estimate probabilities Pr(R|P > τ ) and Pr(R|P < τ ) using Eq. (5) 9: Calculate candidate set C = {τ |∆EiS (τ ) > 0 ∧ ∆ErS (τ ) > 0} using (Eq. 7 Eq. 8) 10: if C ̸= ∅ then P 1 11: τ ∗ ← arg minτ ∈C |Q| (ri ,pi )∈Q |r j − I(pj < τ )| 12: else 13: τ ∗ ← None 14: end if Algorithm 2 Bidirectional Reward Reallocation Require: Optimal threshold τ ∗, A group of verification reward {r i } and corresponding PPL {pi } 1: Initialize reallocated rewards {r ir } ← {r i } 2: if τ ∗ is not None then 3: p = mean({pi }) 4: m = arg maxi pi 5: if ∀i, r i = 0 and p < τ ∗ then 6: // Hard Group in EiS 7: rm r ← 1 {Encourage Exploration} 8: else if ∀i, r i = 1 and p > τ ∗ then 9: // Easy Group in ErS 10: rm r ← 0 {Encourage Exploitation} 11: end if 12: end if

Detailed Experiment Setup

D.1

Mathematical Reasoning

The mathematical reasoning task uses the DAPO-17K Yu et al. [2025] dataset for 700 steps training on 4×8 A100 GPUs. For all comparison methods, all parameters are set from Table 5, and the remaining parameters use the default parameters of VERL Sheng et al. [2025]. For DAPO/w EL, use 0.001 as the coefficient for entropy loss, and the coefficient for entropy loss for the other methods is all 0. For the reproduced CDE Dai et al. [2025], their exclusive hyper-parameters all adopt the default settings from the original paper. The prompt template for the training set and test set is as: "[Question] Let’s think step by step and output the final answer within \boxed{}", [Question] represents a specific mathematical problem. D.2

Function Calling.

The baseline for Function Calling is TooRL Qian et al. [2025], which is the earliest open-source method to introduce GRPO into Function Calling. Its main contribution lies in a series of reward 15

Table 5: Training configurations of mathematical reasoning. train_batch_size 128 ppo_mini_batch 128 temperature 1.2 max_prompt_length 1K clip_ratio_low 0.2 n 8 data actor rollout max_response_length 4K clip_ratio_high 0.28 val_kwargs.n 8 gen_batch_size 256 kl_loss_coef 0 val_kwargs.temperature 0.6

designs related to function calling, where the reward is no longer a binary 0 or 1 but a range of [−3, 4]. The validation rewards of TooRL includes formatting scores and match scores, match scores further includes tool name matching, parameter name matching, and parameter content matching. In order to adapt to DiPO, we denoted the sample with the maximum reward as the correct sample, and other samples as error samples. During BRR, we use 4 and -3 for the correct and error reward reallocation respectively. We implement DAPO and DiPO on 8 A100 GPUs, the hyperparameter settings for DAPO and DiPO are shown in Table 6, which are consistent with TooRL except DAPO’s private parameters. The training dataset and system prompt are both consistent with TooRL. Table 6: Training configurations of function calling. train_batch_size 512 ppo_mini_batch 128 temperature 1.2 max_prompt_length 2K clip_ratio_low 0.2 n 4 data actor rollout clip_ratio_high 0.28 val_kwargs.n 4 max_response_length 1K gen_batch_size 1024 kl_loss_coef 0 val_kwargs.temperature 0.6

More Experiment Results

E.1

Results on Llama3.1-8B-Instruct

To further verify the generality of DiPO, we conduct experiments using Llama3.1-8B-Instruct Grattafiori et al. [2024] on GSM8K Cobbe et al. [2021] and MATH, as shown in Table 7. It can be seen that for Llama3.1-8B-Instruct, entropy loss and CDE are not effective methods, their results are lower than the baseline DAPO, especially for MATH; by contrast, our method shows moderate improvement. Concretely, DiPO achieves the best performance on MATH (56.75%) and the highest overall average (73.39%), demonstrating its consistent effectiveness across model families. E.2

Results of Majority Vote

The ACC/maj@8 metric, which determines the final answer by majority voting across 8 reasoning attempts, primarily reflects a model’s consistency and reliability in mathematical problem-solving. A higher ACC/maj@8 score indicates that the model can reliably generate correct reasoning paths, not just occasionally produce the right answer. As presented in Table 8, the proposed DiPO method yields the highest average performance across the six mathematical benchmarks for all three base models investigated. For the Qwen3-4B-Base model, DiPO achieves an average score of 56.41%, outperforming CDE—the second-performing method with a score of 55.51%—by 0.90 percentage points. It attains state-of-the-art performance on AIME24, AMC and MIN, while maintaining strong competitiveness on the remaining benchmarks. For the Qwen3-8B-Base model, DiPO reaches an average score of 60.65%, exceeding CDE (the Table 7: Comparison of mathematical reasoning in acc/mean@8 using Llama3.1-8B-Instruct as base model. The best results is marked in bold. Method

GSM8K

MATH

AVG

Llama3.1-8B-Instruct DAPO DAPO w/ EL CDE

79.80 89.89 89.66 89.01

47.83 55.75 52.68 53.45

63.82 72.82 71.17 71.23

DiPO (ours)

90.03

56.75

73.39

Table 8: Comparison of mathematical reasoning in ACC/maj@8 on 6 mathematics benchmarks. The best and second-best results are respectively marked in bold and underlined. Model

AIME24

AIME25

Base model GRPO DAPO DAPO w/ EL CDE

13.30 36.67 33.33 33.33 36.67

6.67 26.67 30.00 30.00 30.00

DiPO (ours)

36.67

30.00

MATH

AMC

OLY

MIN

AVG

64.00 89.60 89.60 90.40 89.80

46.99 68.67 69.88 68.67 69.88

37.59 58.10 59.29 60.03 57.06

30.15 50.37 49.26 50.40 49.63

33.12 55.01 55.23 55.47 55.51

90.20

72.29

58.59

50.70

56.41

Qwen3-4B-Base

Qwen3-8B-Base Base model GRPO DAPO DAPO w/ EL CDE

16.67 36.67 36.67 40.00 40.00

13.33 33.33 33.33 33.33 33.33

80.20 92.40 92.40 92.00 93.20

59.04 77.11 77.11 78.31 75.90

44.57 62.11 61.07 61.37 62.11

36.76 54.41 53.68 52.21 53.68

41.76 59.34 59.04 59.54 59.70

DiPO (ours)

43.30

33.33

92.60

79.52

62.90

52.25

60.65

Qwen2.5-7B Base model GRPO DAPO DAPO w/ EL CDE

6.67 26.67 30.00 30.00 23.30

3.33 26.67 23.33 20.00 20.00

52.20 84.40 83.40 83.40 85.00

27.71 65.06 67.47 63.86 63.86

25.85 46.95 47.70 48.14 49.33

20.59 41.18 41.91 43.01 41.18

22.73 48.49 48.97 48.07 47.11

DiPO (ours)

33.33

23.33

84.00

67.47

49.44

41.18

49.79

second-best method with 59.70%) by 0.95 percentage points and securing the top performance on four benchmarks, namely AIME24, AIME25, AMC and OLY. For the Qwen2.5-7B model, DiPO registers an average score of 49.79%, outperforming DAPO—the second-ranked method with 48.97%—by 0.82 percentage points and ranking first on three benchmarks: AIME24, AMC and OLY. Collectively, DiPO attains the top performance in 10 out of the 18 benchmark-model pairs, which validates its consistent and superior capability in generating reliable mathematical reasoning processes. E.3

Coefficient Sensitivity Analysis

As reported in Table 9, we analyzed the parameter sensitivity, which demonstrates the distinct robustness characteristics between the proposed DiPO and entropy loss. For the Qwen3-8B-Base model, DiPO with a coefficient of 0.10 achieves the optimal average score of 54.79%, representing a +1.56 improvement over the DAPO baseline (53.23%). Notably, even with a tenfold larger coefficient of 1.00, DiPO maintains a stable AVG of 53.36%, which remains marginally above the baseline. The performance variation across this tenfold coefficient change is only 1.43 AVG points. In contrast, the entropy loss exhibits significantly higher sensitivity; a small coefficient of 0.001 yields a modest gain (AVG 53.90%, +0.67), increasing it merely to 0.01 causes severe performance degradation, dropping the AVG to 46.00%—a decrease of 7.90 points and 7.23 points below the baseline. This drastic collapse is consistent across all benchmarks; for instance, on the AMC dataset, performance plummets from 69.87% to 56.33%. A similar pattern is observed with the 4B model, where DiPO’s performance remains stable between coefficients of 0.10 and 1.00 (AVG 50.75% vs. 49.66%), while entropy loss at 0.01 causes the AVG to fall to 42.47. The results indicate that DiPO provides a much wider and more forgiving effective coefficient range, offering reliable performance gains without the risk of catastrophic collapse, thereby presenting a more robust and practical RL strategy. E.4

Results of Risk Prediction

In the contemporary landscape of digital platform management, real-time public opinion risk prediction is a critical capability for safeguarding the operational integrity and brand reputation. This task involves a complex analysis of user-generated content to identify potential crises across various 17

Table 9: The impact of the coefficients for DiPO and entropy loss on the results. Mathod

coeff

AIME24

AIME25

MATH

AMC

OLY

MIN

AVG

Qwen3-4B-Base Baseline

0.00

26.25

23.75

86.43

61.90

53.88

44.34

49.43

DiPO

0.10 1.00

29.17 25.83

24.58 24.17

87.00 86.38

64.91 62.05

54.09 54.01

44.76 45.50

50.75 49.66

DAPO /w EL

0.001 0.01

26.67 18.33

24.58 20.00

86.78 81.12

62.95 50.00

54.53 45.99

44.53 39.38

50.01 42.47

Qwen3-8B-Base Baseline

0.00

30.08

25.83

89.43

69.12

56.90

48.02

53.23

DiPO

0.10 1.00

35.00 32.08

27.50 24.58

89.55 88.68

71.23 70.03

57.73 56.71

47.75 48.07

54.79 53.36

DAPO /w EL

0.001 0.01

33.75 22.50

25.42 20.42

89.58 84.18

69.87 56.33

57.21 49.49

47.56 43.06

53.90 46.00

dimensions, such as product experience, fraud, and regulatory compliance. It can be realized through LLMs guided by a specialized prompt that mandates a “risk-first” priority to ensure that no negative sentiment is overlooked. As detailed in the 6, the LLM is instructed to maintain strict standards for “no-risk” classification, where any ambiguity or negative indicator automatically triggers a risk label. Due to confidentiality reasons, the content within the brackets “[]” in the prompt is replaced with placeholders. To evaluate our approach in risk prediction, we constructed 4000 data (3000 for training and 1000 for test), and conducted a comparative analysis of DiPO and DAPO, trained on Qwen3-8B model Yang et al. [2025a]. As reported in Table 10, the experimental results reveal that Qwen3-8B achieves only a 52.06% accuracy, our DiPO method achieves state-of-the-art performance with the highest Accuracy (78.37%), Recall (79.49%), and F1 Score (86.84%). Although the DAPO model maintains a slightly higher precision of 95.96%, DiPO’s superior recall is particularly vital in an emergency analysis context, as it minimizes the likelihood of missing critical risks while still maintaining an exceptionally high precision of 95.69%. This balanced performance demonstrates that DiPO is the most robust and reliable framework for automated public opinion monitoring, providing an effective tool for identifying and mitigating potential threats in a complex information ecosystem. You are a professional Public Opinion Emergency Analyst for [The Company]. You need to analyze given public opinion information related to [The Company]. Based on your role and the standards provided below for risk/no-risk labels, determine whether the public opinion contains a risk, and provide the specific category and the basis for your judgment.

Notes:

- Prioritize identifying risk labels. The criteria for "No Risk" categories must be strictly met and the information must be non-negative. If the content meets

both risk and no-risk criteria, or is in an ambiguous state, classify it as a "Risk" category.

- Ensure the strict accuracy of "No Risk" judgments.

- Ensure the output format is correct for subsequent data processing and auditing.

- Please conduct your reasoning step-by-step. Wrap your final answer in <answer></answer>, where the final answer is risk for risk or norisk for no risk.

Public Opinion Content: [user-generated content]

Risk Categories and Criteria:

- [Risk name]: [Risk description].

- . . .

- Other Risk: Other negative public opinion involving [The Company].

No-Risk Categories and Criteria:

- [No Risk name]: [No risk description].

- . . .

- Others: Positive public opinion related to [The Company] that does not involve regulatory department.

Figure 6: Prompt of risk prediction. E.5

Visualization of PPL Distribution

To further analyze the exploration and exploitation trends during the RL training process, we conducted a visualization analysis of the PPL distribution during the training of DAPO and DiPO. As shown in Figure 7, during the initial training stage, the PPL distributions of DAPO and DiPO are relatively consistent, and it is difficult to distinguish the PPL distributions of correct and error samples, 18

Table 10: The results of risk prediction, including accuracy, precision, recall, and F1 score, and all results are the mean of 8 independent inferences. Method ACC/mean@8 Precision/mean@8 Recall/mean@8 F1/mean@8 Qwen3-8B DAPO DiPO

52.06 76.94 78.37

87.18 95.96 95.69

54.64 77.58 79.49

67.18 85.80 86.84

Figure 7: PPL distribution of correct and error samples for Qwen3-8B-Base trained on DAPO-17K Dataset via DiPO (TOP) and DAPO (Bottom). which is the motivation for introducing advantage judgment in PSD. In the later stages of training, the PPL distributions of DAPO and DiPO gradually converge. The difference is that for DAPO, the overall PPL will converge to a lower range (whether it is error samples or correct samples), while PPL distribution of DiPO is more discriminative, with error samples in a higher PPL range and correct samples in a lower PPL range, reflecting a reasonable tendency for exploration and exploitation. E.6

Case Analysis

Figure 10 8 11 9 show some cases of extreme groups for DAPO and DiPO trained to the 500-th step on DAPO-17K, where darker colors indicate higher entropy. It can be seen that under correct answers, the overall entropy of DiPO cases is smaller, and under incorrect answers, the density of high-entropy tokens in DiPO is greater, which also reflects the effective exploitation and exploration balance of DiPO.

Query: Roslyn has ten boxes. Five of the boxes contain pencils, four of the boxes contain pens, and two of the boxes contain both pens and pencils. How many boxes contain neither pens nor pencils? Let's think step by step and output the final answer within boxed{}. Alright, pens, a

and

**Total

have

two

typical

of theory

number

pencils:**

2\n\nWe

diagram

might

boxes:** need

help

**pens**.\nand

pencils

boxes

with

boxes

pens

Both\n\nNow, boxes

Pencils\n\nTotal pencils

with boxes

Both:

Final

Answer\n\nAfter

2\n-

Neither:

has

and

How

many

step.\n\n###

Understanding

**Boxes

contain

that

contain

pencils.

down

how

many

boxes

two

circles

that

area

represents

the

overlapping

and

boxes

the

boxes

10.\n\nBoxes -

3\n\nAdding through

that

total

Only

Pens\n\nSimilarly,

with

boxes

both

have

with

total

pens

and

boxes

with

only

pens

**only**

**or**

pencils

(only

pens)

with

pens

both

up:

the

Problem\n\nI

that:\n\n-

**or** +

both

(both)

7.\n\nTherefore,

Work\n\nLet's

make

10.

That

number

boxes

that

contain

would =

sure

matches neither

2.\n\nSo,

total

Figure 8: Correct case of DAPO. 19

nor

number pencils

think

circle

boxes =

with

Boxes

pencils

Boxes

with

Total

boxes Only

boxes.

So,

with

that

only

Pens

with

total

Pens

with

3\n-

seems

only

Neither

boxes

Venn

boxes

**both**

with

boxes

pencils:

with

both.\n\nGiven

Calculating

up:\n\n-

using

pencils

only

and

represents

boxes

**only**

seems

information:\n\n-

pens

other

Finding =

adds

with

contain it

the

both

pencils

boxes

with

**neither**

pens

number with

**2**.\n\n###

everything the

boxes.\n\n###

with

The

boxes

pens

be:\n\nBoxes

**7**

boxes

The boxes

boxes

glance,

the

pens

the

first

Visualizing **pencils**.\n-

both

with

summarize contain

with

problem,

pencils.\n\nGiven

let's that

boxes

pens

the

with

four

pencils?**\n\nAt

**Boxes

and

them

4\n-

Pencils\n\nTotal

pencils)

the

pens:**

boxes

pencils,

nor

Problem\n\nFirst,

Only

**either**

pencils

contain

pens

with

(only

Double-Checking

boxes

the

Boxes

with

the

neither

pencils.\n\n###

both

have

**nor** represents

**3**.\n\n###

that

pens

contain

circle

boxes

Five

boxes

One

Finding

pencils

2.\n\nSo,

boxes.

**neither**

2.\n\n###

that

that is,

contain

ten

5\n-

overlap:\n\n-

boxes

area

step

pencils:**

out

with

working

me:\n\n**Roslyn

draw

Boxes

of pens

break

find

both.\n\nThat

both.\n\nSo,

2\n-

**Boxes

both.\n\nGiven

or +

overlapping

Finding

front both

try

Let's

2.\n\nSo,

with

Pencils pens

The

contain I'll

10\n-

here.

pens

**3**.\n\n###

problem

boxes

problem.

with

this

the

set

nor

pens

Only

pens:

correct.\n\n###

is:\n\n\[\n\boxed{3}\n\]<|endoftext|>

Query: There is a collection of $25$ indistinguishable white chips and $25$ indistinguishable black chips. Find the number of ways to place some of these chips in the $25$ unit cells of a $5times5$ grid such that:each cell contains at most one chipall chips in the same row and all chips in the same column have the same colourany additional chip placed on the grid would violate one or more of the previous two conditions. Let's think step by step and output the final answer within boxed{}. Alright, grid

following

colors.\n-

problem Let's

\(5

\times

5\)

the

same

color.

grid

would

violate

one

the

rules.\n\n**Understanding

cell

has

white

without

than

all

black.\n4.

**Any

additional

$j$.\n\nWait, So,

row is

black,

and

they

impossible. to

it's

column

all

this.\n\n**Simplifying

it's

either:\n\n-

chip,

the

perhaps

All

easier

row

the

way

all

black,

All

and

cannot

have

chip.\n\nWait,

white

chip.\n-

\(R_i

that,

the

sum

the

Number

can

the

white of

black and

Black,

it's

assigned

color

\(R_i

better

white

number

chips,

and

there

Empty.

So,

there

are

chips

(white

and

black)

does

not

exceed

better

$e_c$

number

chips

\(w_r

25\).\n\nAlso, row

columns

\times

the

(to

this

seems

and

column

maximize

the

and is

prevent

e_c\).\n\nBut each

assigned

w_c\),

placement

column

for

the

complicated. entirely

White\n-

color

cells

where

both

can't

add

consider

one

Black

have

White

\(w_r

b_r

maximum

number

b_c

25\).\n\nBut

\leq

columns,

the

sure.\n\nAfter total

number

number the

final

the

rest

are

Black set

|B_r|

there

not

least

then

think the

\(R_i

\neq

Otherwise,

\text{White}\)

and

column

number

$b_c$

doesn't

verify:**\n\nEach total

equal

ways exceed

the

available

the

rows

can

number

ways

\(2^5

$R_i$ can

number

placed).\n\nWait,

perhaps

it's

Black\n-

$e_r$

columns

assigned

\(w_r,

b_r,

e_r\)

each

row

entirely

one

color

that

column to

both

total

|W_c| Empty,

|B_c|

must

column,

have

and

the

rest

row of

least

one

Black

the

White, \times

the

complexity, ways

calculating,

non-Empty

(25

and

chips

total

number

Empty,

and

similarly

2^5

1024\),

but

considering

Figure 9: Error case of DAPO.

rows

perhaps

b_r

b_c\),

valid

for

columns.

the

but

|W_c| you better

least and

the w_c

rows

columns

Black

it's

must

available),

where

are

(so

$2^{10}$, and

|W_c|\),

\times

answer

subset

|W_r|

configuration

the

White

\(w_r

maybe

black

get

25.\n\nThus,

each

colors

black

and

for

Black,

is:\n\n\[\boxed{512}\]<|endoftext|>

b_c, empty,

column

there w_c

that

$B_r$

row.\n\nWait,

\(w_r

cell.\n\nTo

\times

Black

white

\(w_c,

Choose

White

where

column,

chip

white

\(|W_r|

one

that

assign the

chips

white of

are

b_c

and

Empty.\n-

\]\n\nAnd:\n\n\[

number

\leq

balance

|E_c|

|B_c|\),

least

number

one

number

b_r

\times

must

maximum key

chips

Black\n-

w_c

chip

column

White

\(|B_r|

each

total

and

\]\n\nThe

least

the

there

but

chips:

color

\(w_r

Black,

number

Black

not

the

colored, to

\]\n\[

there

that

determines

chips

$C_j$

must

and

rows,

the

values

number

there

chips.\n\nAfter

the

number

possible

each.\n\nGiven to

has

empty,

exceeding.\n\nPerhaps

j)\)

has

not

must

\((i,

over

Approach:**\n\nConsider

e_c

cell

total

assigned

number

the

chips

rows =

then

columns,

that

it's

\(C_j

the

ensure can

column

$j$,

iterate

that

the

chips

of 5

conditions.\n\n**Calculating

the

chip

number

the each

each

column,

black

\text{White}\).\n\nSimilarly,

colors.\n\nFor

j)\)

total

for have

additional =

violating

has

column

cell

\((i,

simpler

each

color

C_j\),

then

So, all

choices.\n\nWait,

the

each

cell

\(C_j

it's

for

\text{Empty}\)

\text{Black}\).\n\nThe without

without

\text{White}\),

chip.\n-

that

For

color.\n-

perhaps

and

independent

$i$

\(R_i

are

chips

row

\(C_j

which

column

\text{Empty}\}\).\n-

number

available

terms of

black,

25,

any

have:\n\n-

for

Empty\n\nSimilarly,

color

total

all

chips.\n\nThe

empty.\n\nWait,

and

have

answer

white

the

there

and

both

that

maximized

color.

chips

Black

cells

Black

and

same

column

Empty,

since

the

same

b_c

one

5\).\n\nAlso,

and

the

Insight:**\n\nNotice

number

that

the

are

all

involved.\n\n**Simpler \leq

are

column

and

column

chips.\n\nThe

column at

row

that

white

color

its

number

columns.\n\nSimilarly,

must

that

the

So,

same

and

w_c

where

$i$

total

cells

row

chips

row

White\n-

The

all

row

placed

subset

|E_r|

the

are

both

\]\n\[

possible

Black

White

\(C_j

White,

Empty.\nget

total

and

b_c

reduces

the

columns

rows

maximality:\n\n-

Similarly,

row,

and of

chips

row

the

let's

512\).\n\nThus,

\(w_c

rows

all

black

$b_r$

column

that

chips

and

assigned

e_r

simpler

add

b_c\).\n\nThe

way.\n\n**Alternative

subset

|W_r|

and

it's

intersection

the

\times

is:\n\n\[\boxed{512}\]\n\n**Wait,

ensure

quite

b_r

\(b_r row

columns

chip

can't

that

(no

another

you

and

maximal

all

assignments.\n\nHowever,

25.\n\nWait,

White

row).\n-

5\), is

recall

perhaps

better

column

have:\n\n\[

getting

also

chips

that

\leq

and

$W_c$

non-Empty is

chips

number some,

each this

Black,

row

w_r

every

The

many

and

\]\n\nAlso,

the

chip

that:\n\nFor row.\n\nBut

rows,

columns.\n\nWe

for

empty.

want

that

there's

chips

there).\n\nWait,

number

have:\n\n\[ black

meaning chip

\leq

and

either

these

and

satisfy

assignments.\n\nSimilarly,

assigned

subset

Black

colors

row

rows

|B_c|

row

the

White,

are

for

\times

can

number

$B_c$

choices

possibilities

chip

all

any)

maximal;

white,

either

that

will

\(R_i

additional

$w_c$

|B_r|

Choose

that

with

$w_r$

we'd

rows,

\text{Black}\)

columns:\n\n-

way:\n\n-

White

j)\)

this

\((i,

placement

chips,

set

of \(R_i

the

has

that

The

j)\)

each,

\text{White}\)

\((i,

number

$W_r$

column

think

chips.\n-

number

cell

j)\)

with

such

$3^5$

\(R_i

following chip:**

any)

grid

grid one

the

its

the the

the

placement

most

(if

black\n-

chip

on at

chip

All

chips

contains

every

white\n-

better

number

j)\)

the

these

must

\((i, a

cell

column,

\text{Black},

has

given

chips

placed

given

the

cell

some

All

(if

\{\text{White},

cell

then

it's

2. chip

intersection

all

5\) their

chip.

All say,

\times within

additional

chip

have

determined

the

chips.\n\nLet’s columns

the

\((i,

for

Maybe

one

Empty\n\nWe

that:\n\n-

number

maximal, adding

\text{Black}\),

are

perhaps

\in

the

is,

possibilities

denote:\n\n-

Empty\n\nSimilarly

think

must

$3^5$

then

chips

determine

C_j\),

\(C_j

the

would their

row

j)\)

\text{Empty}\}\).\n\nThen,

\(R_i

parameterize.\n\nLet’s

$i$,

assigned

Wait,

Black,

rows

it's

\((i,

one

Any

**Each

row, all

here:

\(5

indistinguishable

most 4.

determined

each

chips

all

every

column,

either:\n\n-

constraint:

j)\)

interaction

color,

\((i,

opposite

there's

the

some

chips,

row,

any

chips

place

black

any

column.\n\nGiven

row,

each

White,

the

because

chip.

and

So,

chip

it's

let's

that

each

chip

number

Empty.

have

for

place

separately.\n\nFor

there's

chips

black

for

\text{Black},

means

position

columns

can 25

color.

saying.\n\n1. color:**

Similarly,

chips.\n\nSimilarly,

have

ways are

same

column

that

cannot

rules

and

chips.\n\nBut

column

contain

same

This

chip.\n-

perhaps

can

the

configurations

color

\text{Black}\)

Ways:**\n\nFirst,

White,

row

has

total

and

cell

Each

the

each

cell

and

consider

column

\{\text{White},

number

rows

has

color:**

chip

and

valid

ways

white

the

that

\in

total

Empty\n\nBut

For

the

terms

color

chip.\n\nGiven

row of

cannot

realize

many 25

number

conditions:**

color

cell

denote:\n\n-

the

row

how

must

have

same

the

that

row

then

the

then

that

the

all

cell,

\text{Empty}\),

violate

out

**Chips:**

column

row

have

assigned

\(C_j

the what

has

$j$.\n\nLet's a

find

same

black\n-

black,

one

**Rules:**

understand

all

figure

same

the

grid

the

count

color,

think

cells.\n-

column

the

same

Problem:**\n\nLet's

white\n-

intersection an

need

Statement:**\n\n-

need

would

need

about

white,

let's

same

chip I

the

white,

all

intersect

Therefore,

are

better

either either

unit

chips

the

think

column

perhaps

each

rules.\n\nOkay,

and

conditions.\n\nI

all

All

**All

chips

me,

Problem:**\n\nFirst,

chip.\n2.

**All

chips

with 3.

black.\n3.

the

down.\n\n**Problem

grid

all

breaking

front it

above

the

one

conditions.\n\n**Approach:**\n\nLet's and

break

must

row

this rules.

**Grid:**

same

above

have

certain

b_r

and I'm

such

not that

the

configurations However,

constraints,

it's

theEmpty actually

\(2^9

out

how

many

contain

pencils

5\n-

contain

neither

pens

find

principle

boxes

that

with

pens}

\]\n\nSo, nor

problem

boxes Boxes

nor

about

contain that

Roslyn

neither

contain

having

pens

pencils.\n\nAlright,

need

let's

find

out

both

is:\n\n\[

pencils

\text{Number

boxes

contain

is:\n\n\[

neither

pens

both}

\]\n\nPlugging

either

pens

pencils

nor

\text{Boxes

boxes}

pencils

boxes.

Some

4\n-

Boxes

that

contain

out

how

many

boxes

contain

find

many

Let's

contain

pencils.

how

with

\text{Total

pens

ten

nor

inclusion-exclusion.\n\nFirst,

pencils

contain

this

contain -

have

\text{Number in

the

with

pencils,

contain

pens,

Total

both

pens

and

pencils

boxes

have

neither

pens

nor

either or

are

pens

some

down.\n\n**Given:**\n-

and

number

pencils.

pencils both}

\text{Number

with

pencils

pens

total,

the

both.\n\nThe

number 3

do formula

both}

\]\n\nTherefore,

boxes

need

Boxes

can

that

use

the

for

the

number

pencils}

\text{Number

boxes

both.

10\n-

this,

with

contain =

Number

both}

few boxes

2\n\n**Find:**\n-

pens

boxes

pens

\text{Number

there

pencils

numbers:\n\n\[

both.\n\nSince with

break

that

the

contain

number

neither

pens

boxes

that

3.\n\n\[\boxed{3}\]<|endoftext|>

Figure 10: Correct case of DiPO.

grid

same of

have

this

problem

about

placing

\(5

\times

5\)

white

chips

and

chips,

all

indistinguishable.

need

such

that:\n\n1.

contains

most

color.\n3.

ways

allowed

Any

have

means

that

above

conditions:**

and

chip

placed

let's

break

one

column,

and

black

row,

it's

row,

placement

such

"bipartition"

chips

**Rows:**

either

that

25 or

can

model

leave

empty.

Similarly

matrix

where

each

row

and

rows

and

columns

the

better \):

fix

number

number for

the

black\n\nGiven

that

number

white

number

and

Reasoning\n\nAfter be

all

the

number

white,

ensuring

white

re-evaluating,

rows

are

and

are is

some

columns:\n\n\(

r_w

number

white

grid

thought,

the

black,

empty.

Similarly

maximality,

the

total

the

number

the

opposite

valid

configurations of

1024.\n\nBut

for

columns.

chips

However,

number

valid

ways

actually

the

configurations \(

chips

denote:\n\nthat

are

r_{\empty} columns, white

all

interactions is

\boxed{252}

exceed

wait,

let's must

the the

\).\n\n**Final

number

\)\n\n\(

c_w

c_b

c_{\empty}

the

also

into

each

need

rows

all

color,

but

with

grid

such

regions.\n\n###

ways

to the

rows

choose

and

of after

careful

For

each

for

all

chip

limits.\n\nAfter

accounting

that

doesn't

right.

$\boxed{252}$<|endoftext|>

Let's

the

rows

these

rows

think

the

Final

and

the

total

sizes.\n\nGiven

subsets

\).\n\nWait,

seem

and

columns.

constraints

r_b

tangled.\n\n### blocks

chessboard

have

\)\n\nwhere

times

varying

it's

all

the

number

are

the

available.\n\nGiven

black,

that

into

Ways\n\nLet's

white

getting

one

ways

white\n-

monochromatic

entirely

similar

all

that =

each

perhaps

all

ensure are

this

are

columns

that

separate

region

that

partition

Figure 11: Error case of DiPO.

rows

columns).\n\nWait,

divided

where

similar

\):

number

black

grid

Answer:**

white,

c_b

respect

all

white

For

the

chips.\n\nWait,

number

verify:\n\nWe

\boxed{252}

be:

can

available

\):

rows the

Counting

r_w

regions

ways

number

white\n-

respectively.\n\nBut

chips

separate

the exceed

each.\n\n###

columns.\n\nThis

the

and

black)

most

empty.\n\nThis

doesn't

and

(white,

corresponds

number which

denote:\n\n-

rows

color

chips

white

\),

black

the

total

between

choose

and

with

the

Approach\n\nLet's

each

none,

number

that

2^{10}

total

number

choices.\n\nThis

imply

rows

color

intersect

the

for

one the

rows

pattern

lies

rows

color

that

about

color

conditions

ensure

all

think

Similarly,

the

into

color

either

the are

columns.\n\nThis

solution

and

partitioned

answer

number

ensuring is

the

are

not

the

there

the

number

r_b

rows

total

the

ways

empty

violate

perhaps

columns

rows

number

Interpreting

(the

white

25.\n\nThe

that

black,

constraints,

non-empty.\n\nLet's

(since

all

correct

are

number

would

all

the

Each

black

valid

respect

This

columns

doesn't

realize

the

must

color:**

same

**Columns:**

consistency

cells)

we're

Problem\n\nLet's

sets.\n\n###

row,

the

So,

the

ensure

rows:\n\n-

c_w

chips

\times the

find

chip

\(5

rules.\n\n###

white,

have

chip:**

chips.\n-

determine

each

columns

and

have

of to

the

all

uniformity

for

total

terms need

that

columns.\n\nFor

white

maybe

in we

also

the

better

rows

and

all

numbers,

think

can

that

number

condition,

column.\n\nBut

are

row/column,

rows

maximality

each

the

additional

either

column

one

have

Statement:**\n\nWe

cells

need

most

column

Simplifying

there

perhaps

breaking

are

violate

most

the

that,

first.\n\n**Problem

**Any

without

colors.\n\n###

black,

Problem\n\nConsider

empty.\n\nGiven

number

with

same

chips

would

"empty."\n\nBut

better

each the

pool

and

chips

the

contains

columns.\n\n3.

ensure

the

columns,

it's

for

but

white

chips.\n\nGiven

perhaps

it's

the

tiling,

either

for

cell

and

colors

need

\):

chips

same

total

that

are

row

the

black

intersection

columns

c_{\empty}

are

Modeling

and

that

kind

chips

Similarly

black\n-

separated the

specific

all

remaining

rows

and

conditions.\n\nWe

the

**Each

the

columns

and

assign

two

from

are

Approach\n\nPerhaps

are

symmetry

that

row

Conditions\n\n1.

chip

monochromatic

such

same

any

that

columns.\n\nGiven

regions

ways

the

column,

number

there

and

column

any

it.\n\n###

columns.\n\nBut

rows

r_{\empty}

Alternative

for

empty.\n-

chips

similarly

each.\n\nWait,

black,

all

same

add

black,"

exceed

and

the

each

"all

unit

and

white,"

all

and

"all

doesn't

the

can't

all

color,

being

into

and

chips

row

rows

column

understand

each

and

to these

maximal;

there

white,

about

same

try of

chips

maybe

empty.\n\nSimilarly

the

**All

Let's some

the

are

all

think

one

grid. place

row,

violate

Understanding

there

chips

black,

either

colors

are

All

complex,

assign

would

cell.\n\n2.

that

adding

each

seems

chip.\n2.

Patterns\n\nSince

row

used

any)

seems

white

grid

placement

one

single

(if

the 3,

are

each

chips

that

the

any

and

Possible

chips

down.\n\n###

something

chips

chip all

implies

separately.\n\n-

all

think

given

This

the

columns.\n\n### can

than

conditions

columns

cell

additional

any

Moreover,

reminds

Each

this.\n\nOkay,

Conditions\n\nGiven chips.

black

and

columns

consideration, row,

can

constraints

and

again.\n\nUpon

Record · ID 14032 · SHA-256 4d7e0d953e0cfd22
Conceptio Open Knowledge Archive — every document is proof-bundled with source, license, and retrieval metadata.