Interpretable reinforcement learning with decision-tree pruning

(Under revision) manuscript No. (will be inserted by the editor)

Interpretable reinforcement learning with decision-tree pruning Mark Ringer · Michel Tokic

arXiv:2608.07151v1 [cs.LG] 7 Aug 2026

Received: 4th May 2026 / Accepted: Under revision

Abstract Reinforcement learning policies are difficult to inspect, but interpreting them is a prerequisite for trustworthiness. Converting a trained policy into explicit decision-tree rules improves transparency and the resulting artifacts often remain too complex for human understanding. We present a pruning process that simplifies such rule-based policies while preserving task performance and making edits to the policy auditable. The process defines a small set of structural and usage-aware operators and evaluates candidate edits by re-executing the policy to measure return and interpretability proxies. This exposes an transformation process from complex to compact policy structures. We investigate this approach on classic control and MuJoCo benchmarks, where pruning traces reveal consistent interpretability improvements while maintaining high performance. Keywords Interpretable RL · Decision-Tree · Pruning 1 Introduction Reinforcement learning has reached, and in some cases exceeded, human-level performance in control and games [Silver et al., 2017, Vinyals et al., 2019], but the resulting policies are often hard to inspect [Henderson et al., M. Ringer Faculty of Mathematics, Informatics and Statistics, Ludwig-Maximilians-University Munich, Munich, Germany, mark.leon.ringer@gmail.com M. Tokic Siemens AG, Data & Artificial Intelligence, Otto-HahnRing 6, 81739 Munich, Germany and Faculty of Mathematics, Informatics and Statistics, Ludwig-Maximilians-University Munich, Munich, Germany, michel@tokic.com

2018, Zahavy et al., 2016]. In settings where verification and accountability are required [Doshi-Velez and Kim, 2017, Lipton, 2017], intransparent policies, e.g. represented as neural networks, are typically not an option. Program-extracted policies address part of this gap by transforming trained neural network actors into explicit decision trees [Verma et al., 2019, Bastani et al., 2019, Delfosse et al., 2023]. These artifacts are executable and analyzable, but they often remain too large for reliable human understanding. Interpretable policies must be compact enough to read and simulate, not only explicit in form. This paper studies pruning as a principled, posthoc simplification operator on already interpretable, program-extracted policies. We treat simplification as a controlled edit process: apply a candidate operator, re-execute the policy to measure task return and interpretability proxies, and accept the edit only if it passes a non-inferiority test. Each accepted edit is recorded, yielding an auditable trail that ties structural changes to measured effects. Interpretability is thus a property of the transformation trajectory as well as of the final program.

2 Method This sections details the transformation process from common RL policies represented as neural networks, particularly we used the actor network from stablebaselines3 [Raffin et al., 2021], into decision trees using the transformation method proposed by Kohler et al. [2024]. We then employ a lightweight benchmark to evaluate the obtained policies, inspired by Kohler et al. [2025]. The same benchmark is reused to assess our pruned policies under identical conditions.

2

2.1 Distilling neural network actors into decision-tree rules The first step of the process is to transform a neural network policy into a decision tree as described in [Kohler et al., 2024]. Here, the teacher policy, in our case a neural network actor from from stable baselines, is used to generate a large corpus of environment specific stateaction pairs. These pairs are then used to fit a scikit DecisionTreeClassifier [Pedregosa et al., 2011], called the learner.

Mark Ringer, Michel Tokic

gain fine-grained insight into the trade-off between simplicity and reward. The resulting policies are not only quantitatively interpretable, according to Kohler’s proxies of smaller size, but also qualitatively transparent, as their structural evolution can be directly observed. In contrast to Kohler’s approach, which yields compact policies through direct training, our method makes the simplification process itself explicit, revealing how complex trees are incrementally transformed into minimal forms. This transparency enhances the interpretability of both the final policies and the underlying transformation.

2.2 Improving Interpretability In general, the interpretability of decision trees can be improved through pruning strategies. A common way to quantify interpretability is by measuring the number of resulting leaf nodes. Leaf-node-based interpretability proxy We use the number of leaf nodes as a proxy for interpretability. This measure captures the complexity of a given function while remaining invariant to syntactic variations.

2.3 Description of Pruning Algorithms We chose pruning algorithms to optimize the initial policy in terms of interpretability. Max-depth plus subtree collapsing is a structure-only cut followed by deterministic redundancy cleanup. Max-impurity plus subtree collapsing adds local data statistics (node purity) to make targeted reductions. Decision-tree Adaptive Constrained Pruning (DACP) incorporates runtime usage, in form of node visit counts, with reward guards.

Pruning strategies In extending [Kohler et al., 2025], our objective is to advance interpretability by focusing exclusively on Python policy trees and developing a structured pruning framework. Decision trees have consistently been found to be more interpretable than neural policies in human-subject studies [Freitas, 2014, Lipton, 2017], primarily because their hierarchical and rule-based structure aligns with human reasoning. While Kohler’s framework treated trees and MLPs as equally valid policy classes, we deliberately restrict our investigation to tree-based policies to maximize interpretability and enable direct structural analysis. To enhance interpretability in decision trees, we apply systematic pruning to distilled policies. Each tree is iteratively reduced using one of the following three strategies defined in Sec. 2.3:

Subtree collapsing We simplify the decision trees by recursively collapsing decision nodes into leaf nodes if both childern are leafs and contain the same action [Loh, 2011]. We use this method to support the other strategies but don’t treat it as a complete strategy. The behavior of forming subtrees with identical actions could occur because the distilling algorithm used in [Kohler et al., 2024] does not punish unnecessary splits, or later during pruning when alternative actions are removed. This operation does not alter the predictions of the model but simplifies the tree considerably. By eliminating such uniform subtrees, the resulting model becomes smaller, easier to interpret, and more efficient to evaluate, while maintaining identical predictive behavior.

– Max-depth pruning – Max-impurity pruning – Decision-tree Adaptive Constrained Pruning

2.3.1 Max-Depth Pruning

Each structural pruning step is followed by a subtree collapsing pass in order to remove residual redundancies and compress the tree further. Visual inspection of consecutive trees makes it possible to track which branches are pruned, whether these changes affects decision logic, and how they correlate with benchmark performance. By linking each pruning step to its corresponding performance measurement, we

Max-depth pruning restricts the maximum depth of the decision tree to a predefined limit. During training or post-processing, any node that would extend beyond this depth is replaced by a leaf node. The leaf represents the majority class of the samples that fall into it, effectively summarizing the deeper part of the tree. This approach prevents the model from growing overly complex and helps control overfitting by enforcing a global constraint on tree size and depth [Song and Lu, 2015]. In our case, we prune the tree in steps by reducing the

Interpretable reinforcement learning with decision-tree pruning

max depth in discrete steps. We always prune the original tree, the steps are saved to produce a trajectory to find an optimal policy. After each max depth pruning step, uniform pruning is applied. 2.3.2 Max-Impurity Pruning Max-impurity pruning reduces the tree’s complexity by halting further splits once a node becomes sufficiently homogeneous with respect to its class distribution. The impurity of a node is computed from the distribution of classes using the following metric based on the Gini index, Max-Impurity = 1 −

K X

p2k ,

k=1

where pk denotes the proportion of samples belonging to class k in that node and K denotes the set of classes [Loh, 2011]. Nodes with low impurity, meaning that one class dominates the samples within them, are considered pure enough and are converted into leaf nodes. This pruning strategy prevents unnecessary splits in already homogeneous regions of the data, thereby simplifying the tree and often improving its generalization performance. After max-impur ity pruning we apply uniform pruning to simplify the tree.

3

However, a challenge with this approach is, that some nodes with a low visit count (ci ) are actually crucial to maintain a high reward. Therefore, it is necessary to ensure that the reward doesn’t decrease too fast by comparing the performance measured as cumulative reward Ri of the pruned policy πi to the performance Ri−1 of its predecessor πi−1 . To formalize this, we define two parameters where RBase is the average reward of the original policy.: – tolerance factor δ ∈ (0, 1) which controls the maximum allowed reward decrease per iteration ∆ = |RBase | × δ – stability factor ϕ ∈ (0, 1 − δ), which determines the absolute minimum reward that is considered acceptable Φ = RBase − |RBase | × (1 − ϕ) where ∆ is the maximum absolute decrease in reward that is still accepted and Φ is the minimal reward that is still accepted. To prevent the required reward from increasing beyond a reasonable level, we also defined a ceiling value Γ as: Γ = RBase − ∆. Using these definitions, the minimum acceptable reward for each iteration i is given by

2.3.3 Decision-tree Adaptive Constrained Pruning

Rmin,i = min(max(Ri−1 − ∆, Φ), Γ ).

This pruning method, algorithm 1 in the appendix, is inspired by the A* Algorithm [Hart et al., 1968], CostComplexity Pruning [Loh, 2011] and Critical Value Pruning [Mingers, 1987] and based on the assumption that nodes that are less important for the performance of a given policy tree are visited less often than more critical nodes. The idea of heuristic pruning has previously been explored for different problems, for example in [Feldotto et al., 2022] heuristic network pruning, where a similar concept for neural networks is explored. A similar method is also used by humans do to limit cognitive resources if tasked with evaluating probabilistic planning tasks [Sass et al., 2025]. Following the intuition that this is also applicable for RL policies in the form of decision trees, a given policy π can be pruned by counting, for each node n, how often it is visited Ci = (ci,1 , ci,2 , ..., ci,N ) and removing the k nodes with the lowest visit count. This process can be repeated incrementally until the desired policy size is achieved. Obtaining the visit counts is accomplished by by using a counters(π) function that runs the policy and collects for each node how often it was visited.

A pruning step can only be considered successful if the resulting policy πi achieves a reward Ri ≥ Rmin,i . Determining the reward of a policy requires a reliable benchmarking: Ri = benchmark(πi ). This process is computationally expensive, and therefore it is necessary to minimize the number of benchmark calls. Therefore, the pruning is performed on a batches of k candidate nodes. To allow divide and conquer, k fulfills k ∈ {2m | m ∈ N0 }. If the pruning for a batch is rejected, the candidate nodes are split in half and the pruning is recursively repeated. In this way, only those nodes of a batch whose removal causes Ri < Rmin,i are not removed. If |π ′ | = |π|, we assume all nodes in the batch are essential and treat them as failed. Otherwise, we add all newly failed nodes to the failed list, clean the tree by using our collaps function as described in section 2.3, collect new counters and recalculate the batch size if there were any failed nodes. If its no longer possible to remove candidates, we start trying to remove nodes that failed in previous

(1)

4

steps. After we cannot remove any node anymore without the reward falling below the threshold, the algorithm terminates. 3 Results 3.1 Distilling Policies We configured the pruning algorithm to allow a maximum number of up to 1024 leaf nodes. For simple tasks like CartPole, a much smaller tree would have been sufficient, nonetheless the tree grows to a much larger of leaf nodes if allowed. For more complex tasks such as Walker2D a larger number of leaf nodes could still yield improved performance, but to have a comparable starting state we decided to allow all trees to grow up to the same maximum size. Notably, the policy trees for MountainCar never increased in number of leaf nodes beyond 340 for any configured max leaf nodes, up to 2048. Table 1 shows the performance of the distilled policies compared to their respective teachers. It can be observed that for most environments the performance of learner and teacher is similar. For complex tasks such as HalfCheetah and Walker2D, the episodic reward of the learner is much lower, probably due to the limited number of leaf nodes in the initial policy transformation process (neural network → decision tree). Another noteworthy observation is that for LunarLander the teacher performed much worse than the learner. This could possibly be due to an overfitted teacher, reducing complexity turned out to generalize better. 3.1.1 Solved thresholds Whenever possible, we use the solved thresholds provided by the gymnasium library Towers et al. [2024]. Since gymnasium does not provide solved thresholds for Pendulum and Walker2d, we defined them as -200 for Pendulum and 1500 for Walker2d. For Pendulum, the optimal reward is 0 since the reward is calculated using   r = − θ2 + 0.1 × θ̇2 + 0.001 × torque2 . where θ is the pendulums angle, normalized between [−π, π] while torque represents the action space of the environment Towers et al. [2024]. A near-optimal performance where only small deviations from upright position occur could be defined by accumulating on average no more than -1 reward per time step. For a episode length of 200, we therefore define the solved threshold as -200. For Walker2d 1500 is derived from the decomposed reward structure:

Mark Ringer, Michel Tokic

– healthy reward: Every timestep that the Walker2d is alive, +1 as reward. – forward reward: A reward for moving forward, depending on the velocity. – ctrl cost: A negative reward for taking large actions. Towers et al. [2024]. To achieve an average reward of 1500 or higher, constant locomotion in the target direction with and average of 0.5 in rewards is required. Therefore, a reward of 1500 implies constant and stable locomotion in the right direction. Since the theoretical reward for movement has no upper bound, this threshold should be taken with caution.

3.1.2 Observations The following observations can be derived from Figure 1, which summarizes the reward–size trade-offs across all evaluated environments. – General downward trend. Across most environments, we observe a broadly monotonic decrease in reward as pruning progresses, reflecting the expected trade-off between interpretability and performance. – Reward drop-off point. For all pruning algorithms and most environments, a distinct drop-off point emerges after which further pruning leads to a sharp decline in reward. – Occasional reward improvements. In a few cases, such as Acrobot-v1 (see Figure 1), pruning temporarily improves reward despite increasing interpretability, sometimes even above the original performance. This behavior could occur because of overfitted trees; If the overfitting branches are removed, the trees ability to generalize improves. – Small deviations between pruning algorithms. While all pruning strategies follow a similar overall trend, minor deviations appear in the intermediate pruning stages. This leads to the conclusion that the complexity of a given environment influences prunability. In general, we observe that DACP performs superior over most environments leading to the conclusion, that for reinforcement learning tasks, structural pruning is not sufficient and backtracking algorithms that use midpruning reward based evaluation are beneficial. 3.2 Limitations This work is limited by several factors. First, due to computational constrains, we decided to only distill policies to a max tree size of 1024. For decision trees that

Interpretable reinforcement learning with decision-tree pruning

5

Algorithm

Agent Version (Teacher)

PPO PPO SAC PPO SAC DQN SAC PPO TD3 SAC

ppo-Acrobot-v1 ppo-CartPole-v1 sac-HalfCheetah-v3 ppo-LunarLander-v2 sac-LunarLanderContinuous-v2 dqn-MountainCar-v0 sac-MountainCarContinuous-v0 ppo-Pendulum-v1 td3-Swimmer-v3 sac-Walker2d-v3

Learner Reward

Teacher Reward

-86 ± 44 488 ± 59 5023 ± 356 233 ± 58 236 ± 96 -102 ± 11 94 ± 2 -207 ± 212 355 ± 2 1815 ± 1061

-84 ± 25 500 ± 0 8898 ± 124 149± 34 262 ± 64 -101 ± 10 94 ± 2 -174 ± 107 355 ± 2 3917 ± 401

Table 1: Performance comparision of original (Teacher) and distilled (Learner) policies. Note: The term ”Learner” refers to the policy after transforming the neural network Teacher policy into the initial decision tree using approach of Kohler et al. [2024], which will be further optimized as depicted in Fig. 1.

max_impurity

Teacher

24 51 2 25 6 12 684 32 8 2

10

24 51 2 25 6 12 684 32 8 2

10

Walker2d

Random

Solved

2 25 6 12 684 32 8 2

0 51

24

Policy Size (leaf nodes, x^0.4 scale)

max_depth

4500 3000 1500

0 10

2 25 6 12 684 32 8 2

51

24 10

8 2

8 64 32

250 500 750

100

1000 1500

adaptive_constrained

250

300 200

500

40 12

6

0

0

0

Swimmer

2 25 6 12 684 32 8 2

120

40

0

Pendulum

51

80

150 180 210

24 51 2 25 6 12 684 32 8 2

MountainCarContinuous

90

250

500 750

10

24 51 2 25 6 12 684 32 8 2

10

10

24 51 2 25 6 12 684 32 8 2

0 MountainCar

25

0

LunarLanderContinuous

10

150

2500

24

450

7500 5000

10

300

300

LunarLander 250

2 25 6 12 684 32 8 2

450

150

Average Reward (asinh scale)

HalfCheetah

24

CartPole

51

Acrobot

0

Fig. 1: Performance of pruned policies, policy size (leaf nodes) against episodic reward. The teacher threshold marks the performance of the original model, the solved threshold marks the performance at which the environment is considered to be solved and the random threshold shows the performance of an agent taking a random action. For both the environments Pendulum and Walker2d there is no official solved threshold, so we defined solved for Pendulum as -200 and for Walker2d as 1500.

are grown to a significantly larger or smaller size, pruning results may differ. Another limitation is the question wether or not the number of leaf nodes is a sufficient metric for interpretability; the actual contents might impact the interpretability significantly [Freitas, 2014]. For example, a larger tree could be perceived as more readable than a smaller one because of clearly understandable attributes [Freitas, 2014].

metrics truly correlate with human understanding, including those not examined in this work. In addition, the effects of other pruning strategies could be explored, for example by combining different strategies. Finally, pruned and visualized policies could be applied in critical environments where human verification and transparency are necessary.

4 Conclusion 3.3 Future Work Future research could focus on validating interpretability through user studies to better understand which

In this work, we introduce a modern framework for distilling and pruning reinforcement learning policies with the goal of improving interpretability for humans. We build upon the works of Kohler et al. [2025] by using

6

Mark Ringer, Michel Tokic

Cart PolePolicy Decision Tree Pole Angular Velocity <= 0.01294954540207982 True

False

Pole Angular Velocity <= -0.06824247911572456 True

return Left

True

True

False

True

Pole Angle <= -0.0021369809983298182

Cart Velocity <= 0.11782633885741234

return Left

Pole Angular Velocity <= 0.06843433901667595

True

return Left

Pole Angle <= -8.786035687080584e-05

False

return Right

False

True

Cart Velocity <= 0.0034374564420431852

False

True

return Right

return Left

True

return Left

return Right

align with actual human cognitive accessibility. While our leaf-node-based proxy and visualizations try to capture interpretability, controlled user studies remain necessary to validate the true impact of pruning. Overall, pruning offers a promising step toward reinforcement learning policies that are both effective and understandable.

False

return Right

Conflict of Interest

False

return Right

Fig. 2: Last step of pruning the CartPole policy with DACP with a fixed batch size of 4. Here, the gray branches will be removed in the next iterative step, reducing the policy from eight to six leaf nodes while maintaining the performance (cf. Fig. 1, adaptive constrained in CartPole-v1). Leaves

Reward

Teacher %

1024 887 769 713 654 ... 126 121 120 117 116 ... 33 32 31 30 29

236.5 ± 95.5 226.9 ± 96.2 230.6 ± 110.2 218.1 ± 120.1 219.0 ± 125.5 ... 163.5 ± 149.8 136.8 ± 157.8 154.6 ± 146.3 149.0 ± 162.1 137.9 ± 164.4 ... 194.1 ± 119.0 206.4 ± 91.0 168.2 ± 157.2 182.1 ± 135.5 185.3 ± 130.5

89.8% 86.2% 87.6% 82.8% 83.2% ... 62.1% 51.9% 58.7% 56.6% 52.4% ... 73.7% 78.4% 63.9% 69.2% 70.4%

Table 2: Selected DACP steps for LunarLanderContinuousv3

their distilling algorithm. We introduce a similar benchmark, adding the policy size in the form of leaf nodes as a metric, provide a way of visualizing policy trees to improve readability by humans, and explore the effects of pruning to gain interpretability. Our pruning experiments show, that simplification can improve interpretability for reinforcement learning policies without immediate reward loss, though a trade-off emerges once complexity is reduced too far. In some cases, pruning even improved performance by reducing overfitting. Crucially, our method makes the simplification process itself transparent, which is an advantage over directly training smaller models. This study nevertheless faces limitations. Interpretability was measured only through proxies, leaving open the question of how well these

The authors have no conflicts of interest to declare that are relevant to the content of this article.

References O. Bastani, Y. Pu, and A. Solar-Lezama. Verifiable reinforcement learning via policy extraction. arXiv:1805.08328, 2019. Q. Delfosse, H. Shindo, D. S. Dhami, and K. Kersting. Interpretable and explainable logical policies via neurally guided symbolic abstraction. In Proceedings of the 37th International Conference on Neural Information Processing Systems, 2023. F. Doshi-Velez and B. Kim. Towards a rigorous science of interpretable machine learning. arXiv:1702.08608, 2017. B. Feldotto, H. Lengenfelder, F. Röhrbein, and A. C. Knoll. Network layer analysis for a rl-based robotic reaching task. Frontiers in Robotics and AI, 9: 799644, 2022. A. A. Freitas. Comprehensible classification models: a position paper. SIGKDD Explor. Newsl., 15(1):1–10, 2014. P. E. Hart, N. J. Nilsson, and B. Raphael. A formal basis for the heuristic determination of minimum cost paths. IEEE Transactions on Systems Science and Cybernetics, 4(2):100–107, 1968. P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger. Deep reinforcement learning that matters. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), 2018. H. Kohler, Q. Delfosse, R. Akrour, K. Kersting, and P. Preux. Interpretable and editable programmatic tree policies for reinforcement learning. In 7th European Workshop on Reinforcement Learning, 2024. H. Kohler, Q. Delfosse, W. Radji, R. Akrour, and P. Preux. Evaluating interpretable reinforcement learning by distilling policies into programs. arXiv:2503.08322, 2025. Z. C. Lipton. The mythos of model interpretability. arXiv:1606.03490, 2017.

Interpretable reinforcement learning with decision-tree pruning

W.-Y. Loh. Classification and regression trees. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 1:14 – 23, 2011. J. Mingers. Expert systems—rule induction with statistical data. The Journal of the Operational Research Society, 38, 1987. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825– 2830, 2011. A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1–8, 2021. S. H. Sass, L. Gönner, S. Schwöbel, S. Frölich, F. Glöckner, S. J. Kiebel, S. C. Li, and M. N. Smolka. Heuristic pruning of decision trees at low probabilities and probability discounting in sequential planning in young and older adults. Scientific Reports, 15(1):16260, 2025. D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Y. Chen, T. Lillicrap, F. Hui, L. Sifre, G. Driessche, T. Graepel, and D. Hassabis. Mastering the game of go without human knowledge. Nature, 550:354–359, 2017. Y.-Y. Song and Y. Lu. Decision tree methods: applications for classification and prediction. Shanghai archives of psychiatry, 27:130–5, 2015. M. Towers, J. K. Terry, A. Kwiatkowski, J. Balis, A. Kanervisto, and A. Gleave. Gymnasium: A standard interface for reinforcement learning environments. arXiv:2407.17032, 2024. A. Verma, V. Murali, R. Singh, P. Kohli, and S. Chaudhuri. Programmatically interpretable reinforcement learning. arXiv:1804.02477, 2019. O. Vinyals, I. Babuschkin, W. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. Choi, R. Powell, T. Ewalds, P. Georgiev, J. Oh, D. Horgan, M. Kroiss, I. Danihelka, A. Huang, L. Sifre, T. Cai, J. Agapiou, M. Jaderberg, and D. Silver. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575:350–354, 2019. T. Zahavy, N. Ben-Zrihem, and S. Mannor. Graying the black box: Understanding dqns. In M. F. Balcan and K. Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, pages 1899–1908, 2016.

7

5 Appendix

Algorithm 1: DACP (high-level) Input: Policy π, tolerance: 0 ≤ δ ≤ 1, stability 0≤α≤1 Output: Pruned policy π ′ RLast ← RBase ← benchmark(π) ; reset ← False; F ←∅; p k ← |π|, rounded down to power of two; while true do C ← counters(π); Rmin ←
min Rbase − |Rbase |(1 − α), Rlast − |Rbase |δ ; N ← k least visited nodes from C \ F ; if N = ∅ then if reset then break; F ← ∅; reset ← True; p k ← |π|, rounded down to power of two; continue; (π ′ , R′ , Fnew , ok) ← PruneBatch(π, N, RLast , Rmin ); if |π| = |π ′ | then F ← F ∪ N; continue; else F ← F ∪ Fnew ; π ← π′ ; RLast ← R′ ; reset ← False; π ← collaps(π ′ ); if not okpthen k ← |π| − |F |, rounded down to power of two; return π;

8

Mark Ringer, Michel Tokic

Algorithm 2: PruneBatch Input: Policy π, candidate nodes N , last reward R, minimum reward Rmin Output: New policy π ′ , new reward R′ , failed nodes F , success flag ok π ′ ← remove N from π; R′ ← benchmark(π ′ ); if R′ ≥ Rmin then return π ′ , R′ , ∅, T rue; if N = ∅ then return π, R, ∅, F alse; if |N | = 1 then return π, R, N, F alse; NL , NR ← split(N ); (πL , −, FL , −) ← PruneBatch(π, NL , R, Rmin ); (πR , RR , FR , −) ← PruneBatch(πL , NR , R, Rmin ); return πR , RR , FL ∪ FR , F alse;

Related documents

• Explanation through Reward Model Reconciliation using POMDP Tree Search#163973
• Explanation through Reward Model Reconciliation using POMDP Tree Search#163014
• Illuminating the Three Dogmas of Reinforcement Learning under Evolutionary Light#670359
• Illuminating the Three Dogmas of Reinforcement Learning under Evolutionary Light#668294
• ASCT: Attentive Search over Counterfactual Trees for Credit Assignment in Agentic Reinforcement Learning#1129812