Regularized Reward-Punishment Reinforcement Learning Jiexin Wanga , Eiji Uchibea
arXiv:2606.28152v1 [cs.LG] 26 Jun 2026
a
Dept. of Brain Robot Interface, ATR Computational Neuroscience Laboratories, 2-2-2, Hikaridai, Seikacho, Soraku-gun, Kyoto, 619-0288, Japan
Abstract We propose KL-Coupled Policy Regularization (KCPR), a policy coordination framework for Reward–Punishment Reinforcement Learning (RPRL). Based on KCPR, we derive KL-Coupled Soft Optimality (KCSO) and develop its deep realization, klDMP. Unlike existing RPRL approaches that optimize reward-seeking and punishment-related policies largely independently, KCPR enables direct interactions between companion policies by treating each as a dynamically learned prior for the other. KCSO yields coupled soft-optimal policies and KL-regularized Bellman operators, allowing reward and punishment information to jointly influence value propagation. To improve learning stability, we introduce a companion-prior softening mechanism and evaluate separate replay-buffer designs for balancing reward- and punishment-related experience. Experiments in grid-world and Gazebo robotic navigation tasks demonstrate that klDMP improves safety and learning stability while maintaining competitive task performance compared with DQN, SQL and softDMP. These results suggest that policy-level coordination provides an effective mechanism for integrating multiple behavioral objectives and may serve as a useful design principle for reinforcement learning systems with interacting motivational processes. Keywords: reward-punishment reinforcement learning, maxpain, deep reinforcement learning, robot navigation, Turtlebot3, ROS Gazebo 1. Introduction Intelligent agents operating in the real world must continuously negotiate between multiple reward-related signals, including the pursuit of desirable outcomes and the avoidance of harmful consequences under uncertainty.
While standard Reinforcement Learning (RL) typically models this tradeoff using a single scalar reward, biological decision-making systems rely on distributed and interacting motivational circuits rather than a unidimensional signal. Neuroscience studies suggest that reward and punishment are processed through distinct but interacting pathways, shaped by different neuromodulatory mechanisms and contributing jointly to adaptive behavior [1, 2, 3]. Importantly, these systems do not operate independently—they influence each other’s learning dynamics in structured ways that support robust decision-making. This biological view has motivated the Reward–Punishment Reinforcement Learning (RPRL) paradigm, where positive and negative outcomes are processed through separate computational modules. Frameworks such as MaxPain [4], Deep MaxPain (DMP) [5, 6] and softDMP [7] demonstrate that decomposed value systems can accelerate early learning, induce safe exploration and improve sample efficiency and robustness, particularly in robot navigation tasks where adverse outcomes like collisions are essential learning signals. Yet existing RPRL architectures maintain an important simplifying assumption: goal-seeking and risk-aware behaviors develop independently, interacting only through value mixing or uniform-prior entropy. This assumption limits the expressive capability of decomposed RL systems, preventing the two pathways from shaping each other’s behavior in an evolving and structured manner seen in biological agents. Our key insight is that interactions between motivational systems should occur at the policy level rather than solely through value decomposition. To this end, we introduce KL-Coupled Policy Regularization (KCPR), a principle in which each policy is regularized by a dynamically learned companion policy. Unlike conventional KL regularization, which is typically used for soft-optimality and entropy regularization [8, 9, 10, 11, 7] or trust-region policy optimization [12, 13, 14], KCPR uses KL divergence as a mechanism for coordinating multiple motivational systems. Based on KCPR, we derive KL-Coupled Soft Optimality (KCSO), a soft-optimality framework in which companion policies act as adaptive priors, yielding coupled soft-optimal policies and Bellman operators that allow reward and punishment information to influence each other’s value propagation. We further develop klDMP as a practical realization of KCSO within the RPRL paradigm. To improve learning stability and data efficiency, we introduce a softening mechanism for companion priors and revisit the separate replay-buffer scheme proposed in softDMP [7] under the KL-coupled setting. 2
Figure 1: Overview of the proposed framework. KCPR introduces bidirectional KL coupling at the objective level, where the reward-seeking and risk-related subsystems mutually regularize one another through coupled optimization objectives. Based on KCPR, KCSO derives a pair of companion policies with dynamically learned behavioral priors, establishing policy-level coordination between the two motivational subsystems.
Experiments in both grid-world and deep robotic navigation tasks demonstrate that KCPR and the resulting KCSO framework enable structured interactions between reward-seeking and punishment-related learning. These interactions lead to safer and more risk-aware behaviors, more stable value propagation and learning dynamics, and competitive navigation performance compared with DQN, SQL and softDMP. Together, the results highlight the importance of policy-level coordination as an effective mechanism for balancing efficiency and safety in decomposed reinforcement learning. More broadly, this work highlights a fundamental question in modern AI and robotics: how multiple internal objectives should be coordinated within a single learning system. While current foundation-model and vision-languageaction approaches [15, 16, 17, 18] primarily focus on integrating perception, language, and action, comparatively less attention has been paid to the organization of internal motivational processes. More generally, the proposed KCPR framework may provide a foundation for studying interactions among additional motivational signals, such as safety, curiosity, or uncertainty, in future embodied agents. 3
Our main contributions are: 1. KL-Coupled Policy Regularization (KCPR). We introduce a new regularization principle in which companion policies act as dynamically learned priors, enabling policy-level interactions between reward and punishment pathways. 2. KL-Coupled Soft Optimality (KCSO) and klDMP. We derive KCSO as the soft-optimality formulation of KCPR and develop klDMP as its practical realization within the RPRL framework. 3. Stabilization mechanisms. We introduce a softening criterion for stable KL-coupled learning and provide an empirical analysis of replaybuffer designs for balancing reward- and punishment-driven updates. 4. Deep robotic realization Through grid-world and robotic navigation experiments, we demonstrate that KL coupling improves safety, convergence stability, and risk-aware behavior compared with DQN, SQL and softDMP. 2. Related Work 2.1. Reward-Punishment RL Early modular reinforcement learning investigated learning separate value functions for different reward components or sub-goals [19, 20, 21]. More recent methods, including Hybrid Reward Architecture (HRA) [22] and Reward Decomposition with Representation Disentanglement (RD2 ) [23], extended this idea to deep reinforcement learning by decomposing complex rewards into additive components. Reward–Punishment Reinforcement Learning (RPRL) instead separates environmental feedback into positive and negative channels, inspired by the dual valuation systems observed in biological decision making. Early formulations [24, 25] employed separate actor–critic architectures for positive and negative rewards, while more recent methods, such as Reward–Punishment Actor–Critic [26], demonstrated their effectiveness in robotic manipulation. RPRL has also been used as a computational model of neurological and psychiatric disorders [27, 28]. Among RPRL methods, MaxPain [4] and Deep MaxPain (DMP) [5] most effectively leverage the punishment pathway. MaxPain introduces a “min” Bellman operator for negative rewards, enabling sharper propagation of aversive signals compared to standard value decomposition methods such as 4
HRA. DMP further shows that directly optimizing negative return improves the shaping of positive value learning through shared dynamics. SoftDMP [7] further introduced entropy regularization, replacing the hard max/min operators with mellow-max/mellow-min. This soft treatment reduces greedy exploitation of action values, and therefore improves robustness and sample efficiency. Despite these advances, existing RPRL methods optimize reward and punishment pathways largely independently, with interactions limited to value aggregation or entropy regularization. Consequently, policy-level coordination between motivational systems remains largely unexplored. 2.2. Regularized RL Kullback–Leibler (KL) divergence [29] plays a central role in modern reinforcement learning and is commonly used either to derive soft-optimal policies or to constrain policy updates during optimization. A major line of work employs KL regularization to soften value updates and encourage stochastic exploration. Dynamic Policy Programming (DPP) [8] introduces a KL penalty relative to a baseline policy, yielding a log-sumexp Bellman operator that stabilizes temporal-difference learning. G-learning [9] further replaces the baseline with a uniform prior, leading to a connection with the mellow-max operator [30]. This perspective underlies Soft Q-Learning (SQL) [10] and Soft Actor–Critic (SAC) [11], where KL regularization to a uniform prior is equivalent to entropy maximization. SoftDMP [7] extends these ideas to RPRL. A second line of work uses KL divergence as a trust-region constraint. Methods such as REPS [12], TRPO [13], and PPO [14] constrain the divergence between successive policies to ensure stable policy improvement. Among existing KL-regularized RL methods, our work is most closely related to soft-optimality approaches, but differs from existing approaches in two key aspects. First, KCPR replaces fixed reference distributions with dynamically learned companion policies. Second, KL regularization is used to coordinate interacting motivational subsystems rather than to encourage stochasticity, stabilize optimization, or constrain successive policy updates. These changes transform KL divergence from a policy regularizer into a mechanism for policy coordination.
5
2.3. Policy Distillation and Multi-Task Regularization KL regularization is also central in multitask, transfer, and distillationbased reinforcement learning, where the goal is to share knowledge across tasks or agents. In Policy Distillation [31], a student policy is trained to imitate one or more teacher policies while optimizing its own objective, effectively compressing multiple specialized behaviors into a single network. Actor-Mimic [32] extends this idea by combining imitation with feature regression to improve cross-task generalization. Distral [33] further introduces a shared distilled policy that captures common behavioral structure across tasks while regularizing task-specific policies toward this shared prior. In these methods, KL regularization facilitates knowledge sharing across different tasks or agents. By contrast, KCPR applies KL regularization between companion policies within the same MDP, where the policies optimize complementary reward components and co-adapt throughout learning. Consequently, KL divergence serves as a mechanism for coordinating interacting motivational systems rather than transferring knowledge across tasks. 2.4. Relationship to Classical RL Our method is conceptually related to Expected SARSA [34], where value updates take an expectation over a policy rather than a greedy maximization. In klDMP, this expectation is instead defined by a learned companion policy. Unlike Expected SARSA, which evaluates the same policy used for sampling, klDMP remains off-policy because the expectation distribution is provided by a separately learned companion policy rather than the behavior policy. The deep implementation follows a standard actor–critic-style architecture with a shared encoder and separate value and policy heads, similar to A3C [35], SAC [11], and PPO [14]. Unlike conventional actor–critic methods, however, the policy heads are not optimized by policy gradients. Instead, they are directly supervised to match the analytically derived KCSO policies, yielding an explicit coupling between value estimation and policy formation. 2.5. Safe RL Safe Reinforcement Learning aims to maximize cumulative rewards while explicitly enforcing safety constraints throughout learning [36, 37]. It is commonly formulated as a Constrained Markov Decision Process (CMDP), where safety is represented as an explicit cost function or risk constraint. In contrast, RPRL treats safety implicitly through negative rewards, allowing failures during exploration to improve subsequent value learning. 6
These paradigms therefore differ in their treatment of safety. Safe RL emphasizes preventing constraint violations and often provides theoretical or probabilistic safety guarantees, whereas RPRL relies on trial-and-error learning in which failures serve as informative training signals. Accordingly, Safe RL focuses on preventing failures, whereas RPRL focuses on learning from them. 3. KL-regularized Deep MaxPain 3.1. KL-Coupled Policy Optimization We consider an infinite-horizon discrete-time Markov Decision Process (MDP) defined by the tuple (S, A, P, R, γ), where S denotes the state space, A the discrete action space, P the transition probability, R the reward function, and γ ∈ [0, 1) the discount factor. At each time step t, the agent selects an action at ∈ A in response to the current state st ∈ S, according to a stochastic behavior policy π(at | st ). The environment then transitions to the next state st+1 ∼ P(· | st , at ), yielding a scalar reward from the reward function rt = R(st , at , st+1 ) as an evaluation signal. Figure 1 provides an overview of the proposed framework. At the objective level, KL-Coupled Policy Regularization (KCPR) introduces bidirectional coupling between the reward-seeking and punishment-related optimization objectives. Based on KCPR, KL-Coupled Soft Optimality (KCSO) derives the corresponding companion policies, enabling policy-level interactions between the two motivational subsystems. The remainder of this section formalizes these two components. Following the RPRL paradigm, the environmental reward is decomposed into positive and negative components, r− = min(R, 0),
r+ = max(R, 0),
where r+ ≥ 0 represents the goal-seeking reward signal and r− ≤ 0 represents the punishment signal. At the objective level, KCPR replaces the fixed reference prior in conventional KL regularization with a dynamically learned companion policy. Formally, given a reward function r, a companion policy πc , and a regularization coefficient η, KCPR defines the following objective: "∞ # X 1 J(π; πc , r, η) = Eπ,P γ t rt − DKL (π(·|st )||πc (·|st )) . η t=0 7
For comparison, entropy-regularized methods such as softDMP use a fixed uniform prior, πc = πuniform . Within the reward–punishment setting considered in this work, the positive reward signal r+ is regularized by the pain-avoiding policy ¬π − , while the negative reward signal r− is regularized by the goal-seeking policy π + : J + (π) = J(π; ¬π − , r+ , η + ), J − (π) = J(π; π + , r− , η − ).
(1a) (1b)
The corresponding optimal policies are defined as π − = arg min J − (π).
π + = arg max J + (π), π
π
The asymmetric max–min formulation plays a key role in KCPR. Maximizing J + encourages the goal-seeking policy to align with the pain-avoiding companion prior, while minimizing J − yields a pain-seeking policy whose behavior remains differentiated from the goal-seeking policy. The coupled objectives above define the optimization problem. We next derive their corresponding soft-optimal solutions, referred to as KL-Coupled Soft Optimality (KCSO). Formally, consider a policy regularized by a companion policy πc . The KCPR objective admits the following soft-optimal solution: πc (a|s) exp ηQ(s, a) ∗ . π (a|s) = P ′) π (a|s) exp ηQ(s, a ′ c a KCSO treats the prior as a dynamically learned companion policy: ¬π − (a | s) exp[η + Q+ (s, a)] , − ′ + + ′ a′ ¬π (a | s) exp[η Q (s, a )] π + (a | s) exp[η − Q− (s, a)] π − (a | s) = P + ′ , − − ′ a′ π (a | s) exp[η Q (s, a )] π + (a | s) exp[−η − Q− (s, a)] ¬π − (a | s) ≜ P + ′ , − − ′ a′ π (a | s) exp[−η Q (s, a )] π + (a | s) = P
(2a) (2b) (2c)
where Eq. (2a) defines the goal-seeking policy π + using the pain-avoiding policy ¬π − as its companion prior. Eq. (2b) defines the pain-seeking policy 8
π − using the goal-seeking policy π + as its companion prior, while Eq. (2c) defines the complementary pain-avoiding policy ¬π − under the same companion prior π + by reversing the sign of the punishment-related action values. The sign of η determines whether the policy favors high- or low-value actions: η + > 0 yields the reward-seeking policy π + , whereas η − < 0 yields the painseeking policy π − . The complementary policy ¬π − favors actions associated with lower expected punishment and serves as the companion prior for the goal-seeking subsystem. 3.2. Tabular Learning rules The KCPR objective and the resulting KCSO principle naturally give rise to a practical reinforcement learning algorithm. In the tabular setting, we refer to this algorithm as klMP. The deep neural network realization introduced later is referred to as klDMP. This section first presents the tabular learning rules underlying klMP, which form the foundation of the deep implementation. Under KCSO, value propagation is governed by a pair of KL-coupled Bellman operators associated with the reward-seeking and punishment-related subsystems. These operators are conditioned on the dynamically evolving companion policies derived in Eq. (2), allowing each subsystem to propagate value information under the behavioral guidance of its companion policy. For an experience tuple (s, a, r, s′ ), the KL-coupled Bellman targets are defined as: X γ+ log ¬π − (a′ |s) exp[η + Q+ (s′ , a′ )], η+ a′ − X γ y − = r− + − log π + (a′ |s) exp[η − Q− (s′ , a′ )]. η a′
y + = r+ +
(3a) (3b)
where the first target propagates reward information under the pain-avoiding companion policy ¬π − , while the second propagates punishment-related information under the goal-seeking companion policy π + . The corresponding temporal-difference errors are δ − = y − − Q− (s, a).
δ + = y + − Q+ (s, a),
and the action-value functions are updated using standard temporal-difference
9
learning, Q+ (s, a) ← Q+ (s, a) + α+ δ + , Q− (s, a) ← Q− (s, a) + α− δ − .
(4a) (4b)
where α+ and α− denote the learning rates for the goal-seeking and punishmentrelated modules, respectively. After each value update, the companion policies are renewed according to Eq. (2). Consequently, value estimation and policy formation are coupled throughout learning: updated value functions determine new companion policies, which in turn define subsequent KL-coupled Bellman targets. 3.3. Behavior policy The KCSO operator defines the soft-optimal policies associated with the goal-seeking and punishment-related subsystems. However, directly executing these policies may lead to insufficient exploration, particularly during the early stages of learning when both value estimates and companion priors remain inaccurate. Therefore, klDMP constructs a separate behavior policy for data collection while preserving the coupling structure induced by KCPR and KCSO. To regulate exploration during learning, an annealable temperature parameter τ > 0 is introduced. The resulting softened policies are defined as π̃ + (a | s) ∝ ¬π − (a | s)1/τ · exp[ηQ+ (s, a)/τ ],
(5a)
¬π̃ − (a | s) ∝ π + (a | s)1/τ · exp[−ηQ− (s, a)/τ ],
(5b)
π̃ − (a | s) ∝ π + (a | s)1/τ · exp[ηQ− (s, a)/τ ].
(5c)
As τ → 1, the softened policies converge to their original soft-optimal forms: π̃ + → π + , π̃ − → π − , and ¬π̃ − → ¬π − (Eq.(2)). Conversely, as τ → ∞, all sub-policies approach uniform distributions, ensuring adequate exploration during early training. The overall behavior policy is defined as a mixture of the softened sub-policies: π̄(a | s) = wπ̃ + (a | s) + (1 − w)¬π̃ − (a | s),
(6)
where the mixing weight w ∈ [0, 1] controls the relative influence of the goalseeking and pain-avoiding sub-modules. Similar to DMP, the mixing weight w may either be fixed or adapted online [6] according to the relative strengths of the two motivational systems. 10
Note that the behavior policy is used solely for data collection, whereas policy optimization remains governed by KCSO. This separation allows exploration to be adjusted independently without altering the KL-coupled optimality principle. 3.4. Softening companion priors The effectiveness of KCPR relies on the use of learned companion policies as dynamic priors. However, during the early stages of learning, these companion policies are derived from immature value estimates and may therefore become prematurely concentrated on suboptimal actions. When such overconfident priors are incorporated into the KCSO operator and the corresponding Bellman backups, value propagation through alternative actions can be significantly suppressed, leading to unstable learning dynamics. To alleviate this issue, we introduce a softening mechanism that interpolates the companion priors with a uniform distribution: − ¬πsoften (a | s) = ϵπuniform + (1 − ϵ)¬π − (a | s), + πsoften (a | s) = ϵπuniform + (1 − ϵ)π + (a | s).
(7a) (7b)
1 , and ϵ ∈ [0, 1] controls the degree of softening. The prowhere πuniform = |A| posed formulation establishes a continuous transition between entropy-based and KL-coupled regularization. When ϵ = 1, the companion priors reduce to a uniform distribution, recovering entropy regularization. When ϵ = 0, the original learned companion priors are recovered, yielding the standard klDMP formulation. Note that the softened companion priors are used consistently throughout learning, including the KL-regularized Bellman backups of Q+ and Q− , the computation of KCSO policies, and the construction of the behavior policy.
3.5. Separate replay buffer To alleviate experience imbalance between the two motivational subsystems, we adopt the separate replay buffer scheme introduced in softDMP [7]. In practice, the pain-avoiding component often dominates data collection, resulting in insufficient training samples for the pain-seeking value function Q− . To ensure balanced learning, experiences are assigned to separate replay buffers using a discriminator: D(s, a, r, s′ ) =
π̃ − (a | s) π̃ − (a | s) + π̃ + (a | s) 11
(8)
which quantifies the relative contribution of the pain-seeking policy to a sampled transition. Transitions with higher discriminator values are stored in the negative replay buffer D− for updating Q− , while the remaining samples are assigned to the positive replay buffer D+ for updating Q+ . By decoupling experience storage, each subsystem receives training data that better reflects its behavioral objective. This improves the propagation of punishment-related value information, facilitates more balanced learning between the two motivational subsystems, and reduces interference between reward-seeking and punishment-related updates. 3.6. Deep neural network realization
Figure 2: Dual-network realization of the proposed klDMP framework.
In the deep implementation (klDMP), each motivational subsystem is realized by a neural network with separate value and policy heads (Figure 2). Both modules share the same multimodal perception pipeline, where RGB images and LiDAR measurements are encoded into a shared feature representation before branching into value and policy heads. The goal-seeking 12
Algorithm 1 klDMP Learning Algorithm Require: hyperparameters α± , γ ± , η ± , ϵ − + − 1: Initialize online value and policy networks Q+ θ+ , Qθ− , πϕ+ , ¬πϕ− with parameter θ+ , θ− , ϕ+ , ϕ− 2: Initialize target networks θ̄ + ← θ + , θ̄ − ← θ − , ϕ̄+ ← ϕ+ , ϕ̄− ← ϕ− 3: Initialize replay buffers D + and D − 4: for each episode do 5: Initialize state s 6: while s is not terminal do 7: Construct the behavior policy π̄(a|s) using Eq. (7) and Eq. (6) 8: Sample action a ∼ π̄(·|s) 9: Execute action a, observe r+ , r− , s′ , d 10: Assign (s, a, r+ , s′ , d) to D+ and (s, a, r− , s′ , d) to D− using Eq. (8) 11: Sample mini-batches from D+ and D− 12: Compute yQ+ , yQ− using Eq. (10) 13: Update θ+ , θ− by minimizing LQ+ , LQ− using Eq. (9) 14: Compute yπ+ , y¬π− using Eq. (12) 15: Update ϕ+ , ϕ− by minimizing Lπ+ , L¬π− using Eq. (11) 16: if target update condition is satisfied then 17: Update target networks: θ̄+ ← θ+ , θ̄− ← θ− , ϕ̄+ ← ϕ+ , ϕ̄− ← ϕ− 18: end if 19: s ← s′ 20: end while 21: end for network outputs (Q+ , π + ), while the punishment-related network outputs (Q− , ¬π − ). The value heads propagate gradients through the entire network, ensuring that representation learning is driven by temporal-difference errors. In contrast, the policy heads are restricted to backpropagation only up to the shared feature layer, decoupling policy optimization from low-level feature learning stability. − Let Q+ θ+ and Qθ− denote the value function estimators parameterized by + − θ and θ , and let πϕ++ and ¬πϕ−− denote the corresponding policy heads
13
parameterized by ϕ+ and ϕ− . The value losses are defined as: h 2 i + LQ+ = ED+ Q+ (s, a) − y , Q θ+ h i 2 − LQ− = ED− Q− (s, a) − y . − Q θ
(9a) (9b)
The deep implementation follows the same KL-coupled learning rules as the tabular algorithm. Specifically, the Bellman backups introduced in Eq. (3) are applied to neural value functions and target networks to construct the training targets: X + + ′ ′ (1 − d)γ + − ′ ′ ¬π (a | s ) exp η Qθ̄+ (s , a ) , (10a) log − ϕ̄ soften η+ a′ X − − ′ ′ (1 − d)γ − + ′ ′ yQ− ≜ r− + π (a | s ) exp η Qθ̄− (s , a ) . (10b) log + ϕ̄ soften η− a′
yQ+ ≜ r+ +
Here, D+ and D− denote the positive and negative replay buffers, respectively. Compared with the tabular formulation, the deep implementation additionally incorporates target networks and terminal-state masking through (1−d), where d indicates episode termination. The target networks θ̄+ , θ̄− , ϕ̄+ , ϕ̄− are periodically updated copies of the online networks and are used to stabilize KL-coupled value propagation during training. The policy heads are trained by matching analytically derived KL-optimal targets: h i Lπ+ = ED+ DKL (yπ+ (· | s)||πϕ++ (· | s)) , (11a) h i L¬π− = ED− DKL (y¬π− (· | s)||¬πϕ−− (· | s)) . (11b) The corresponding target policies are defined as: ¬πϕ̄−− soften (a | s) exp[η + Q+ (s, a)] θ̄+ , yπ+ (a | s) = P − + + ′ a′ ¬πϕ̄− soften (a | s) exp[η Qθ̄+ (s, a )]
(12a)
πϕ̄++ soften (a | s) exp[−η − Q− (s, a)] θ̄− . y¬π− (a | s) = P + − − ′ a′ πϕ̄+ soften (a | s) exp[−η Qθ̄− (s, a )]
(12b)
Figure 2 and Algorithm 1 summarize the complete deep realization of the proposed framework. 14
4. Experiments 4.1. Grid-world We evaluate klMP in low-dimensional grid-world environments to analyze how KCPR influences policy coupling, value propagation, and learning dynamics under reward–punishment decomposition. Specifically, we use a 9×9 U-maze [6, 7] and a 36×19 Three-room maze to investigate three key components of the proposed framework: • Effects of KL regularization: how companion-policy regularization shapes policy coupling and the trade-off between goal-seeking and riskavoiding behaviors. • Effects of companion-prior softening: how mitigating overconfident companion priors stabilizes value propagation during early learning. • Effects of separate replay buffers: how separating reward- and punishment-related experiences improves learning stability and punishmentrelated value propagation. 4.1.1. Effects of regularization To isolate the effect of KL-regularization, we first construct a baseline setting consisting of two independently trained agents under a fixed environment. The goal-seeking agent receives a reward of +1 upon reaching the goal, while the pain-avo iding agent receives a penalty of −0.1 when colliding with obstacles. All other transitions yield zero reward, and episodes terminate upon reaching the goal. Both agents are trained using model-based value iteration (QVI), following a MaxPain-style update scheme where goal-directed learning employs a max operator and pain-avoiding learning employs a min operator. The resulting optimal policies are obtained via a low-temperature softmax over action values. Figure 3 shows the resulting value functions and policies. The goalseeking agent learns a monotonic value landscape toward the goal, resulting in shortcut-oriented navigation behavior. In contrast, the pain-avoiding agent forms structured low-value regions around obstacles, effectively inducing safety zones that shape cautious navigation behavior. To study crosssubsystem interaction, we next introduce KL-coupling by treating the independently learned policies π +∗ and ¬π −∗ as fixed companion priors in a 15
second round of QVI, referred to as klQVI. In this setting, each Bellman update is regularized toward the behavior of the opposite subsystem, as defined in Eq. (3).
Figure 3: Optimal state-value functions and the corresponding reward-seeking and painavoiding policies learned independently. These policies are subsequently used as fixed companion priors in the klQVI experiments to analyze the effects of KL-Coupled Policy Regularization.
Figure 4 illustrates the resulting optimal policies under different KL regularization strengths. A clear and systematic blending between goal-directed and risk-averse behaviors emerges as a function of η. When η + is large, the KL term becomes weak and the goal-seeking policy closely follows the standard QVI solution. As η + decreases, the influence of the pain-avoiding prior ¬π −∗ becomes stronger, leading to increasingly conservative navigation behavior. A symmetric trend is observed in the punishment-learning case, where smaller |η − | amplifies the influence of the goal-seeking prior. These results provide empirical evidence for KCPR. By treating companion policies as learned priors, KCPR enables continuous behavioral shaping between reward-seeking and punishment-related subsystems. The resulting KCSO policies exhibit smooth transitions between independent and strongly coupled behaviors, demonstrating how policy-level coupling can regulate the exploration–safety trade-off. 16
(a) the optimal goal-seeking policy derived from Q+ learned from klQVI with different η +
(b) the optimal pain-avoiding policy derived from Q− learned from klQVI with different η −
Figure 4: Optimal policies obtained from klQVI under different KL regularization strengths. It illustrates how KCPR controls policy coupling through the companion prior. Smaller values of |η| increase the influence of the companion policy, leading to stronger behavioral blending between reward-seeking and pain-avoiding policies, whereas larger values recover behavior closer to independent learning.
17
4.1.2. Effects of softening criterion We evaluate klDMP with and without the proposed softening criterion in a grid-world environment where reward and punishment signals coexist. The agent receives a reward of +1 upon reaching the goal and a penalty of −0.1 for collisions, with all other rewards set to zero. The goal state is absorbing. The algorithm maintains two action-value functions Q+ and Q− , together with their corresponding KL-coupled policies, which are updated jointly according to Eq. (2) and Eq. (4).
(a) ϵ = 0 (klMP case)
(b) ϵ = 0.3 (softened klMP case)
Figure 5: Heatmaps of the learned state-value functions and state visitation counts with and without the proposed softening criterion. Without softening (ϵ = 0), the companion prior becomes overconfident during early learning, suppressing the propagation of punishment-related values and resulting in shortcut-oriented exploration. With softening (ϵ = 0.3), both reward and punishment values propagate more effectively, leading to a more balanced visitation distribution and safer navigation behavior.
Figure 5 compares pure klMP (ϵ = 0) and softened klMP (ϵ = 0.3) under identical KL regularization strength (η + , η − ) = (1000, −1000). Without softening, the rapidly sharpening companion priors lead to premature policy 18
concentration, which in turn suppresses the propagation of negative value signals in Q− . As a result, the agent exhibits shortcut-oriented behavior dominated by goal-seeking dynamics, with insufficient risk-awareness. In contrast, introducing softening preserves early-stage exploration by preventing overconfident companion policies. This allows more balanced propagation of both positive and negative value signals, resulting in a more structured representation of risk regions and improved trade-off between goal reaching and collision avoidance.
(a) The step length convergence
(b) The collision convergence
Figure 6: Learning and evaluation curves of step length and collision rate under different KL regularization strengths and softening levels. Results are averaged over five independent runs. Softening the companion priors (ϵ = 0.3, 0.6) consistently improves the efficiency–safety trade-off, particularly under stronger KL coupling (smaller |η|).
Figure 6 reports averaged learning and evaluation curves over multiple runs for step length and collision rate under different KL strengths η and softening levels ϵ. Across all settings, softened klDMP consistently achieves a better balance between efficiency and safety. In particular, intermediate values such as ϵ ∈ [0.3, 0.6] provide faster convergence in step length while maintaining low collision rates. When KL regularization is strong (small |η|), 19
softening becomes especially beneficial, indicating that it stabilizes learning under aggressive coupling. These results suggest that companion-prior softening serves as an effective stabilization mechanism for KCPR. By reducing the influence of immature companion priors during early learning, the proposed criterion stabilizes learning while preserving the long-term benefits of policy coupling. The proposed criterion better balances exploration, safety, and goal-directed efficiency than both pure klMP and fully entropy-regularized baselines. 4.1.3. Effects of separate replay buffer We evaluate the effect of replay-buffer design by comparing MP and klMP-ϵ under three replay configurations: no replay buffer, a single shared replay buffer, and separate replay buffers for positive and negative experiences. Experiments are conducted in a 36×19 three-room grid-world environment containing both goal and punishment signals. The no-buffer setting performs online updates after each transition, while buffer-based variants use episodic experience collection with mini-batch updates. Figure 7 reports the learning curves across all methods and KL settings. Introducing any form of replay buffer significantly improves stability compared to the no-buffer baseline. More importantly, separating experiences by motivational source consistently accelerates convergence and improves both step efficiency and collision performance. A critical observation emerges under strong KL regularization (small |η|), where the coupling between rewardseeking and punishment-related learning becomes highly asymmetric. In this regime, both no-buffer and single-buffer variants fail to maintain stable navigation behavior, often collapsing into shortcut-dominated trajectories. In contrast, the separate-buffer design preserves informative negative transitions, enabling stable propagation of Q− and restoring balanced exploration between goal-seeking and risk-avoidance behaviors. Figure 8 further illustrates the effect of replay-buffer design on value propagation. While the learned reward-value function V + remains largely unchanged, separate replay buffers produce a substantially more structured punishment-value landscape than a single shared replay buffer. This indicates that separating positive and negative experiences facilitates the propagation of punishment-related value information without degrading reward learning. These results show that disentangled experience replay substantially improves learning under reward–punishment decomposition. By reducing interference between reward- and punishment-related experiences, the separate20
buffer design strengthens the propagation of punishment-related values and yields more stable learning.
(a) MP
(b) klMP-ϵ when ϵ = 0.3
Figure 7: Learning curves of step length and collision rate for MP and klMP-ϵ under three replay-buffer configurations (no buffer, single buffer, and separate buffers). The separate replay-buffer design consistently improves learning stability and convergence, with the largest benefit observed under stronger KL coupling (smaller |η|).
21
(a) klMP with one buffer when ϵ = 0.3, η = 1000
(b) klMP with separate buffer when ϵ = 0.3, η = 1000
Figure 8: Heatmaps of the learned reward-seeking and punishment-related state-value functions in the 36×19 Three-room maze using a single shared replay buffer and separate replay buffers. The separate replay-buffer design yields a more structured punishmentvalue landscape while preserving the reward-value function, leading to improved risk-aware learning.
4.2. Gazebo navigation
Figure 9: Three mazes with different complexity in Gazebo
We evaluate klDMP in simulated robotic navigation tasks using TurtleBot3 in ROS Gazebo. The objective is to investigate whether the proposed KL-coupled learning framework generalizes to continuous, noisy, and visually 22
Table 1: Comparison of reinforcement learning frameworks. Method DQN SQL DMP softDMP klDMP
Reward Signals Single Single Reward + Punishment Reward + Punishment Reward + Punishment
Prior Policy – Uniform – Uniform Companion
Motivational Interaction None None Value mixing Value mixing KCPR
Operator Max Soft Max/Min Soft KCSO
grounded navigation environments. Three maze environments of increasing complexity are considered: a U-maze, a T-maze, and a Three-room maze (Figure 9), which progressively increase the difficulty of long-horizon navigation, obstacle density, and ambiguity in safe path selection. In all environments, the robot is initialized with random orientations from a fixed start region and selects discrete motion primitives to navigate toward a goal cylinder (absorbing state). A sparse reward of +5 is provided upon reaching the goal, while collisions incur a penalty of −0.5. Observations consist of RGB images and LiDAR scans. All methods employ the same multimodal perception backbone for feature extraction to ensure a fair comparison. DQN, SQL, and softDMP estimate only action values on top of the shared representation, whereas klDMP additionally introduces dedicated value and policy heads for both the reward-seeking and punishment-related subsystems, implementing the KCPR/KCSO framework described in Section 3.6. Table 1 summarizes the main characteristics of the compared methods. The baseline methods are evaluated using their best-performing configurations reported in previous studies [7]. Specifically, SQL adopts the η = 0 ("mean") operator, while softDMP uses the best-performing regularization coefficient reported in the original work (η = 10000). For klDMP, both the KL regularization coefficient and the companion-prior softening coefficient are selected through parameter scanning in the present work, yielding the best overall performance at η = 1000 and ϵ = 0.6, respectively. Furthermore, since the Grid-world experiments demonstrated the superiority of the separate replay-buffer design for reward–punishment learning, all DMP-based methods employ separate replay buffers throughout the Gazebo experiments. For both softDMP and klDMP, behavior selection is generated by hard-max mixing of the reward-seeking and pain-avoiding value functions following [6]. All remaining hyperparameters are kept identical across methods. Figure 10 summarizes the navigation performance of all methods across the three environments. Although all methods successfully learn goal-reaching 23
80
1000 500
average collisions
step length
DQN SQL-0 softDMP-sep-10000 klDMP-sep-1000-eps0.6
1500
0 0
100
200 DQN SQL-0 softDMP-sep-10000 klDMP-sep-1000-eps0.6
collisions
600 400
60 40 20
200 0
0
100 episode
0
200
0
100
(a) T-maze learning curves 70
1000
60
500
average collisions
step length
1500
0
collisions
100
200 DQN SQL-0 softDMP-sep-1000 klDMP-sep-1000-eps0.6
400 200
600
DQN SQL-0 softDMP-sep-1000 klDMP-sep-1000-eps0.6
50 40 30
10 0 0
100 episode
0
200
(c) U-maze scattered metrics
200
400 600 average step length
2000 1500 1000 500 0
100
100
200 DQN SQL-0 softDMP-sep-1000 klDMP-sep-1000-eps0.5
600 400 200
1000
DQN SQL-0 softDMP-sep-1000 klDMP-sep-1000-eps0.5
120
0
800
(d) U-maze scattered metrics
average collisions
step length
500
20
0
collisions
300 400 average step length
(b) T-maze scattered metrics 80
0
200
80 60 40 20 0
0 0
100
episode
0
200
(e) 3R-maze scattered metrics
250
500
750 1000 1250 1500 average step length
1750
2000
(f) 3R-maze scattered metrics
Figure 10: Navigation performance of DQN, SQL, softDMP, and klDMP across the Tmaze, U-maze, and Three-room Gazebo environments. The left column shows the learning curves of trajectory length and collision rate, illustrating the different learning dynamics of the compared methods. The right column summarizes the final trade-off between navigation efficiency and safety using the average trajectory length and collision rate over evaluation episodes. All results are averaged over five independent runs.
24
behaviors, they exhibit fundamentally different trade-offs between navigation efficiency and safety. DQN produces the longest trajectories and the highest collision rates, particularly in the Three-room maze. Introducing entropy regularization through SQL substantially improves safety. As reported in our previous study [7], the SQL-η = 0 ("mean") operator avoids the extreme behaviors induced by conventional max and min operators, yielding more balanced action evaluation and consistently lower collision rates. However, because SQL optimizes only a single reward function, its safety emerges from entropybased averaging rather than an explicit representation of risk, resulting in conservative navigation and longer trajectories. softDMP further improves early learning by decomposing reward and punishment into separate value functions. One possible explanation is that the punishment-related subsystem initially exerts a stronger influence on the shared behavior policy, resulting in rapid collision reduction during early training. However, because the two subsystems remain independently optimized and interact only through the behavior policy, safety knowledge cannot directly influence reward-value learning. This limitation is particularly evident in the Three-room maze, where efficient and safe behaviors are more difficult to reconcile, leading to higher collision variability despite efficient trajectories. In contrast, klDMP consistently maintains competitive navigation efficiency while producing more stable collision performance than softDMP. Although convergence is slightly slower during early training, companion-policy regularization and KL-coupled Bellman updates enable the reward-seeking and punishment-related subsystems to interact directly during optimization. One possible explanation is that these optimization-level interactions stabilize value propagation as the companion policies become increasingly informative during training, leading to more consistent policy learning and, ultimately, more stable navigation performance. These results suggest that the primary benefit of KL coupling is not faster optimization but more stable coordination between reward-seeking and punishment-related learning. Compared with softDMP, which combines independently optimized subsystems only through the behavior policy, klDMP embeds the interaction directly into the optimization process, resulting in more stable navigation while maintaining competitive efficiency.
25
5. Conclusions This paper introduced KL-Coupled Policy Regularization (KCPR), a policy coordination framework for RPRL, together with its practical realization, klDMP. By treating companion sub-policies as dynamically learned priors, KCPR enables direct interactions between reward-seeking and punishmentrelated learning processes. We further derived KL-Coupled Soft Optimality (KCSO), which yields coupled soft-optimal policies and KL-regularized Bellman operators. Experiments in grid-world and Gazebo navigation tasks demonstrated that KL coupling improves safety and learning stability while maintaining competitive task performance. Beyond its algorithmic contribution, KCPR provides a new perspective on coordinating multiple decision policies through learned behavioral priors. Rather than optimizing reward and punishment independently, the proposed framework allows interacting policies to continuously shape one another’s optimization process. More broadly, these results suggest that policy coordination may serve as a useful design principle for reinforcement learning systems with multiple behavioral objectives. Several directions remain for future work. In the current framework, both the KL regularization strength and the softening parameter are manually specified. Developing adaptive mechanisms for regulating policy coupling and exploration may further improve robustness across tasks. Although this work focuses on discrete-action navigation, extending the proposed framework to continuous-control reinforcement learning remains an important direction for future work. More broadly, the same framework could naturally be extended to additional motivational systems, such as curiosity, uncertainty, and homeostatic regulation. References [1] B. Seymour, N. Daw, P. Dayan, T. Singer, R. Dolan, Differential encoding of losses and gains in the human striatum, Journal of Neuroscience 27 (2007) 4826–31. [2] B. Seymour, N. Daw, J. P. Roiser, P. Dayan, R. Dolan, Serotonin selectively modulates reward value in human decision-making, Journal of Neuroscience 32 (2012) 5833–42.
26
[3] E. Eldar, T. U. Hauser, P. Dayan, R. J. Dolan, Striatal structure and function predict individual biases in learning to avoid pain, Proceedings of the National Academy of Sciences of the United States of America 113 (2016) 4812–7. [4] S. Elfwing, B. Seymour, Parallel reward and punishment control in humans and robots: safe reinforcement learning using the maxpain algorithm, in: Proc. of the 7th Joint IEEE International Conference on Development and Learning and on Epigenetic Robotics, 2017, pp. 140–7. [5] J. Wang, S. Elfwing, E. Uchibe, Deep reinforcement learning by parallelizing reward and punishment using the maxpain architecture, in: Proc. of the 8th Joint IEEE International Conference on Development and Learning and on Epigenetic Robotics, IEEE, 2018. [6] J. Wang, S. Elfwing, E. Uchibe, Modular deep reinforcement learning from reward and punishment for robot navigation, Neural Networks 135 (2021) 115–26. [7] J. Wang, E. Uchibe, Reward-punishment reinforcement learning with maximum entropy, in: 2024 International Joint Conference on Neural Networks (IJCNN), IEEE, 2024, pp. 1–7. [8] M. G. Azar, V. Gómez, H. J. Kappen, Dynamic policy programming, Journal of Machine Learning Research 13 (2012) 3207–45. [9] R. Fox, A. Pakman, N. Tishby, Taming the noise in reinforcement learning via soft updates, in: Proc. of the 32nd Conference on Uncertainty in Artificial Intelligence, 2016. [10] T. Haarnoja, H. Tang, P. Abbeel, S. Levine, Reinforcement learning with deep energy-based policies, in: Proc. of the 34th International Conference on Machine Learning, 2017, pp. 1352–61. [11] T. Haarnoja, A. Zhou, P. Abbeel, S. Levine, Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, in: Proc. of the 35th International Conference on Machine Learning, 2018, pp. 1861–70.
27
[12] J. Peters, K. Mulling, Y. Altun, Relative entropy policy search, in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 24, 2010, pp. 1607–1612. [13] J. Schulman, S. Levine, P. Abbeel, M. Jordan, P. Moritz, Trust region policy optimization, in: International conference on machine learning, PMLR, 2015, pp. 1889–1897. [14] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov, Proximal policy optimization algorithms, arXiv preprint arXiv:1707.06347 (2017). [15] R. Bommasani, On the opportunities and risks of foundation models, arXiv preprint arXiv:2108.07258 (2021). [16] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, et al., Rt-1: Robotics transformer for real-world control at scale, arXiv preprint arXiv:2212.06817 (2022). [17] B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahid, et al., Rt-2: Vision-language-action models transfer web knowledge to robotic control, in: Conference on Robot Learning, PMLR, 2023, pp. 2165–2183. [18] K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al., pi_0: A vision-language-action flow model for general robot control, arXiv preprint arXiv:2410.24164 (2024). [19] J. Karlsson, Learning to solve multiple goals, Ph.D. thesis, University of Rochester, 1997. [20] M. Humphrys, Action selection methods using reinforcement learning, in: From Animals to Animats 4: Proceedings of the Fourth International Conference on Simulation of Adaptive Behavior, 1996, pp. 135–144. [21] N. Sprague, D. Ballard, Multiple-goal reinforcement learning with modular sarsa(0), in: Proc. of the 18th International Joint Conference on Artificial Intelligence, 2003, pp. 1445–1447.
28
[22] H. van Seijen, M. Fatemi, J. Romoff, R. Laroche, T. Barnes, J. Tsang, Hybrid reward architecture for reinforcement learning, in: Advances in Neural Information Processing Systems 30, 2017. [23] Z. Lin, D. Yang, L. Zhao, T. Qin, G. Yang, T.-Y. Liu, Rd2 : Reward decomposition with representation decomposition, in: Advances in Neural Information Processing Systems 33, 2020. [24] H. Okada, H. Yamakawa, T. Omori, Two dimensional evaluation reinforcement learning, in: International Conference on Artificial Neural Networks, Springer, 2001, pp. 370–377. [25] R. Lowe, T. Ziemke, Exploring the relationship of reward and punishment in reinforcement learning, in: Proc. of the 2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL), IEEE, 2013, pp. 140–147. [26] T. Kobayashi, T. Aotani, J. R. Guadarrama-Olvera, E. Dean-Leon, G. Cheng, Reward-punishment actor-critic algorithm applying to robotic non-grasping manipulation, in: 2019 Joint IEEE 9th International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob), IEEE, 2019, pp. 37–42. [27] B. Lin, G. A. Cecchi, D. Bouneffouf, J. Reinen, I. Rish, A story of two streams: Reinforcement learning models from human behavior and neuropsychiatry, in: Proc. of the 19th International Conference on Autonomous Agents and Multi-Agent Systems, 2020, pp. 744–752. [28] B. Liebenow, R. Jones, E. DiMarco, J. D. Trattner, J. Humphries, L. P. Sands, K. P. Spry, C. K. Johnson, E. B. Farkas, A. Jiang, et al., Computational reinforcement learning, reward (and punishment), and dopamine in psychiatric disorders, Frontiers in Psychiatry 13 (2022) 886297. [29] S. Kullback, R. A. Leibler, On information and sufficiency, The Annals of Mathematical Statistics 22 (1951) 79–86. [30] K. Asadi, M. L. Littman, An alternative softmax operator for reinforcement learning, in: Proc. of the 34th International Conference on Machine Learning, 2017. 29
[31] A. A. Rusu, S. G. Colmenarejo, C. Gulcehre, G. Desjardins, J. Kirkpatrick, R. Pascanu, V. Mnih, K. Kavukcuoglu, R. Hadsell, Policy distillation, arXiv preprint arXiv:1511.06295 (2015). [32] E. Parisotto, J. L. Ba, R. Salakhutdinov, Actor-mimic: Deep multitask and transfer reinforcement learning, arXiv preprint arXiv:1511.06342 (2015). [33] Y. Teh, V. Bapst, W. M. Czarnecki, J. Quan, J. Kirkpatrick, R. Hadsell, N. Heess, R. Pascanu, Distral: Robust multitask reinforcement learning, Advances in Neural Information Processing Systems 30 (2017). [34] H. Van Seijen, H. Van Hasselt, S. Whiteson, M. Wiering, A theoretical and empirical analysis of expected sarsa, in: Proc. of the IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning, IEEE, 2009, pp. 177–184. [35] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, K. Kavukcuoglu, Asynchronous methods for deep reinforcement learning, in: International conference on machine learning, PmLR, 2016, pp. 1928–1937. [36] J. Garcia, F. Fernandez, A comprehensive survey on safe reinforcement learning, Journal of Machine Learning Research 16 (2015) 1437–1480. [37] S. Gu, L. Yang, Y. Du, G. Chen, F. Walter, J. Wang, A. Knoll, A review of safe reinforcement learning: Methods, theory and applications, arXiv preprint arXiv:2205.10330 (2022).
30