Reason-guided Human Reward Modeling for Preference-based Reinforcement Learning

Reason-guided Human Reward Modeling for Preference-based Reinforcement Learning Anshul Nayak ∗ , Shahil Shaik ∗ , Jonathon Smereka ∗∗ , Yue Wang ∗

arXiv:2610.23550v1 [cs.RO] 16 Oct 2026

∗ Clemson University, Clemson, SC 29631 USA (e-mail: anshuln@clemson.edu; shahils@clemson.edu; yue6@clemson.edu). ∗∗ U.S. Army DEVCOM Ground Vehicle Systems Center, Warren, MI 48092 USA (e-mail: jonathon.m.smereka.civ@army.mil).

Abstract: Reinforcement learning (RL) policies optimized for task rewards may produce behaviors that are efficient but misaligned with human expectations, safety preferences, or intent. This paper presents a reason-guided preference-based reinforcement learning (PbRL) framework that learns a human reward model from trajectory-level feedback. In addition to preference labels, the framework incorporates reason tags explaining why a behavior is preferred, enabling the learned reward to capture both human choices and their underlying rationale. Compared with preference-only reward learning, the reason-guided formulation provides additional supervision for learning human-aligned rewards and improves the interpretability of the resulting policy. The learned human reward is combined with the environment reward during policy optimization, allowing the agent to improve task performance while producing more interpretable, humanaligned, and preference-sensitive behavior. Keywords: Reinforcement-learning from Human Feedback (RLHF), Reward Modeling, Human-robot Interaction (HRI), Preference-based Reinforcement Learning (PbRL) 1. INTRODUCTION

Human preference plays a pivotal role in human-robot collaboration (Wang et al. (2025)). A robot optimized only for task performance may generate behaviors that are efficient from the robot’s perspective but difficult for humans to trust or accept. For example, in social navigation, a robot may minimize travel time by passing close to obstacles or moving aggressively near humans, even though such behavior may appear unsafe or uncomfortable. In these scenarios, the quality of a decision should not be determined only by task performance, but also by whether the behavior aligns with human expectations, safety concerns, and preferences. Classical approaches to human-aware decision-making have used optimal control, probabilistic inference, and temporal models to account for uncertainty in humanrobot interaction (HRI). In particular, Hidden Markov Models (HMM), Dynamic Bayesian Networks (DBN) (Xu and Dudek (2015)), and Trust-POMDPs (Chen et al. (2018)) have been used to model latent human states such as intent, workload, or trust, and to update robot decision-making based on observations of human behavior or feedback. Although these models provide explicit uncertainty modeling, they often require carefully designed transition models, observation models, and reward functions. Online adaptation can also become challenging in

⋆ This research was supported, in part, by Cooperative Agreement W56HZV-24-2-0001 U.S. Army DEVCOM Ground Vehicle Systems Center.

high-dimensional or long-horizon settings, where repeated human queries and online re-planning may increase cognitive load and computational cost. Alternatively, learning from human feedback has emerged as an effective alternative for aligning autonomous agents with human-preferred decisions (Poddar et al. (2024)). In robotics, preference-based reinforcement learning (PbRL) allows humans to evaluate robot behaviors at the trajectory level and provide feedback that can be used to learn a reward model (Liu et al. (2023)). Reinforcement Learning from Human Feedback (RLHF) extends this idea by incorporating the learned human reward during policy optimization, enabling the robot to optimize not only for task completion but also for human-preferred behavior (Liu et al. (2023)). This is particularly important in HRI, where the environment reward may specify what the robot should accomplish, while human feedback provides information about how the task should be accomplished. However, preference labels alone provide limited information about the factors that drive human decisions. A human may prefer one trajectory over another because it maintains safer obstacle clearance, follows a smoother path, avoids low-traversability regions, or better balances efficiency and safety. Standard preference-based reward learning captures which behavior is preferred, but does not explicitly model why it is preferred (Wang (2026)). Prior work has explored interpretable PbRL through structured reward representations, such as tree-based reward functions (Bewley and Lecue (2021)), while broader studies

on human-AI interpretability emphasize the importance of explaining agent behavior in terms that align with human expectations (Sreedharan et al. (2021)). However, standard preference-based reward learning typically does not use human-provided rationale as direct supervision for reward learning.

2.2 Preference-based RLHF

Preference-based RLHF uses trajectory-level human feedback to train a human reward model offline. A trajectory clip is a finite sequence of state-action transitions τ = (s0 , a0 , s1 , a1 , · · · , sT ), where T is the clip horizon length. For each transition, we construct an observable feature To address this limitation, we propose a reason-guided vector x = ϕ(s , a , s ) ∈ Rd , where ϕ(·) is a feature t t t t+1 preference-based RLHF framework for human-aware robotic extraction function. The feature vector xt represents taskdecision-making. In addition to trajectory preference la- and safety-relevant quantities observable by the agent, bels, humans provide reason tags that explain why a be- such as progress, traversability, distance to goal, minimum havior is preferred, such as predictability, legibility, safety, collision distance, and other domain-specific features. For smoothness, or task efficiency. These reason labels provide a clip τ , the corresponding sequence of features are auxiliary supervision during offline reward learning, imX(τ ) = [x0 || x1 || · · · || xT −1 ] ∈ RT ×d , (3) proving the interpretability of the learned human reward model and supporting downstream policy optimization. where || denotes concatenation along the temporal diThe learned human reward is then combined with the mension. The human reward model is parameterized by task reward during reinforcement learning, enabling the a neural network with trainable parameters θ. It maps robot policy to better balance task performance, safety, the per-step feature vector xt to a scalar human reward rθH (xt ) : Rd → R. A simple neural reward model can be and human-aligned behavior. written as The main contributions of this paper are as follows: ht = fθenc (xt ), (4) H rew • We formulate a preference-based RLHF framework r̂θ (xt ) = fθ (ht ), (5) for human-aware robotic decision-making, where tra- where f enc (·) denotes the feature encoder and f rew (·) θ θ jectory level human feedback is used to learn an of- denotes the scalar human reward head. fline reward signal that complements the task reward In addition to predicting the scalar per-step human reduring policy optimization. • We develop a reason-guided human reward model ward, the model also predicts the reason labels associated that jointly predicts trajectory preferences and reason with the preferred trajectory clip. For this, a reason pretags, enabling the robot to learn both which behaviors diction head is added on top of the aggregated trajectory representation: humans prefer and why they are preferred.
ĉ(τ ) = σ fθreason (h̄τ ) , (6) • We show that incorporating reason-based feedback provides richer supervision than preference-only re- where σ(·) is the logistic sigmoid function, ĉ(τ ) ∈ [0, 1]K ward learning, improving reward interpretability and is the predicted probability vector over K reason labels, supporting policies that better balance task perfor- f reason (·) is the reason classification head, and h̄ is τ θ mance, safety, and human-aligned behavior. obtained by aggregating the per-step embeddings: T −1

h̄τ =

2. METHODS 2.1 Problem Statement An RL agent must learn to leverage human preferences or feedback to improve the policy so that the agent prefers human-aligned policy. We consider the episodic agent navigating in an environment as a MDP M = {S, A, P, r, γ} where st ∈ S is the agent state and at ∈ A is a continuous action with transition probability st+1 ∼ P (· |st , at ). The environment gives reward rtenv = renv (st , at , st+1 ). However, in preference-based RLHF, human might prefer an action aligned with their own belief and preferences contrary to an optimal policy seeking safety over performance. Therefore, we design the reward as a combination of learned human model and environment reward per step. rtπ = rtenv + λH r̂tH

(1)

where λH controls the human preference strength and r̂tH represents the predicted human scalar reward at time t. The goal is to learn a policy πφ (at | st ) with the objective of maximizing a joint objective of task performance and human aligned preference. " T # X ∗ t π π = argmax Eτ ∼π γ rt (2) π

t=0

1 X ht . T t=0

(7)

Let the trajectory clip be denoted by τ and the corresponding sequence of clip features be X(τ ). The predicted human reward for the full trajectory clip is computed by aggregating the per-step reward predictions: T −1 X R̂θH (X(τ )) = γ t r̂θH (xt ), (8) t=0

where γ ∈ [0, 1] is a discount factor. In the current preference-based reward learning, humans compare trajectory clips generated under identical environmental conditions, including start and goal locations and obstacle configurations. For each pair {τiA , τiB }, where i indexes an environment configuration, the human selects the preferred trajectory based on the agent’s behavior. The reward model fθrew (·) is then trained to align its predictions with these preferences. The human preference index is represented as  1, if τiA ≻ τiB , yi = (9) 0, if τiB ≻ τiA . where τiA ≻ τiB implies trajectory A is preferred over B. Let ci ∈ {0, 1}K denote the multi-hot encoded reason vector for the human-preferred trajectory in the i-th

Fig. 1. Offline Human Preference Learning: Human observes a pair of clips and records preference as well as reason labels. NN model is trained offline to predict preference index f rew using cross entropy (CE) and reason tags f reason using binary cross entropy (BCE). Policy Optimization: Learned human reward and task reward jointly perform policy optimization that is both task-efficient and aligned with interpretable human feedback. comparison, where K is the number of predefined reasons. Each entry ci,k = 1 indicates the selection of reason k (0 otherwise), allowing multiple reasons to be selected simultaneously for a single preference. The reward model assigns each clip a scalar score RA = R̂θH (τiA ) and RB = R̂θH (τiB ). The probability that the human prefers trajectory clip τiA over τiB is modeled using the Bradley-Terry preference model (Rafailov et al. (2023)) as Pθ (τiA ≻ τiB ) = σ(RA − RB ), (10) The reward model is trained by minimizing the crossentropy (CE) loss over the preference dataset DH = {(τiA , τiB , yi , ci )}N i=1 . The preference loss using the reward difference as the logit is represented as N 1 Xh Lpref (θ) = − yi log σ(RA − RB )+ N i=1 (11) i (1 − yi ) log σ(RB − RA ) . Preference comparisons constrain only relative ordering between trajectory clips and do not uniquely identify the per-step reward function. Hence, we regularize the learned reward to reduce large reward magnitudes and high temporal variation before using it for policy optimization. "

N

1 X Lreg (θ) = N i=1

ĉi = yi ĉ(τiA ) + (1 − yi )ĉ(τiB ).

X

(12)

The first term controls reward magnitude, while the second encourages temporal smoothness. In addition, reason labels provide semantic supervision for reward learning. For the preferred clip, the predicted

(13)

The reason-guided binary cross-entropy (BCE) loss is defined as N K 1 XXh ci,k log ĉi,k + Lreason (θ) = − N i=1 (14) k=1 i (1 − ci,k ) log(1 − ĉi,k ) . This loss encourages the reward model to capture not only which trajectory is preferred, but also why it is preferred. The final offline reward-model training objective is LH (θ) = Lpref (θ)+αreason Lreason (θ)+βreg Lreg (θ), (15) where αreason and βreg are weights for the reason and regularization loss terms, respectively.

2.3 Policy Optimization The trained model r̂θH∗ is then used as a learned human reward function during agent learning where the human reward model parameters θ∗ are frozen. At each time step, the reward is queried per-step for policy optimization as rttotal = r˜t env + λH r̃θH∗ (xt ),

T −1

2 1 X H rθ (xt ) + T t=0 τ ∈{τiA ,τiB } # T −2
2 1 X H H r (xt+1 ) − rθ (xt ) . T − 1 t=0 θ

reason vector is

(16)

where, r̃ represents normalized per-step scalar reward. The policy πφ (at | st ) is trained to maximize the objective: T X J(φ) = Eτ ∼πφ [ γ t (r˜t env + λH r̃θH∗ (xt ))]

(17)

t=0

where J(φ) represents the expected discounted return and the trainable policy parameters φ are optimized using gradient descent ∇φ J(φ).

3. EXPERIMENTS 3.1 Data Collection We designed an off-road single-agent navigation simulator. The overall task for the agent is to navigate to the goal (xg , yg ) while adhering to robot dynamics and obstacles. The agent can navigate using pre-trained RL policy or model-based control approaches. The robot has stochastic dynamics and the robot pose [x, y, θ]T evolves under control command [ω, v]. The environment also contains static obstacles with varying sizes and configuration to induce collision avoidance. . Pre-trained RL policy or model-based planners are used to create the offline preference dataset. The dataset D contains multiple sampled clips along with per-step features xt for the same start and goal locations and obstacle configurations. The human is shown pair-wise episodic clips {τA , τB } and needs to select the preferred trajectory along with reason labels for the selection. For, a simple go-To-goal environment with obstacle avoidance, the reason labels for preferred trajectory can be due to ”safe clearance”, ”shorter path”, ”smoothness” and so on. The dataset stores the preferred clip index {0, 1} along with the reason labels represented as multi-hot encoded vector for each query.

Table 1. Hyperparameters Parameter

Value

Hidden layer dim Optimizer Learning Rate Effective Batch Size epochs λH γ αreason βreg

256 SGD 2 × 10−4 8 20 [0, 0.5] 0.9 0.2 1e − 3

(a) safe clearance to obstacles

3.2 Training Implementation A simple MLP with one hidden layer maps the clip-level input features X(τ ) into embedded representation. The output contains two heads, a scalar reward head, fθrew that minimizes the cross entropy (CE) loss between predicted and actual human preferred index as well as a reason head fθreason that minimizes binary cross entropy (BCE) loss across multiple labels simultaneously (Fig.1). The dataset D contains dedicated human annotation across 350 queries with 90% queries used for training and 10% for validation. The parameters of the trained human reward model R̂θH∗ are frozen and queried at step-level during policy training. The training hyperparameters are shown in Table.1 3.3 Policy Training The policy πφ (at | st ) is trained on a vectorized 2D off-road environment for go-To-goal task with obstacle avoidance. The normalized task reward r̃env is designed to encourage progress toward the goal while penalizing unsafe or inefficient behavior. The environment provides a large positive terminal reward for reaching the goal and large negative terminal penalties for collisions or going out-of bounds. Reward shaping include progress towards goal, obstacle-proximity and control-related penalties for inducing trajectory smoothness. Adding human reward r̃θH∗ and training the policy with rtotal (16) is expected to align the agent’s decision with human’s preference. 3.4 Reason-guided Ablation To isolate the contribution of reason supervision and reward regularization, we train the human reward model offline under four ablation objectives: preference loss

(b) smoother and shorter path

Fig. 2. Policy Evaluation Comparison: successful rollout comparisons between the environment-only and the combined environment and human policy under similar environment conditions. (Lpref ) only, preference loss with reward regularization (Lpref + Lreg ), preference loss with reason supervision (Lpref + αreason Lreason ), and preference loss with both reason supervision and reward regularization (Lpref + αreason Lreason + Lreg ). After training, each reward model is saved and kept frozen during policy optimization. The downstream policy is then trained using the combined environment and learned human reward, where the human reward contribution is scaled by λh ∈ {0.1, 0.2}. This design separates reward-model learning from policy optimization and allows us to evaluate whether reasonguided and regularized reward models produce policies with better task performance, obstacle clearance, path efficiency, and success rate (agent avoided collisions and reached goal) compared with the environment-only simple preference (Lpref ) baseline.

4. RESULTS This section evaluates whether incorporating the learned human reward improves the alignment of the trained policy towards human preferences when compared against a policy optimized using only the task reward. Initially, we compare two reward settings: (i) an environment-only reward baseline, denoted as Renv , and (ii) a combined reward formulation, denoted as Renv +λH R̂θH , where R̂θH is obtained by querying the learned human feedback model. Secondly, we also highlight the importance of reasonguided reward training (sec. 3.4) and regularization over simple preference based training. We evaluate the policies under the same environment configuration comparing human reward model trained only using preference loss, preference and reason loss and finally, regularized preference and reason loss. We analyze policy rollouts across multiple scene and policy seeds using task-level and safety-related trajectory metrics. 4.1 Rollout-Based Policy Comparison

Path length 11.02 ± 0.96 10.92 ± 0.19

(b) Validation accuracy

(c) Policy Training

To compare the behavior induced by the two reward formulations, we evaluate the trained policies using a fixed rollout protocol. For each method, we generate 30 policy rollouts across 4 scene seeds and multiple policy seeds, using the same start and goal conditions along with obstacle configuration to ensure a fair comparison. This setup allows us to compare the behavior of the task-only policy and the human-reward-augmented policy under same environmental conditions. Table 2. Reward metrics for rollout evaluation at λH = 0.2. Policy Renv Renv + λH Rhuman

(a) Training loss

Success (%) 67.5 ± 4.2 77.5 ± 5.7

Figure 2 shows representative trajectory rollouts for two reward formulations, Renv and Rtot = Renv + R̂θH . Each scene has same environment configuration and only successful rollouts are considered for both the policies such that the agent reaches goal to effectively capture human preference. In Fig.2a, the environment-only policy generates a path that is much closer to the obstacle while the policy trained with Rtot generates a smoother path while maintaining safe clearance with obstacles. This aligns with human preference data where the human preferred paths that are ”shorter”, ”smoother” and had ”safer clearance” from obstacles when selecting among a pair of trajectories. Similarly, Fig.2b shows the combined human and environment reward model prefers a shorter and smoother path while reaching the goal. Since, the environment reward does not explicitly encode human preference, it may yield trajectories closer to obstacles, while combined reward Renv + λH R̂θH∗ encourages trajectories that better align with human preferences for safer obstacle clearance and smoother motion. We quantitatively evaluate each policy over four held-out scene seeds and report the mean and standard deviation of trajectory-level metrics. The table summarizes combined reward, obstacle clearance, path length, and task success

Fig. 3. Reward-model validation and policy-training curves for ablation variants. rate. Combined reward measures the total rollout return under the evaluation reward, while obstacle clearance captures the minimum separation from obstacles or unsafe regions along the trajectory. Path length measures navigation efficiency, and success rate reports the percentage of rollouts in which the robot reaches the goal without collision within the allowed horizon. The learned human reward policy achieves a shorter average path length while maintaining a higher success rate, suggesting more efficient and smoother trajectories aligning with human preferences.

4.2 Reason-guided Reward Training Figure 3 presents the reward-model training, validation, and downstream policy-training curves across the ablation variants. The decreasing training loss and improving validation accuracy indicate stable reward learning across objectives. The reason-guided variants incorporate additional semantic supervision, while reward regularization constrains reward magnitude and temporal variation without destabilizing training. Table.3 evaluates the effect of these objectives on downstream policy performance. At λH = 0.1, combining reason supervision and regularization improves task success from 66.7 ± 4.7% to 78.3 ± 2.4% compared with preference-only learning, while increasing obstacle clearance from 1.82 ± 0.20 to 1.99 ± 0.30. At λH = 0.2, the combined objective achieves the highest obstacle clearance (2.18±0.07) with a path length of 11.20 ± 1.33. However, regularization alone achieves the highest success rate (81.7±7.1%), highlighting the trade-offs between safety and task completion. Overall, the results suggest that reason supervision and regularization can improve downstream policy behavior, although their relative benefits depend on the human-reward weight and evaluation metric.

Table 3. Ablation study comparing reward-model training objective. Preference-only baseline Lpref , while environment+human rows use the corresponding ablation objective. λH

Training objective

Obstacle clearance

Path length

Success rate (%)

subjective Evaluation (%)

0.1 0.1 0.1 0.1

Lpref Lpref + αreason Lreason Lpref + Lreg Lpref + αreason Lreason + Lreg

1.82 ± 0.20 1.90 ± 0.27 1.80 ± 0.23 1.99 ± 0.30

11.44 ± 1.46 11.25 ± 0.52 11.16 ± 0.36 11.29 ± 0.87

66.7 ± 4.7 73.3 ± 0.0 75.0 ± 2.4 78.3 ± 2.4

33.1 ± 6.4 65.4 ± 7.2 68.3 ± 8.1 67.2 ± 5.1

0.2 0.2 0.2 0.2

Lpref Lpref + αreason Lreason Lpref + Lreg Lpref + αreason Lreason + Lreg

1.78 ± 0.14 2.01 ± 0.14 2.05 ± 0.05 2.18 ± 0.07

11.72 ± 1.33 11.65 ± 0.52 12.04 ± 1.11 11.20 ± 1.33

65.0 ± 2.4 68.3 ± 2.4 81.7 ± 7.1 73.3 ± 4.7

23.1 ± 4.9 66.6 ± 4.7 81.8 ± 4.1 80.0 ± 4.7

4.3 Blind subjective study Three annotators independently assessed matched trajectory pairs in a blinded evaluation. Trajectories were presented with neutral A/B labels with policy randomized such that policy identities and reward values hidden from annotator. Each annotator selected a preferred trajectory or skipped the comparison; responses were subsequently unblinded using the stored ground truth mappings. Preference rates were computed by aggregating choices across multiple runs and seeds within each annotator, then mean and sample standard deviation across the three annotators are reported in Table.3. Adding reason supervision to the preference loss improved both task success and subjective evaluation: subjective scores increased from 33.1 ± 6.4% to 65.4 ± 7.2% and from 23.1 ± 4.9% to 66.6 ± 4.7% for lambdas 0.1 and 0.2 respectively. The Lpref +Lreg objective also performed comparably to the full objective, achieving subjective scores of 68.3 ± 8.1% versus 67.2 ± 5.1% at λH = 0.1, and 81.8 ± 4.1% versus 80.0 ± 4.7% at λH = 0.2. These results support the benefit of reason supervision for learning rewards aligned with human preferences, while showing that preference learning with regularization provides a strong alternative to the full objective.

the framework to HRI settings with richer reward signals and trust-based reward for multi-agent robotic teams. REFERENCES

Bewley, T. and Lecue, F. (2021). Interpretable preferencebased reinforcement learning with tree-structured reward functions. arXiv preprint arXiv:2112.11230. Chen, M., Nikolaidis, S., Soh, H., Hsu, D., and Srinivasa, S. (2018). Planning with trust for human-robot collaboration. In Proceedings of the 2018 ACM/IEEE international conference on human-robot interaction, 307–315. Liu, Y., Datta, G., Novoseller, E., and Brown, D.S. (2023). Efficient preference-based reinforcement learning using learned dynamics models. arXiv preprint arXiv:2301.04741. Poddar, S., Wan, Y., Ivison, H., Gupta, A., and Jaques, N. (2024). Personalizing reinforcement learning from human feedback with variational preference learning. Advances in Neural Information Processing Systems, 37, 52516–52544. Rafailov, R., Sharma, A., Mitchell, E., Manning, C.D., Ermon, S., and Finn, C. (2023). Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 53728–53741. Sreedharan, S., Kulkarni, A., Smith, D.E., and Kamb5. CONCLUSION hampati, S. (2021). A unifying bayesian formulation of measures of interpretability in human-ai. arXiv preprint This paper presents a reason-guided preference-based arXiv:2104.10743. RLHF framework for human-aware robotic decision-making. Wang, J., Yan, Y., Hu, Y., and Yang, X. (2025). A The proposed approach learns an offline human reward reinforcement learning from human feedback based model from trajectory-level preference feedback and augmethod for task allocation of human robot collaboments preference learning with reason labels that explain ration assembly considering human preference. Adwhy a behavior is preferred. By jointly learning trajecvanced Engineering Informatics, 66, 103497. doi: tory preferences and their associated rationale, the rehttps://doi.org/10.1016/j.aei.2025.103497. ward model provides a more interpretable representation Wang, R. (2026). Preference-based reinforcement learning of human-aligned behavior than preference-only reward for robotics: Advancements and open challenges. Aulearning. The learned human reward is then combined thorea Preprints. with the task reward during policy optimization, allow- Xu, A. and Dudek, G. (2015). Optimo: Online probaing the robot to account for both task performance and bilistic trust inference model for asymmetric humanhuman-sensitive criteria such as safety, smoothness, prerobot collaborations. In Proceedings of the tenth annual dictability, and traversability. The results show that inACM/IEEE international conference on human-robot corporating learned human feedback can influence policy interaction, 221–228. behavior beyond what is obtained from the task reward alone. In particular, the reason-guided reward model provides additional supervision during offline reward learning and supports policies that better balance efficiency with human-aligned behavioral preferences. The ablation study further highlights the role of reason supervision and reward regularization in shaping the learned reward model and downstream policy performance. Future work will extend

Related documents

• RPS: Information Elicitation with Reinforcement Prompt Selection#14045
• Official Review · ICLR 2026#787023
• Specifying Reward Functions for RL Without Environment Sampling#1109541
• Learning Process Rewards via Reasoning State Propagation#1144187
• Using Context Is Not Enough: Test-Time Training for Personalized Reward Modeling#1129710