Understanding Helpfulness and Harmless Tension in Reward Models

Reward models (RMs) are a key component of reinforcement learning from human feedback, enabling the alignment of large language models with human values such as helpfulness and harmlessness. However, these alignment goals can conflict, leading to interference when optimised jointly. We refer to this