Better Reward Models Are Not Always Less Biased: Social Alignment Beyond Standard Reward Benchmarks

Reward models are central to LLM alignment, yet current benchmarks reveal little about how they represent diverse and sometimes competing social preferences. We extend reward model evaluation to four socially consequential domains: bias, safety, morality, and ethical reasoning. Our analysis shows th