Optimizing Sparse Outcomes Through Dense Behavioral Signals via Value-Guided Preference Distillation

Aligning multi-turn dialogue agents is usually framed as matching turn-level human preferences, yet direct optimization of long-term outcomes is often ineffective and prone to reward hacking. We formulate long-horizon dialogue optimization as a multi-objective reinforcement learning problem and trai