Official Review · NEURIPS 2024

This paper addresses the issue that humans cannot always provide helpful demonstrations or supervision on tasks beyond their expertise. Based on the observation that evaluation is easier than generation, the authors propose "easy-to-hard generalization," training a verifier on easy tasks and leveraging its generalization ability to supervise the generator on hard tasks. Experimenting mainly in math reasoning tasks, they demonstrate that easy-to-hard generalization from evaluators can enable easy-to-hard generalization from generators. The paper introduces the Outcome & Process Reward Model (OPRM), which harnesses the complementary strengths of ORMs and PRMs. Experiments show that OPRM is more efficient. It also conducts a systematic experimental setup, testing various generators and evaluators, as well as optimization algorithms (BoN, Weighted Voting, and RL). It provides numerous experimental analyses in scalable alignment. While the paper is well-written and presents a solid analysis, several weaknesses need addressing: 1. What is the difference between "easy-to-hard" and "weak-to-strong"? You state that human supervision is available but not reliable in the weak-to-strong setting, but in the [OpenAI paper](https://cdn.openai.com/papers/weak-to-strong-generalization.pdf) it says, "We find that when we naively finetune strong pretrained models on labels generated by a weak model, they consistently perform better than their weak supervisors, a phenomenon we call weak-to-strong generalization." In that study, they finetune GPT-4 with a GPT-2-level supervisor. Can a GPT-2-level supervisor be seen as a verifier on easy tasks? The novelty should be considered. 2. In Table 1, the Full ICL setting performs worse than the Easy-To-Hard ICL setting. How do you explain this? The intuition is that sampling from both easy and hard exemplars may help solve hard problems more effectively than just demonstrating easy exemplars. Although the quality of PRM800K is lower than MetaMATH, your explanation for why the Full ICL setting is worse than the Easy-To-Hard ICL setting is insufficient (line 198). 3. Can you report the average accuracy of verifying each step in a solution when you evaluate the evaluators? Section 3.5 does not explain why this is not reported. 4. When citing images and sections, consider adding automatic jumps for easier navigation. 1. What is the difference between "easy-to-hard" and "weak-to-strong"? 2. In Table 1, the Full ICL setting performs worse than the Easy-To-Hard ICL setting. How do you explain this? 3. In Table 1, the Easy-To-Hard SFT setting performs slightly worse than the Full SFT setting, which is expected. Do you think the evaluation of the Easy-To-Hard setting is influenced more by the format of problems or by the true grasp of the principles of solving hard tasks? Are there any better evaluation methods? This is an open question, and I would appreciate your thoughts on it.