Official Review · ICLR 2025

This paper employs a method proven effective in training language models to train embodied intelligence models, using reinforcement learning to further refine the model. Following Supervised Fine-Tuning (SFT), the model undergoes an online self-improvement process. In the first phase, the model is trained to clone actions and predict the remaining steps to action success. In the second phase, the predicted remaining steps to task completion serve as a reward for training, enabling reinforcement learning-based optimization. The step count to task completion is also predicted by the model, achieving self-improvement. Validation on the PaLI model showed that the self-improvement procedure increased action success rates by over 1.5 times. This paper introduces reinforcement learning methods into the training of embodied intelligence models, offering a training approach that is more sample-efficient than supervised learning alone. The two-stage training method improves training efficiency, and the second phase enables the model to learn a broader range of actions, enhancing its generalizability. The reinforcement learning section provides detailed explanations of parameter settings. Experiments with the PaLI model validate the practical effectiveness of this training method, supported by real-world robot testing. The paper lacks a concluding section, which might impact the overall readability and leave readers without a clear summary of the findings and contributions. Additionally, the arguments supporting the model's generalizability are somewhat limited. The reinforcement learning section would also be strengthened by including more theoretical derivations to better explain the underlying principles. 1) In the paper, which specific actions are included in the action steps mentioned? Is there a fixed set of actions? When training the model to perform new types of tasks, is it possible to add new actions, or is it limited to the predefined action range? 2) In the middle subfigure of Figure 3, it seems that these exists a more orange box. 3) The authors mentions that the main work is to design an effective and sample-efficient procedure for fine-tuning pretrained multimodal foundation models. Are there any requirements or limitations for multimodal foundation models？What is the impact of the parameter size of the multimodal foundation models on the results, and will the types of multimodal data involved affect the results？ 4）In the stage 2, the authors use an online RL to train. Has the online training process considered the safety and coherence of the generated action set?