Official Review · ICLR 2025

**Summary** This paper presents a novel method that leverages SAEs to steer the generalization behavior of language models during fine-tuning. By identifying and ablating latent features associated with unintended generalizations (such as gender bias and task misalignment in double multiple choice scenarios), the authors aim to guide the model toward the intended behavior. Experiments on a gender bias task and a double multiple choice task show that the method can significantly improve out-of-distribution performance relative to baselines, including random ablations and interventions applied only at test time. **Strengths** * **Innovative Methodology**: Combines SAEs with targeted ablation to steer model behavior, offering a novel solution to generalization control. * **Empirical Validation**: Results on toy tasks—specifically in mitigating spurious correlations in gender bias and improving focus in double multiple choice tasks—demonstrate the potential of the proposed approach. * **Strong Results**: Demonstrates near-complete elimination of gender bias and significant improvements in 12/16 double-choice scenarios. **Weaknesses** * Computational Overhead: The reliance on SAEs and the associated markup for interpreting and ablating features introduces additional compute requirements. The paper does not sufficiently quantify this overhead or discuss its impact on scalability. * Scalability Concerns: Experiments are limited to a 2B-parameter model and synthetic tasks; applicability to larger models or real-world scenarios is unclear. * Limited Baselines: There is a noticeable absence of comparisons with alternative methods that do not rely on SAE-based approaches. Such comparisons could clarify whether the performance gains are specific to SAE techniques or are achievable through other means. **Questions** * How much does the SAE-based pipeline increase training time compared to standard fine-tuning? Quantifying this would clarify practicality. * Have you explored alternative methods that might offer similar benefits? A discussion and comparisson on possible non-SAE alternatives would enhance the paper. * In instances where ablation did not improve performance, could you provide further diagnostics or insights to help understand these shortcomings?