Official Review · ICLR 2026

This paper studies the generalization error of linear regression using spiked covariance data models, characterizing the impacts of spike strength, target-spike alignment, target model misspecification, and train-test covariate shift. The theoretical output of the paper is an exact expression for the generalization error in the considered setting. This allows the authors to identify the regimes where spike strength and target-spike alignment lead to improved generalization. 1. Rigorous asymptotic analysis of the generalization error of linear regression in a setting involving multiple interesting variables (i.e., spike strength, target-spike alignment, target model misspecification, and train-test covariate shift). 2. The writing of the paper (including figures and theoretical results) is of good quality. Therefore, it is easy to follow. 3. The studied setting is somewhat interesting, and it can be useful for understanding some effects of data distribution on the generalization error. 4. The paper includes experimental extension (results) for multilayer neural networks, which is beneficial, although the experimental results are also limited to the synthetic data setting considered in the paper. 1. The paper lacks highly relevant literature and contextualization relative to the prior work. Specifically, there are at least three papers (about generalization errors with spiked covariance) that are quite related to this work but not mentioned in the paper: - [1] studied "generalization for least squares regression with spiked covariances", which is also the fundamental topic of this work. Also, the settings are quite similar for [1] and this paper. - [2,3] analyzed two-layer neural networks with spiked covariance, which leads to spiked covariance for the features learned by the model (as discussed in the introduction of this work as well), so these works are also relevant. Specifically, [2] focused on the effect of spike magnitude and spike-target alignment, which is also the focus of this paper. Therefore, without proper contextualization relative to these prior works, this paper is weak in terms of positioning and novelty. 2. The fundamental theoretical challenge relative to the prior work should be discussed. Specifically, considering the closeness of this work to related setups in [1], Hastie et al. (2022), Sonthalia & Nadakuditi (2023), and [4], the difference in the proof/derivation techniques should be explicit. Since this is currently missing, the results in this work can be considered trivial generalizations/extensions of the other results. 3. The motivation for this specific data model (in line 45) is unclear. While being the main object of study and the source of most of the claimed results, the authors do not explicitly motivate the data model $\mathbf{y} = \alpha_Z \boldsymbol{\beta}^T \mathbf{z} + \alpha_A \boldsymbol{\beta}^T \mathbf{a} + \boldsymbol{\epsilon}$, other than saying it is a non-linear function of $\mathbf{x} = \m