Official Review · ICLR 2025

This paper quantifies and highlights the important issue of seed variance, amongst different sources of variance, in evaluating language model performance *during pretraining*, emphasizing how insufficient variance quantification in benchmarks can obscure statistically significant performance differences within the same pretraining run. **Methodology**: It retrains 10 LLaMA 7B models from scratch, and then studies the seed variance along with different metrics throughout the pretraining runs, such as-- 95% confidence intervals, and monotonicity across diverse benchmarks, offering a reference for understanding performance variation across setups. **Claim 1**: Continuous metrics exhibit higher monotonicity and lower variance than discrete metrics and should be used more widely. *Application*: Continuous metrics and cloze formats show lower variance and higher signal-to-noise ratios compared to traditional discrete measures, especially in smaller LLaMA 7B models. Simple modifications, such as reframing choice tasks as completion tasks, appear promising for reducing variance. **Claim 2**: Efficient benchmarking methods can inadvertently increase variance, and it’s essential to verify if methods are distinguishable before using them. *Application*: Techniques from standardized testing, such as item analysis and item response theory, are found ineffective in meaningfully reducing variance for large language models. Key takeaways I appreciated: - Continuous metrics show higher monotonicity and lower variance than discrete metrics, making them preferable. I really like this point! - Variance (likely from benchmarking, not seed variance) could obscure the effectiveness of pretraining by making it harder to track performance improvements. Benchmarking variance is a promising and underexplored area, especially given the recent focus on reasoning and the close performance of current models on benchmarks-- and seed variance can have a lot of effect. See Weaknesses in the order of importance. If there is lack of time, please prioritize the earlier questions: **P1: Seed Variance and Its Impact on Benchmark Scores** > We provide a comprehensive reference guide for what magnitudes of variance are expected for what benchmarks across various circumstances I worry the paper title/abstract/contribution1 is overclaiming as the evidence does not support the above claim-- It seems to me that paper focuses only on a single source of variance: seed variance. Given that studying variance is a common practice (the most popular benchmark, Chatbot Arena provides variance along with scores: https://lmsys.org/blog/2024-04-19-arena-hard/) and previous research on variance sources such as benchmarking details [1] and prompt design [2], the question studied here becomes: Does seed variance contribute overwhelmingly to the overall variance which makes it a critical factor to study, beyond [1,2,Arena]? I would appreciate: a) A comparative analysis showing the relativ