The Benchmarking Epistemology: Validity Theory for Evaluating Machine Learning Models

Predictive benchmarking, evaluating machine learning models based on predictive performance and competitive ranking, is central to machine learning research and scientific inquiry. However, benchmark scores at best measure performance relative to a specific dataset and learning problem. Drawing subs