Statistical Uncertainty Quantification for Aggregate Performance Metrics in Machine Learning Benchmarks

Modern artificial intelligence is supported by machine learning models (e.g., foundation models) that are pretrained on a massive data corpus and then adapted to solve a variety of downstream tasks. To summarize performance across multiple tasks, evaluation metrics are often aggregated into a summar