Inventing a coin-flip classifier: Accounting for multiplicity in machine learning benchmark performance

State-of-the-art (SOTA) performance refers to the highest performance achieved by some model on a test sample, preferably under controlled conditions such as public data (reproducibility) or public challenges (independent sample). Thousands of classifiers are applied, and the highest performance bec