SelectArena: A Modern Benchmark for Feature Selection on Real-world Tabular Data

S ELECTA RENA : A M ODERN B ENCHMARK FOR F EA TURE S ELECTION ON R EAL - WORLD TABULAR DATA

A BSTRACT Selecting a subset of features (columns) from a tabular dataset is a ubiquitous challenge in real-world applications. Industry experts seek to minimize model training costs, climate scientists require an interpretable subset of factors to understand extreme weather events, and clinicians need to choose among diagnostic tests due to financial and time constraints. When looking at the scientific literature to determine which feature selection method to use, practitioners across domains face an unrepresentative landscape of benchmarking. Despite extensive research, existing feature selection benchmarks provide limited guidance because they rely on outdated models, ad hoc evaluation protocols, or artificially generated data that do not reflect real-world problems. To close the gap between scientific benchmarking and real-world applications, we introduce SelectArena, a modern benchmark for feature selection. We compare 15 widely used feature selection methods on 71 manually curated real-world tabular datasets, covering classification and regression problems, using standardized evaluation protocols and state-of-the-art tabular (foundation) models. Our results reveal clear trade-offs across predictive performance, runtime, stability, and validity. Tree-based methods, CART and RFImportance, along with F-Test are Pareto-optimal when taking into account predictive performance and runtime. F-Test, information-theory-, and distance-based methods lead on stability and validity, with F-Test emerging as a balanced choice across our metrics. Searchbased methods are slow, unstable and achieve low performance. SelectArena turns these findings into practical guidance and, as an extensible platform, supports future feature selection research. valid 1.0 0.8 0.6 0.4 0.2 0.0 invalid

1300 1200 Elo

arXiv:2610.13644v1 [cs.LG] 9 Oct 2026

Bastian Schäfer1,∗ Dominika Matus2,∗ Lennart Purucker3,2 Ricardo Knauer4 Erik Rodner4 Frank Hutter3,5,2 Anton Sokolov1 Arnaud Liefooghe1 1 Université du Littoral-Côte-d’Opale 2 University of Freiburg 3 Prior Labs 4 5 ∗ HTW Berlin ELLIS Institute Tübingen Equal contribution. bastian.schafer@univ-littoral.fr, matusd@cs.uni-freiburg.de

1100 1000 900

ore om LOCO nSc Rand

cia apla

L

SBE

SFS nRatio mRMR ReliefF lanket Lasso F-Test (R) rkovB Gai Ma

MI icNet CART ance t t por Elas RFIm

stable 1.0 0.8 0.6 0.4 0.2 0.0 unstable

Figure 1: SelectArena at a glance. We present Elo ratings of the 15 feature selection methods ordered left to right from worst- to best-performing method. Elo rating summarizes how often a method beats another on the same dataset, budget, and downstream model, with random selection anchored at 1000 (Appendix D.2). Error bars are 95% confidence intervals from 1000 cluster bootstrap resamples over the 71 datasets. The left bar of each pair is shaded by stability (how much the selected subset changes across resamples) and the right bar by validity (how well injected noise features are rejected), where darker is better for both.

1

1

I NTRODUCTION

Tabular data dominates many medical, industrial, and scientific applications (Borisov et al., 2022; Erickson et al., 2025). However, collecting such data is often costly and requires labor-intensive measurements, expensive instruments, or access to specialized facilities (Hédou et al., 2024; Wang et al., 2025). At the same time, the growing dimensionality of datasets decreases interpretability, increases model training and inference costs, and forces practitioners to trade off performance against computation time. Feature selection addresses these challenges by identifying a subset of the most informative features, thereby reducing data collection and training costs while improving transparency and interpretability. Early work (Guyon & Elisseeff, 2003) has framed the objectives of feature selection around predictive performance, computational cost, and interpretability. Later work has added stability (Nogueira & Brown, 2016) and validity (Guyon et al., 2007a) as standard criteria for reliable feature selection. Within this problem formulation, hundreds of methods have been developed (Li et al., 2018), each making different trade-offs across these metrics. The benchmarking literature, however, remains highly fragmented and despite its long history (Jain & Zongker, 1997; Kohavi & John, 1997; Reunanen, 2003), does not provide sufficient guidance to practitioners. Existing benchmarks have not kept pace with the development of modern tabular models, in particular foundation models, and rely on unrealistic evaluation protocols. Most use small-scale comparisons on synthetic or limited real-world datasets with ≤ 10 methods and ≤ 20 real-world datasets (Bommert et al., 2020; Cherepanova et al., 2023; Bakhmach et al., 2026) (Section 2, Table 1). To address these gaps, we introduce SelectArena, a benchmark of 15 feature selection methods evaluated on 71 manually curated real-world tabular datasets across predictive performance, runtime, stability, and validity, using state-of-the-art tabular (foundation) models. Our contributions can be summarized as follows: • We analyze 368 datasets from the feature selection and machine learning literature and manually curate a set of 71 real-world tabular datasets following a strict data curation protocol. We release the resulting datasets. • We review 296 feature selection methods and select a set of 15 methods according to a strict method curation protocol. Where well-maintained codebases are unavailable, we provide our own implementations. • We provide a standardized evaluation pipeline covering predictive performance, stability, and validity, alongside runtime, and evaluate all methods with five downstream models including state-of-the-art tabular foundation model. We share our code to make our pipeline open-source. Our results reveal clear trade-offs across predictive performance, runtime, stability, and validity. Tree-based (RFImportance, CART) and regularization-based methods (ElasticNet, Lasso) achieve the highest predictive performance, while F-Test and GainRatio are the fastest and, alongside MI and LaplacianScore, lead on stability and validity. With all four criteria taken into account, F-Test emerges as the most balanced choice across them. Search-based methods are the counterexample: they are slow and rank near the bottom on performance and stability. However, no method wins on every criterion, so the right choice still depends on which objective a practitioner prioritizes. SelectArena turns these findings into practical guidance and, as an extensible open-source platform, supports future feature selection research.

2

R ELATED W ORK

Recent years have seen numerous feature selection benchmarks. Despite extensive research, practitioners still lack clear guidance on selecting the most suitable method for a given context. Existing benchmarks are typically limited in scope by examining only a small number of methods, most often no more than 10 (Bouchlaghem et al., 2022; Kohavi & John, 1997), or by focusing exclusively on specific subgroups (Bommert et al., 2020; Hopf & Reifenrath, 2021). Moreover, many studies rely on unrepresentative or limited data sources, such as synthetic data (Bolón-Canedo et al., 2013; Hopf & Reifenrath, 2021), domain-specific data (Bakhmach et al., 2026; Li et al., 2022; Umar et al., 2025), 2

or non-tabular data (text or image) (Guyon et al., 2007a; Kou et al., 2020; Miao & Niu, 2016). In most cases, the number of real-world tabular datasets does not exceed 20. Furthermore, most benchmarks focus only on predictive performance (Chandrashekar & Sahin, 2014; Reunanen, 2003; Sheikhpour et al., 2017). Few studies consider the stability of the selected subsets (Bommert et al., 2020; Brown et al., 2012; Kou et al., 2020; Saeys et al., 2008), even fewer the validity (Bolón-Canedo et al., 2013; Cherepanova et al., 2023; Guyon et al., 2007a) or both (Bakhmach et al., 2026; Hopf & Reifenrath, 2021). In addition, even recent works predominantly rely on classical downstream models, with little consideration of state-of-the-art approaches like tuned gradient-boosted trees or pretrained tabular foundation models (Bakhmach et al., 2026; Cherepanova et al., 2023; Umar et al., 2025). In contrast, SelectArena comprehensively benchmarks a diverse set of feature selection methods across a large and broad dataset collection, evaluates the predictive performance as well as the stability and validity, and incorporates state-of-the-art models. Table 1 provides a systematic comparison of existing feature selection benchmarks and SelectArena with respect to included methods, datasets, models, and evaluation metrics.

Table 1: Comparison of feature selection benchmarks. We present former benchmarks in comparison to SelectArena. # Methods: how many out of the total number of benchmark methods correspond to our first three curation criteria (not a variant of a method and applicable across tabular classification and regression tasks). # Datasets: how many out of the total number of benchmark datasets satisfy our curation criteria for high-quality tabular data (Section 3.2). Domain: visualizes the dataset domain: (synthetic), (environmental), (biomedical), (technological), (scientific), (financial) or (not applicable). SotA models: whether the benchmark includes state-of-the-art downstream models, such as tuned gradient-boosted trees or pretrained tabular foundation models (Erickson et al., 2025), with the options (yes) or (no). The final three columns (Perf, Stab, Val) indicate whether the benchmark reports an evaluation of the predictive performance, stability, or validity. Benchmark

# Methods

# Datasets

Kohavi & John (1997) Jain & Zongker (1997) Reunanen (2003) Guyon et al. (2007a) Saeys et al. (2008) Brown et al. (2012) Bolón-Canedo et al. (2013) Chandrashekar & Sahin (2014) Miao & Niu (2016) Sheikhpour et al. (2017) Li et al. (2018) Kou et al. (2020) Bommert et al. (2020) Hopf & Reifenrath (2021) Bouchlaghem et al. (2022) Li et al. (2022) Cherepanova et al. (2023) Schütz et al. (2025) Umar et al. (2025) Bakhmach et al. (2026) SelectArena (Ours)

3/4 8/15 2/2 1/3 4/4 7/9 7/11 3/4 8/8 3/9 18/45 1/10 12/22 12/58 5/5 7/8 8/9 8/10 1/1 14/22 15/15

5/14 0/1 2/7 0/5 6/6 4/15 0/11 2/6 1/12 0/3 7/25 0/10 1/16 0/13 0/7 0/15 1/12 1/11 0/3 0/1 71/71

3

Domain

SotA models

Perf

Stab

Val

3

S ELECTA RENA

3.1

P ROBLEM S ETUP

In practical applications, feature acquisition is often constrained by limited resources such as time, financial cost, or computational overhead. As a result, only a subset of all available features can be collected or processed, making it essential to select the most informative features under a given budget b. This setting can be formalized as follows: Let F = {1, . P . . , p} denote the full feature set and let S ⊆ F be a selected subset with associated cost c(S) = i∈S ci . We define a modeldependent subset utility function uM : 2F → R, which evaluates the utility of a feature subset S under a given model M . The budget-constrained feature selection problem can then be formulated as: maxS⊆F uM (S) subject to c(S) ≤ b. In certain cases, this problem can be solved to optimality, for example via exhaustive enumeration when the feature (sub)set size is small or via best subset selection in some models (Tillmann et al., 2024). In practice, however, exact computation is often infeasible (Amaldi & Kann, 1998), and approximate (heuristic) solution methods are therefore more commonly employed. As a special case, when feature costs are uniform (ci = 1) and the budget is fixed to the cardinality (b = k), the constraint reduces to |S| ≤ k, recovering the classical cardinalityconstrained feature selection problem (Knauer & Rodner, 2023). While the budget-constrained formulation is more general, it may introduce a degree of arbitrariness in practice, as small changes in the budget can significantly alter S without necessarily reflecting meaningful differences in resource availability. By contrast, cardinality constraints provide a simpler, more natural abstraction in many settings and are therefore mostly adopted in practice and used in SelectArena. 3.2

B ENCHMARKING S ETUP

Method Curation Protocol. We reviewed 33 prominent survey, benchmark, and related papers, covering seminal works and specialized contributions across the feature selection landscape, from foundational theory to domain-specific applications and recent benchmarking with full list of works in Table B.1. From this pool, we identified 296 unique feature selection methods (Table B.2) and selected a representative set of methods based on the following criteria: (1) suitability for tabular data, (2) suitability for all tasks (binary classification, multi-class classification and regression), (3) methodological independence (not merely a variant of another method), (4) number of citations in the relevant literature, and (5) subgroup coverage. The 15 feature selection methods that have been included in SelectArena reflect a mix of widely adopted and methodologically diverse approaches, with Table 2 and Figure 3 summarizing their key characteristics. For methods requiring model coupling, we use LGBM. More details on the curation criteria, curation insights, as well as an overview of all investigated methods, their source, the number of citations in the relevant literature, and the final decision on inclusion is available in Appendix B. Our method implementation builds on AutoGluon (Erickson et al., 2020) and TabArena (Erickson et al., 2025). We introduce a feature selection routine into TabArena that is integrated in the modelagnostic preprocessing pipeline of AutoGluon. The routine enforces a cardinality constraint k (Section 3.1), a time limit and a fallback to random selection in case of timeout (Appendix D.2) and custom preprocessing per method (Appendix B.4). To support time-bounded execution, the remaining runtime is periodically checked and intermediate results are returned when the limit is reached. Dataset Curation Protocol. Real-world tabular datasets are the foundation of SelectArena, so we build on the curation standards of TabArena (Erickson et al., 2025) and extend its dataset collection towards feature selection. In order to do so, we surveyed the feature selection survey and benchmark literature, the papers of our selected methods, and the related explainable AI literature, covering 73 articles in total (Appendix C.1), from which we identified 368 candidate datasets. We then applied the following curation criteria: (1) unique predictive task definition, (2) tabular data (e.g., no text or image data), (3) minimum of 10 features for meaningful selection, (4) real-world origin (no synthetic data), (5) IID data suitable for random splitting, (6) genuine predictive task, (7) no ethical concerns with the predictive task, and (8) accessible source data. Implementing these criteria results in 71 high-quality real-world datasets in a curation process presented in Figure 2. The datasets span many domains (Table 1), cover diverse prediction tasks and differ in size as well as feature dimensionality (Figure 3). More details on the curation criteria, curation insights, and on overview of the selected datasets with their key properties can be found in Appendix C. 4

Table 2: Benchmarked feature selection methods. We report all included methods in SelectArena. Determinism: whether a method yields the same subset for the same data across repeated evaluations (also referred to as consistency in Dembinsky et al. (2026)). The options are (yes), (no), or (model-dependent). Model coupling: whether feature selection can be performed in conjunction with the downstream model M (related to portability in Dembinsky et al. (2026)). Subgroup: we specify which subgroup a method belongs to (Appendix A). Citations: reflects a method’s popularity and impact within the relevant feature selection literature, measured as the frequency of occurrence across relevant survey and benchmark works (Section 3.2). For implementation details, please refer to Appendix B.4. Name

DeterModel minism coupling

CART (Breiman et al., 1984) ElasticNet (Zou & Hastie, 2005) F-test (Ståhle et al., 1989) GainRatio (Quinlan, 1986) LaplacianScore (He et al., 2005) Lasso (Tibshirani, 1996) LOCO (Duch, 2006) MarkovBlanket (Koller & Sahami, 1996) mRMR (Peng et al., 2005) Mutual Information (Battiti, 1994) Random (Baseline) (R)ReliefF (Kononenko et al., 1997) RFImportance (Breiman, 2001) SBE (Guyon et al., 2002) SFS (Cover & Van Campenhout, 1977)

- 28 No unique task

368

Subgroup

Cita- Implementitons tation

Tree Regularization Correlation Info theory Distance Regularization Leave-one-out Backward search Info theory Info theory / Distance Tree Backward search Forward search

4 3 6 3 5 8 3 3 12 6 / 10 8 8 10

scikit-learn scikit-learn scikit-learn Us Us scikit-learn Us Us Us scikit-learn Us Us scikit-learn Us Us

- 10 0 No tabular - 4 4 No real- world - 9 No predictive - 57 Not accessible data origin task - 35 Not at least - 21 Non- IID - 3 Ethical 10 features data concern

340

240

205

161

140

131

128

71

Figure 2: Dataset curation process. The reason and number of datasets excluded in each step of our dataset curation pipeline are shown, along with the remaining number of datasets after each step.

Downstream Models. To evaluate the selected subset of features on the subsequent prediction tasks, we use five downstream models: linear model (Linear), random forest (RF) (Breiman, 2001), LightGBM (LGBM) (Ke et al., 2017), RealMLP (Holzmüller et al., 2025) and TabICLv2 (Qu et al., 2026) spanning diverse inductive biases. We use AutoGluon’s default configuration in our experiments. For LGBM, we conduct an ablation study on the impact of hyperparameter tuning (Appendix D.2). Hardware usage. CPU experiments (stability, validity and performance with the Linear, RF and LGBM model) ran on an AMD EPYC 9655 with 96 cores and GPU experiments (performance with RealMLP and TabICLv2) primarily on an NVIDIA L40 with 48 GB VRAM and in rare cases on NVIDIA H200 141 GB VRAM. Our performance experiments span 15 methods, 71 datasets, five budgets, five downstream models, and three seeds, yielding 79,875 runs, totaling approximately 292 GPU days and 685 CPU days. Validity with 3 noise levels and stability results with 25 bootstrapped repetitions and respectively 15 methods, 71 datasets and 5 repetitions took additional 2243 CPU days. 5

10

2

s 00 20

90 19

80 19

s

0

res sio n

Re g

Mu

ltic

Bin

Model Coupling Number of Methods

4

las s

0

10

Number of Features

20

Method Publication Decade

s

Number of Methods

6

Classification Regression

104

20 103

Number of Classes

30

ary

Number of Datasets

Datasets by Problem Type 40

19 80 19 s 90 20 s 00 20 s 10 20 s 20 s

Number of Datasets

Dataset Publication Decade 18 16 14 12 10 8 6 4 2 0

15 10

102

5

8 6

101

4 2 0

Yes

102

104 105 103 Number of Samples

No

Figure 3: Method and dataset characteristics. SelectArena represents a diverse collection of widely adopted feature selection methods as well as high-quality real-world datasets. 3.3

E VALUATION

We assess the quality of a feature selection method in four dimensions: predictive performance, stability, validity, and runtime. For cardinality-constrained feature selection, we need to consider two hyperparameters that are independent of the specific method: the cardinality constraint k and the time constraint. To obtain a diverse performance estimate across different application scenarios, we vary k as a function of the number of features p. For datasets with ≤ 100 features, we vary k = ⌈p × j/6⌉, where p is the number of features and j ∈ {1, . . . , 5} is an index variable. For bigger datasets, we use an exponential regime: k = ⌈pj/6 ⌉. For more details, see Appendix C.5. If methods do not respect the cardinality constraint, we use random fallback (see Appendix D.2). With respect to the time constraint, we use a limit of one hour per feature selection experiment consisting of a (dataset, budget, model)-tuple for all feature selection methods. The downstream model training and inference time is set to one hour per experiment as well. Predictive Performance. The predictive performance of a feature selection method is the performance of the downstream model using the selected feature subset on a given task. It is measured by ROC AUC for binary classification problems, log-loss for multi-class classification, and RMSE for regression. We discuss the choice of evaluation metrics in Appendix D.2. To compare across these metrics, we report Elo
ratings (Appendix D.2) and improvement over R ANDOM, I(m, d) = 100 · Err(r, d) − Err(m, d) /Err(r, d), where Err(m, d) is the metric error of method m on dataset d and r denotes random selection (Appendix D.2). We repeat all experiments for three seeds. Stability. For interpretability, feature selection methods should not only achieve strong predictive performance, but also yield similar subsets for similar data, for instance via resampling of the original data. We refer to this desideratum as stability, which is sometimes also described as robustness, reliability, or continuity (Dembinsky et al., 2026). We assess the stability with the Nogueira metric (Nogueira & Brown, 2016) across 25 bootstrap samples. The Nogueira estimator corrects for chance, asymptotically assigns random selections a score of 0, and reaches a maximum of 1. Values below 0.40 indicate poor stability, values between 0.40 and 0.75 reflect intermediate to good stability, and values above 0.75 denote excellent stability (Nogueira & Brown, 2016). Validity. In addition to being stable, feature selection methods should also recover relevant rather than irrelevant features, an interpretability desideratum that we refer to as validity. In the literature, this property is also described as truthfulness, faithfulness, or fidelity (Dembinsky et al., 2026). However, assessing validity is challenging in the absence of a synthetic ground truth (Bakhmach et al., 2026; Bolón-Canedo et al., 2013; Hopf & Reifenrath, 2021). We follow prior work (Cherepanova et al., 2023; Guyon et al., 2003; 2004; 2007a) and evaluate validity by augmenting the original feature set with uninformative Gaussian noise variables at 50%, 75%, and 100% of the original feature count. 6

We then operationalize validity by quantifying the proportion of selected features from the original feature set, which we term selection precision. This measure ranges from 0 to 1. In our setting, the selection precision is equivalent to the selection recall, i.e., the proportion of original, informative features selected, when the recall is normalized by its maximum possible value for k. Runtime. Lastly, feature selection methods should produce well-performing, stable, and valid subsets of features as fast as possible. Therefore, we measure the wall-clock time a method needs to produce a feature subset for a given budget k with the upper bound of one hour. Where the implementation allows it, exceeding the limit interrupts the method and triggers a random fallback that restores the subset to size k, either by adding randomly chosen unselected features or, for backward methods, by randomly subsampling the current set. To make runtimes comparable across datasets of different dimensionality, we normalize by the number of features and report seconds per 1,000 features.

4

R ESULTS

We evaluated 15 widely used feature selection methods on 71 manually curated real-world tabular datasets with five cardinality constraints per dataset and five downstream predictors across three seeds in our performance experiments. The results are shown in Figures 1 and 4 and discussed further in the next paragraphs. For stability and validity runs, we use the same 15 methods and 71 datasets respectively and 5 repetitions. Each stability run consists of 25 bootstrapped repetition, each validity run includes 3 noise levels. The results are shown in Figures 5 and 6, respectively, and discussed in the last paragraphs.

CART F-Test

GainRatio

Impr. over Random (%)

RFImportance MI ElasticNet mRMR Lasso (R)ReliefF MarkovBlanket SFS

100

101

102

103

104

10

0

TabICLv2 29.0 25.5 25.9 24.2 23.6 23.5 21.7 20.3 20.8 21.7 19.3 14.6 13.2 -19.6 RF 27.7 25.8 24.8 23.8 21.1 23.6 22.1 21.2 20.6 20.3 17.5 12.7 13.0 -17.0 RealMLP 27.0 24.2 24.9 23.1 21.6 21.8 20.9 19.2 20.3 20.6 18.7 14.8 12.5 -18.4 LGBM 25.5 22.9 23.5 21.5 21.7 21.2 21.6 21.1 19.8 18.5 18.1 13.0 12.3 -23.2 Linear 24.6 23.2 22.0 22.9 21.8 19.0 22.6 21.7 18.4 16.3 17.7 12.0 8.4 -14.4

rta

nc

Dominated Methods LOCO SBE Pareto Front

Selection Time (s) per 1k Features

20

e MI CA R F-T T Ma est rko mR vB MR Ela lanke sti t cN et L Ga ass in o (R) Rati Re o lie fF SF LO S La pla CO cia SB nS E co re

optimal

Im po

28 26 24 22 20 18 16 14 12 10

RF

Impr. over Random (%)

Strong Methods Can Be Fast. The left plot in Figure 4 shows improvement over Random (%) versus selection time (seconds per 1k features), with predictive performance and runtime on the axes; exact scores are in Table D.2. The Pareto front consists of RFImportance and CART, the tree-based methods, together with the correlation-based F-Test. All three rank in the top four on predictive performance alone (MI ranks second) and in the six fastest on runtime alone (with LaplacianScore, GainRatio, and MarkovBlanket), so practitioners who value both face little trade-off. The searchbased methods (LOCO, SBE, and SFS) sit at the opposite corner, performing poorly while being the slowest, although this, at least in part, can be attributed to the imposed time limit.

Figure 4: Predictive performance. Both panels report improvement over R ANDOM (%, higher is better), averaged over three seeds, five cardinalities, and 71 datasets. (Left): Improvement averaged over downstream models against feature selection time (s) per 1,000 features with Pareto front marked in orange. We exclude LaplacianScore as it performs worse than Random. (Right): Improvement for every model–method interaction. Good Feature Selection Helps Every Model. The heatmap in Figure 4 presents improvement over Random (%) for each combination of feature selection method and downstream model averaged over seeds, budgets, and datasets. We observe that model ordering is rather consistent across methods, with TabICLv2 ranking first for eight of the 14 methods and Linear last for seven. The improvement spread across models within a method (at most 5.4pp) is far smaller than the spread across methods within a model (15.8pp under TabICLv2). This asymmetry suggests that even the most expressive model, like TabICLv2, cannot recover predictive signal from a poorly selected subset and so the choice of feature selection method is crucial. We find no evidence that aligning the inductive biases 7

F-Test LaplacianScore MI GainRatio (R)ReliefF mRMR RFImportance ElasticNet MarkovBlanket CART Lasso SFS SBE LOCO Random 2799.7

stable 1.0 0.8 0.6 0.4 0.2 0.0 unstable 217.4

15.1

3.4 0.2 Events per Variable

0.0

Figure 5: Stability. Nogueira score (Nogueira & Brown, 2016) across datasets. We use the events per variable, i.e., the number of samples per feature for regression problems or the number of samples in the minority class per feature for classification problems, as a proxy for the selection difficulty (Harrell, 2015; Heinze et al., 2018). of selector and predictor helps. Although Lasso and ElasticNet peak with the Linear model (but also only by 0.5pp), the tree-based RFImportance and CART achieve best performance with TabICLv2 rather than with RF or LGBM. Strong Methods are not Necessarily Stable. The stability of the feature selection methods is shown in Figure 5. Most methods are stable when the number of samples (in the minority class) per feature is sufficiently large, but become increasingly unstable in low-data regimes (Harrell, 2015; Heinze et al., 2018). F-Test and distance-based methods exhibit the highest stability contributing to a better interpretability of the selected features, followed by information-theoretic methods. F-Test and MI are the only high-performing methods that rank high in stability. Due to its high fallback rates, search-methods are only slightly more stable than a random selection. An extended stability analysis can be found in Appendix D.3.

F-Test GainRatio MI LaplacianScore ElasticNet Lasso (R)ReliefF SFS RFImportance mRMR LOCO SBE CART Random MarkovBlanket 2799.7

valid 1.0 0.8 0.6 0.4 0.2 0.0 invalid 40.0

6.3 0.7 Events per Variable

0.1

0.0

Figure 6: Validity. Selection precision across datasets at a noise level of 50%. Other noise levels can be found in Appendix D. Like for the stability, we use the events per variable, i.e., the number of samples per feature for regression problems or the number of samples in the minority class per feature for classification problems, as a proxy for the selection difficulty (Harrell, 2015; Heinze et al., 2018). Strong Methods do not Necessarily Select Informative Features. Figure 6 shows the validity of the feature selection methods. In contrast to the stability, the validity is less sensitive to the number 8

of samples (in the minority class) per feature. F-Test and information-theoretic methods achieve the highest validity aiding interpretability of the selection, followed by distance-based methods. F-Test and MI are the only high-performing methods that rank high in validity. MarkovBlanket exhibits lower validity than even a random selection, potentially because it favors removing redundant features that are prevalent in real-world data (Guyon & Elisseeff, 2003; Koller & Sahami, 1996). An extended validity analysis at different noise levels can be found in Appendix D.4. F-Test Is Strong on All Four Axes. Taken together, no single method dominates on all four evaluation axes, but among the strong candidates F-Test comes closest to being a well-balanced choice. It lies on the Pareto front of predictive performance and runtime, and ranks highest on both stability and validity (Figure 1). RFImportance and CART achieve higher Elo ratings but are less stable, and the model-coupled search methods are competitive on none of the four dimensions under our one-hour budget.

5

C ONCLUSIONS

Feature selection is essential across domains where practitioners must balance predictive performance, computational constraints, and interpretability. However, existing benchmarks offer limited guidance to select methods that best suit different real-world requirements. To address this gap, we introduce SelectArena, a large-scale benchmark for feature selection on 71 manually curated real-world tabular datasets, spanning classification and regression tasks. We evaluate 15 widely used feature selection methods using standardized protocols and five downstream models, including a tabular foundation model. SelectArena is not only the largest general feature selection benchmark to date, but also the only one to include leading tabular models. Limitations. While SelectArena substantially advances the state of feature selection benchmarking, several limitations remain. (1) All feature selection methods are evaluated under a one-hour time limit, which results in slower methods not always completing their task (up to 47% of runs for SBE). Since we handle timeouts with a random fallback that keeps the feature budget fixed, the results for the affected methods reflect their performance under the limit on our hardware and are subject to change if given more compute. Appendix D.2 provides an ablation without the random fallback. (2) Several methods are implemented by us in cases where no sufficiently maintained alternative was available, rather than taken from an established library (Table 2). Although we follow the original algorithms, our implementations may be less optimized than library equivalents, which can disadvantage them under a wall-clock limit. (3) Our analysis of stability and validity relies on bootstrapping and Gaussian noise, respectively. Both are arguably simpler than what methods face in practice, since bootstrapping resamples preserve the original feature correlations and Gaussian noise variables are independent of both the target and the informative features. Other forms of perturbation may affect feature selection behavior differently, e.g., non-Gaussian noise, outliers, missing values, distribution shifts, or class imbalances. (4) All methods are constrained to return exactly k features, however, some of them natively determine the subset size intrinsically, such as Lasso and ElasticNet via the regularization path or MarkovBlanket via conditional independence tests. Imposing a fixed cardinality makes them comparable but does not evaluate them as originally intended, and may understate their performance. (5) The observed performance of feature selection methods strongly depends on the evaluation metrics used. Using the ROC AUC for binary classification, log-loss for multiclass classification, and RMSE for regression (Appendix D.2), we employ widely adopted standard metrics, but alternative criteria such as calibration or fairness may lead to different results. Recommendations. Our results lead to several practical recommendations: (1) When predictive performance is the primary objective, practitioners might want to choose a tree-based method such as RFImportance or CART. These methods are also strong candidates when runtime is considered. (2) For interpretability-focused applications, F-Test emerges as the best choice. It ranks highest in terms of stability and validity, next to information-theory and distance-based methods, with a competitive predictive performance and runtime. (3) The choice of a suitable downstream model for a feature selection method can be important. A state-of-the-art model like TabICLv2 is mostly, but not always, a better predictor than RF, LGBM or even a linear model. Therefore, practitioners should evaluate whether simpler downstream models may be preferable to more complex and computationally expensive alternatives.

9

R EPRODUCIBILITY STATEMENT SelectArena is an open-source framework. We publish the code and our curated datasets. More information can be found in Appendix D.1. In the Benchmarking Setup (Chapter 3.2), we detail the hardware setup used for the experiments. With fully providing soft- and hardware details as well as the datasets used, SelectArena is fully reproducible.

ACKNOWLEDGMENT S TATEMENT We thank the dataset donors for sharing their data, especially those under a public license. Furthermore, we thank the community, particularly OpenML, Kaggle, and Hugging Face, for developing tools that facilitate the sharing of tabular data. We thank the author of the scikit feature library for the inspiration through the provided implementations of feature selection methods. B. Schäfer, A Liefooghe and A. Sokolov have benefited from the support of the National Research Agency under France 2030, MAIA Project ANR-22-EXES-0009, and from the "Pôle Métropolitain Côte d’Opale" (PMCO). D. Matus, L. Purucker, and F. Hutter acknowledge funding by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under SFB 1597 (SmallData), grant number 499552394 R. Knauer and E. Rodner acknowledge being supported by the European Union’s Erasmus+ program through EUonAIR, reference 101177370, and the Deutsche Forschungsgemeinschaft (DFG) through Berlin Initiative for Applied Foundation Model Research, reference 528483508.

AUTHOR C ONTRIBUTIONS B. Schäfer co-led the code development, led the method curation, co-led the dataset curation, contributed the method implementation, co-led the experiments, co-led the evaluation and the visualization and co-led the writing of the paper. D. Matus co-led the code development, contributed to the method curation, co-led the dataset curation, led the method implementation, co-led the experiments, co-led the evaluation and the visualization and co-led the writing of the paper. L. Purucker co-led the code development, supervised the method curation, supervised the dataset curation, supervised the experiments and supervised the writing of the paper. R. Knauer supervised the project and contributed to writing and reviewing the paper. E. Rodner supervised R. Knauer and helped with reviewing and editing the paper. F. Hutter supervised L. Purucker and D. Matus. A. Sokolov supervised B. Schäfer. A. Liefooghe supervised B. Schäfer and helped with reviewing and editing the paper.

C OMPETING I NTERESTS L. Purucker and F. Hutter are a subset of the authors of TabArena. The authors declare no other competing interests.

10

R EFERENCES Kevork N Abazajian, Jennifer K Adelman-McCarthy, Marcel A Agüeros, Sahar S Allam, Carlos Allende Prieto, Deokkeun An, Kurt SJ Anderson, Scott F Anderson, James Annis, Neta A Bahcall, et al. The seventh data release of the sloan digital sky survey. The Astrophysical Journal Supplement Series, 182(2):543–558, 2009. Chirag Agarwal, Satyapriya Krishna, Eshika Saxena, Martin Pawelczyk, Nari Johnson, Isha Puri, Marinka Zitnik, and Himabindu Lakkaraju. Openxai: Towards a transparent evaluation of model explanations. Advances in neural information processing systems, 35:15784–15799, 2022. Uri Alon, Naama Barkai, Daniel A. Notterman, Kurt Gish, Suzanne Ybarra, Daniel Mack, and Arnold J. Levine. Broad patterns of gene expression revealed by clustering analysis of tumor and normal colon tissues probed by oligonucleotide arrays. Proceedings of the National Academy of Sciences, 96(12):6745–6750, 1999. doi: 10.1073/pnas.96.12.6745. URL https://doi.org/ 10.1073/pnas.96.12.6745. Edoardo Amaldi and Viggo Kann. On the approximability of minimizing nonzero variables or unsatisfied relations in linear systems. Theoretical Computer Science, 209(1):237–260, 1998. doi: 10.1016/S0304-3975(97)00115-1. Arashnic. Hr analytics: Job change of data scientists, 2021. URL https://www.kaggle.com/ datasets/arashnic/hr-analytics-job-change-of-data-scientists. Anastasiia Bakhmach, Paul Dufossé, Andrea Vaglio, Florence Monville, Laurent Greillier, Fabrice Barlési, and Sébastien Benzekry. Roofs: Robust biomarker feature selection. arXiv preprint arXiv:2601.05151, 2026. E Ray Bareiss, Bruce W Porter, and Craig C Wier. Protos: An exemplar-based learning apprentice. In Machine learning, pp. 112–127. Elsevier, 1990. Roberto Battiti. Using mutual information for selecting features in supervised neural net learning. IEEE Transactions on neural networks, 5(4):537–550, 1994. Mohamed Karim Belaid, Richard Bornemann, Maximilian Rabus, Ralf Krestel, and Eyke Hüllermeier. Compare-xai: Toward unifying functional testing methods for post-hoc xai algorithms into a multidimensional benchmark. In World Conference on Explainable Artificial Intelligence, pp. 88–109. Springer, 2023. Mohamed Bennasar, Yulia Hicks, and Rossitza Setchi. Feature selection using joint mutual information maximisation. Expert systems with applications, 42(22):8520–8532, 2015. Ashish Bhardwaj. Framingham heart study dataset, 2022. URL https://www.kaggle.com/ dsv/3493583. Arindam Bhattacharjee, William G. Richards, Jane Staunton, Cheng Li, Stefano Monti, Priya Vasa, Christine Ladd, Javad Beheshti, Raphael Bueno, Michael Gillette, et al. Classification of human lung carcinomas by mrna expression profiling reveals distinct adenocarcinoma subclasses. Proceedings of the National Academy of Sciences, 98(24):13790–13795, 2001. doi: 10.1073/pnas. 191502998. URL https://doi.org/10.1073/pnas.191502998. Verónica Bolón-Canedo, Noelia Sánchez-Maroño, and Amparo Alonso-Betanzos. A review of feature selection methods on synthetic data. Knowledge and information systems, 34(3):483–519, 2013. Andrea Bommert, Xudong Sun, Bernd Bischl, Jörg Rahnenführer, and Michel Lang. Benchmark for filter methods for feature selection in high-dimensional classification data. Computational Statistics & Data Analysis, 143:106839, 2020. V. Borisov, T. Leemann, K. Seßler, J. Haug, M. Pawelczyk, and G. Kasneci. Deep neural networks and tabular data: A survey. IEEE Transactions on Neural Networks and Learning Systems, 2022. Benjamin Bossan, Josef Feigl, and Wendy Kan. Otto group product classification challenge. https://kaggle.com/competitions/ otto-group-product-classification-challenge, 2015. Kaggle competition. 11

Younes Bouchlaghem, Yassine Akhiat, and Souad Amjad. Feature selection: a review and comparative study. In E3S web of conferences, volume 351, pp. 01046. EDP Sciences, 2022. Steven C Bourassa, Martin Hoesli, Louis Merlin, and John Renne. Big data, accessibility and urban house prices. Urban Studies, 58(15):3176–3195, 2021. Leo Breiman. Random forests. Machine learning, 45(1):5–32, 2001. Leo Breiman, Jerome Friedman, Richard A Olshen, and Charles J Stone. Classification and regression trees. Chapman and Hall/CRC, 1984. Gavin Brown, Adam Pocock, Ming-Jie Zhao, and Mikel Luján. Conditional likelihood maximisation: a unifying framework for information theoretic feature selection. The journal of machine learning research, 13(1):27–66, 2012. Deng Cai, Chiyuan Zhang, and Xiaofei He. Unsupervised feature selection for multi-cluster data. In Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 333–342, 2010. Girish Chandrashekar and Ferat Sahin. A survey on feature selection methods. Computers & electrical engineering, 40(1):16–28, 2014. Tianqi Chen. Xgboost: A scalable tree boosting system. Cornell University, 2016. Valeriia Cherepanova, Roman Levin, Gowthami Somepalli, Jonas Geiping, C Bayan Bruss, Andrew G Wilson, Tom Goldstein, and Micah Goldblum. A performance-driven benchmark for feature selection in tabular deep learning. Advances in Neural Information Processing Systems, 36: 41956–41979, 2023. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios N. Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Hao Zhang, Michael I. Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: an open platform for evaluating llms by human preference. In Proceedings of the 41st International Conference on Machine Learning (ICML), pp. 8359–8388, Vienna, Austria, 2024. PMLR. Paulo Cortez and Aníbal de Jesus Raimundo Morais. A data mining approach to predict forest fires using meteorological data. 2007. Paulo Cortez, Antonio Cerdeira, Fernando Almeida, Telmo Matos, and Jose Reis. Modeling wine preferences by data mining from physicochemical properties. Decision Support Systems, 47 (4):547–553, 2009. doi: 10.1016/j.dss.2009.05.016. URL https://doi.org/10.1016/j. dss.2009.05.016. Thomas M Cover and Jan M Van Campenhout. On the possible orderings in the measurement selection problem. IEEE transactions on systems, man, and cybernetics, 7(9):657–661, 1977. Ian Covert, Scott M Lundberg, and Su-In Lee. Understanding global feature contributions with additive importance measures. Advances in neural information processing systems, 33:17212– 17223, 2020. Ian Covert, Scott Lundberg, and Su-In Lee. Explaining by removing: A unified framework for model explanation. Journal of Machine Learning Research, 22(209):1–90, 2021. Darrel, Stephen D. Stayton, and Will Cukierski. Homesite quote conversion. https://kaggle. com/competitions/homesite-quote-conversion, 2015. Kaggle competition. Manoranjan Dash and Huan Liu. Consistency-based search in feature selection. Artificial intelligence, 151(1-2):155–176, 2003. Fernando Lucas de Oliveira Farias and Raphael Fontes. Ensino 5a > datacamp. https://kaggle. com/competitions/ensino-5a-datacamp, 2020. Kaggle. 12

David Dembinsky, Adriano Lucieri, Stanislav Frolov, Hiba Najjar, Ko Watanabe, and Andreas Dengel. Unifying VXAI: A systematic review and framework for the evaluation of explainable AI. Transactions on Machine Learning Research, 2026. ISSN 2835-8856. URL https:// openreview.net/forum?id=wAvFLe7o0E. Survey Certification. Robert Detrano, Andras Janosi, Walter Steinbrunn, Matthias Pfisterer, Johann-Jakob Schmid, Sarbjit Sandhu, Kern H Guppy, Stella Lee, and Victor Froelicher. International application of a new probability algorithm for the diagnosis of coronary artery disease. The American journal of cardiology, 64(5):304–310, 1989. Włodzisław Duch. Filter methods. In Feature extraction: foundations and applications, pp. 89–117. Springer, 2006. Bradley Efron and Gail Gong. Statistical theory and the computer. In Computer science and statistics: Proceedings of the 13th Symposium on the Interface, pp. 3–7. Springer, 1981. Agoston E Eiben and James E Smith. Introduction to evolutionary computing. Springer, 2 edition, 2015. N. Erickson, J. Mueller, A. Shirkov, H. Zhang, P. Larroy, M. Li, and A. Smola. Autogluon-tabular: Robust and accurate automl for structured data. arXiv:2003.06505 [stat.ML], 2020. Nick Erickson, Lennart Purucker, Andrej Tschalzev, David Holzmüller, Prateek Mutalik Desai, David Salinas, and Frank Hutter. Tabarena: A living benchmark for machine learning on tabular data. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025. URL https://openreview.net/forum?id=jZqCqpCLdU. Dana Ferguson, Meg Risdal, NoTrick, Sara R. Sillah, Tim Emmerling, and Will Cukierski. Allstate claims severity. https://kaggle.com/competitions/ allstate-claims-severity, 2016. Kaggle competition. Daniel Fryer, Inga Strümke, and Hien Nguyen. Shapley values for feature selection: The good, the bad, and the axioms. Ieee Access, 9:144352–144360, 2021. Credit Fusion and Will Cukierski. Give me some https://kaggle.com/competitions/GiveMeSomeCredit, 2011. Kaggle.

credit.

url-

Lingyun Gao, Mingquan Ye, Xiaojie Lu, and Daobin Huang. Hybrid method based on information gain and support vector machine for gene selection in cancer classification. Genomics, proteomics & bioinformatics, 15(6):389–395, 2017. Pieter Gijsbers, Marcos L. P. Bueno, Stefan Coors, Erin LeDell, Sébastien Poirier, Janek Thomas, Bernd Bischl, and Joaquin Vanschoren. Amlb: an automl benchmark, 2023. URL https: //arxiv.org/abs/2207.12560. Sergey E Golovenkin, Jonathan Bac, Alexander Chervov, Evgeny M Mirkes, Yuliya V Orlova, Emmanuel Barillot, Alexander N Gorban, and Andrei Zinovyev. Trajectories, bifurcations, and pseudo-time in large clinical datasets: applications to myocardial infarction and diabetes data. GigaScience, 9(11):giaa128, 2020. Todd R. Golub, Donna K. Slonim, Pablo Tamayo, Christine Huard, Michelle Gaasenbeek, Jill P. Mesirov, Hilary Coller, Mignon L. Loh, James R. Downing, Michael A. Caligiuri, Clara D. Bloomfield, and Eric S. Lander. Molecular classification of cancer: class discovery and class prediction by gene expression monitoring. Science, 286(5439):531–537, 1999. doi: 10.1126/ science.286.5439.531. URL https://doi.org/10.1126/science.286.5439.531. Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Advances in neural information processing systems, 34:18932–18943, 2021. Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? Advances in neural information processing systems, 35: 507–520, 2022. 13

Sadia Gul, Fatima Rahim, Selen Isin, et al. Structure-based design and classifications of small molecules regulating the circadian rhythm period. Scientific Reports, 11:18510, 2021. doi: 10. 1038/s41598-021-97962-5. URL https://doi.org/10.1038/s41598-021-97962-5. Isabelle Guyon and André Elisseeff. An introduction to variable and feature selection. Journal of machine learning research, 3(Mar):1157–1182, 2003. Isabelle Guyon, Jason Weston, Stephen Barnhill, and Vladimir Vapnik. Gene selection for cancer classification using support vector machines. Machine learning, 46(1):389–422, 2002. Isabelle Guyon, Steve Gunn, Asa Ben-Hur, and Gideon Dror. Result analysis of the nips 2003 feature selection challenge. Advances in neural information processing systems, 17, 2004. Isabelle Guyon, Jiwen Li, Theodor Mader, Patrick A Pletscher, Georg Schneider, and Markus Uhr. Competitive baseline methods set new standards for the nips 2003 feature selection benchmark. Pattern recognition letters, 28(12):1438–1444, 2007a. Isabelle Guyon, Amir Saffari, Gideon Dror, and Gavin Cawley. Agnostic learning vs. prior knowledge challenge. In 2007 International Joint Conference on Neural Networks, pp. 829–834, 2007b. doi: 10.1109/IJCNN.2007.4371065. Isabelle Guyon, Constantin Aliferis, Greg Cooper, André Elisseeff, Jean-Philippe Pellet, Peter Spirtes, and Alexander Statnikov. Analysis of the kdd cup 2009: Fast scoring on a large orange customer database. In KDD-Cup 2009 Competition. PMLR, 2009. URL https://proceedings.mlr. press/v7/guyon09.html. Isabelle Guyon et al. Design of experiments of the nips 2003 variable selection benchmark. In NIPS 2003 workshop on feature extraction and feature selection, volume 253, pp. 40, 2003. Mark A Hall. Correlation-based feature selection for machine learning. PhD thesis, The University of Waikato, 1999. Kam Hamidieh. A data-driven statistical model for predicting the critical temperature of a superconductor. Computational Materials Science, 154:346–354, 2018. Ben Hamner, dcthompson, and Jorg. Predicting https://kaggle.com/competitions/bioresponse, 2012. Kaggle.

a

biological

response.

Frank E Harrell. Regression modeling strategies: with applications to linear models, logistic regression, and survival analysis. Springer, 2 edition, 2015. Xiaofei He, Deng Cai, and Partha Niyogi. Laplacian score for feature selection. Advances in neural information processing systems, 18, 2005. Julien Hédou, Ivana Marić, Grégoire Bellan, Jakob Einhaus, Dyani K Gaudillière, Francois-Xavier Ladant, Franck Verdonk, Ina A Stelzer, Dorien Feyaerts, Amy S Tsai, et al. Discovery of sparse, reliable omic biomarkers with stabl. Nature Biotechnology, 42(10):1581–1593, 2024. Georg Heinze, Christine Wallisch, and Daniela Dunkler. Variable selection–a review and recommendations for the practicing statistician. Biometrical journal, 60(3):431–449, 2018. S. Herbold. Autorank: A python package for automated ranking of classifiers. Journal of Open Source Software, 5(48):2173–2173, 2020. Sunil R. Hingorani, Emanuel F. Petricoin, Anirban Maitra, Vinodh Rajapakse, Cynthia King, Michael A. Jacobetz, Sarah Ross, Thomas P. Conrads, Timothy D. Veenstra, Ben A. Hitt, Yoshiya Kawaguchi, David Johann, Lance A. Liotta, Howard C. Crawford, Mary E. Putt, Tyler Jacks, Christopher V. Wright, Ralph H. Hruban, Andrew M. Lowy, and David A. Tuveson. Preinvasive and invasive ductal pancreatic cancer and its early detection in the mouse. Cancer Cell, 4(6): 437–450, 2003. doi: 10.1016/s1535-6108(03)00309-x. URL https://doi.org/10.1016/ s1535-6108(03)00309-x. Robert C Holte. Very simple classification rules perform well on most commonly used datasets. Machine learning, 11(1):63–90, 1993. 14

David Holzmüller, Léo Grinsztajn, and Ingo Steinwart. Better by default: Strong pre-tuned mlps and boosted trees on tabular data, 2025. URL https://arxiv.org/abs/2407.04491. Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretability methods in deep neural networks. Advances in neural information processing systems, 32, 2019. Konstantin Hopf and Sascha Reifenrath. Filter methods for feature selection in supervised machine learning applications–review and benchmark. arXiv preprint arXiv:2111.12140, 2021. Addison Howard, Adriano Moala, and Walter Reade. Porto seguro’s safe driver prediction. https: //kaggle.com/competitions/porto-seguro-safe-driver-prediction, 2017. Kaggle competition. Lina Huang, Peineng Liu, and Xiaojie Huang. Interdia: Interpretable prediction of drug-induced autoimmunity through ensemble machine learning approaches. Toxicology, 511:154064, 2025. IDA2016Challenge. Ida2016challenge [dataset]. urlhttps://doi.org/10.24432/C5V60Q, 2016. UCI Machine Learning Repository. inversion, Jin Paik, Maggie, mrbhbs, Steven Randazzo, and tnat1031. Mechanisms of action (moa) prediction. urlhttps://kaggle.com/competitions/lish-moa, 2020. Kaggle. Anil Jain and Douglas Zongker. Feature selection: Evaluation, application, and small sample performance. IEEE transactions on pattern analysis and machine intelligence, 19(2):153–158, 1997. Daniel P Jeong, Zachary Chase Lipton, and Pradeep Kumar Ravikumar. LLM-select: Feature selection with large language models. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL https://openreview.net/forum?id=16f7ea1N3p. Alan Jović, Karla Brkić, and Nikola Bogunović. A review of feature selection methods with applications. In 2015 38th international convention on information and communication technology, electronics and microelectronics (MIPRO), pp. 1200–1205. Ieee, 2015. Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/ 2017/file/6449f44a102fde848669bdd9eb6b76fa-Paper.pdf. Doaa Sami Khafaga, El-Sayed M El-kenawy, Fadwa Alrowais, Sunil Kumar, Abdelhameed Ibrahim, and Abdelaziz A Abdelhamid. Novel optimized feature selection using metaheuristics applied to physical benchmark datasets. Computers, Materials & Continua, 74(2), 2023. Samina Khalid, Tehmina Khalil, and Shamila Nasreen. A survey of feature selection and feature extraction techniques in machine learning. In 2014 science and information conference, pp. 372–378. IEEE, 2014. Javed Khan, Jun S. Wei, Markus Ringnér, Lao H. Saal, Marc Ladanyi, Frank Westermann, Frank Berthold, Manfred Schwab, Cristina R. Antonescu, Carsten Peterson, and Paul S. Meltzer. Classification and diagnostic prediction of cancers using gene expression profiling and artificial neural networks. Nature Medicine, 7(6):673–679, 2001. doi: 10.1038/89044. URL https://doi.org/10.1038/89044. Kenji Kira and Larry A Rendell. A practical approach to feature selection. In Machine learning proceedings 1992, pp. 249–256. Elsevier, 1992. Ricardo Knauer and Erik Rodner. Cost-sensitive best subset selection for logistic regression: A mixed-integer conic optimization perspective. In German Conference on Artificial Intelligence (Künstliche Intelligenz), pp. 114–129. Springer, 2023. 15

Ricardo Knauer, Mario Koddenbrock, Raphael Wallsberger, Nicholas M Brisson, Georg N Duda, Deborah Falla, David W Evans, and Erik Rodner. ’oh llm, i’m asking thee, please give me a decision tree’: Zero-shot decision tree induction and embedding with large language models. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pp. 1196–1206, 2025. Ron Kohavi and George H John. Wrappers for feature subset selection. Artificial intelligence, 97 (1-2):273–324, 1997. Daphne Koller and Mehran Sahami. Toward optimal feature selection. Technical report, Stanford InfoLab, 1996. Igor Kononenko, Edvard Šimec, and Marko Robnik-Šikonja. Overcoming the myopia of inductive learning algorithms with relieff. Applied Intelligence, 7(1):39–55, 1997. Gang Kou, Pei Yang, Yi Peng, Feng Xiao, Yang Chen, and Fawaz E Alsaadi. Evaluation of feature selection methods for text classification with small datasets using multiple criteria decision-making methods. Applied Soft Computing, 86:105836, 2020. Changki Lee and Gary Geunbae Lee. Information gain and divergence-based feature selection for machine learning-based text categorization. Information processing & management, 42(1): 155–165, 2006. Jing Lei, Max G’Sell, Alessandro Rinaldo, Ryan J. Tibshirani, and Larry Wasserman. Distribution-free predictive inference for regression, 2017. URL https://arxiv.org/abs/1604.04173. Dawei Li, Zhen Tan, and Huan Liu. Exploring large language models for feature selection: A data-centric perspective. ACM SIGKDD Explorations Newsletter, 26(2):44–53, 2025. Jundong Li, Kewei Cheng, Suhang Wang, Fred Morstatter, Robert P Trevino, Jiliang Tang, and Huan Liu. Feature selection: A data perspective. ACM Computing Surveys (CSUR), 50(6):94, 2018. Yingxia Li, Ulrich Mansmann, Shangming Du, and Roman Hornung. Benchmark study of feature selection strategies for multi-omics data. BMC bioinformatics, 23(1):412, 2022. Yun Li, Tao Li, and Huan Liu. Recent advances in feature selection and its applications. Knowledge and Information Systems, 53(3):551–577, 2017. Zechao Li, Yi Yang, Jing Liu, Xiaofang Zhou, and Hanqing Lu. Unsupervised feature selection using nonnegative spectral analysis. In Proceedings of the AAAI conference on artificial intelligence, volume 26, pp. 1026–1032, 2012. Deron Liang, Chia-Chi Lu, Chih-Fong Tsai, and Guan-An Shih. Financial ratios and corporate governance indicators in bankruptcy prediction: A comprehensive study. European journal of operational research, 252(2):561–572, 2016. Dahua Lin and Xiaoou Tang. Conditional infomax learning: An integrated framework for feature extraction and fusion. In European conference on computer vision, pp. 68–82. Springer, 2006. Huan Liu and Rudy Setiono. Chi2: Feature selection and discretization of numeric attributes. In Proceedings of 7th IEEE international conference on tools with artificial intelligence, pp. 388–391. Ieee, 1995. Huan Liu, Rudy Setiono, et al. A probabilistic approach to feature selection-a filter solution. In ICML, volume 96, pp. 319–327, 1996. Shiyu Liu and Mehul Motani. Improving mutual information based feature selection by boosting unique relevance. Journal of Artificial Intelligence Research, 82:1267–1292, 2025. Yang Liu, Sujay Khandagale, Colin White, and Willie Neiswanger. Synthetic benchmarks for scientific research in explainable machine learning. arXiv preprint arXiv:2106.12543, 2021. Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017. 16

Jiaqi Ma, Vivian Lai, Yiming Zhang, Chacha Chen, Paul Hamilton, Davor Ljubenkov, Himabindu Lakkaraju, and Chenhao Tan. Openhexai: An open-source framework for human-centered evaluation of explainable machine learning. arXiv preprint arXiv:2403.05565, 2024. Shuangge Ma and Jian Huang. Penalized feature selection and classification in bioinformatics. Briefings in bioinformatics, 9(5):392–403, 2008. Kamel Mansouri, Tine Ringsted, Davide Ballabio, Roberto Todeschini, and Viviana Consonni. Quantitative structure–activity relationship models for ready biodegradability of chemicals. Journal of chemical information and modeling, 53(4):867–878, 2013. George A Marcoulides. Discovering knowledge in data: An introduction to data mining, 2005. Mónica V Martins, Daniel Tolledo, Jorge Machado, Luís MT Baptista, and Valentim Realinho. Early prediction of student’s performance in higher education: A case study. In World Conference on Information Systems and Technologies, pp. 166–175. Springer, 2021. Akshay Mathur, Laxmi Mounika Podila, Keyur Kulkarni, Quamar Niyaz, and Ahmad Y Javaid. Naticusdroid: A malware detection framework for android using native and custom permissions. Journal of Information Security and Applications, 58:102696, 2021. Stan Matwin. Collective bargaining review. 1988. Nina Mazyavkina, Sergey Sviridov, Sergei Ivanov, and Evgeny Burnaev. Reinforcement learning for combinatorial optimization: A survey. Computers & Operations Research, 134:105400, 2021. Matthew B. A. McDermott, Haoran Zhang, Lasse Hyldig Hansen, Giovanni Angelotti, and Jack Gallifant. A closer look at auroc and auprc under class imbalance, 2025. URL https://arxiv. org/abs/2401.06091. Mary McLeish and Matt Cecile. Horse Colic. UCI Machine Learning Repository, 1989. DOI: https://doi.org/10.24432/C58W23. Nicolai Meinshausen and Peter Bühlmann. Stability selection. Journal of the Royal Statistical Society Series B: Statistical Methodology, 72(4):417–473, 2010. Patrick E Meyer and Gianluca Bontempi. On the use of variable complementarity for feature selection in cancer classification. In Workshops on applications of evolutionary computation, pp. 91–102. Springer, 2006. Jianyu Miao and Lingfeng Niu. A survey on feature selection. Procedia computer science, 91: 919–926, 2016. Ryszard S. Michalski and R. L. Chilausky. Learning by being told and learning from examples: An experimental comparison of the two methods of knowledge acquisition in the context of developing an expert system for soybean disease diagnosis. International Journal of Policy Analysis and Information Systems, 4(2), 1980. Sérgio Moro, Paulo Cortez, and Paulo Rita. A data-driven approach to predict the success of bank telemarketing. Decision Support Systems, 62:22–31, 2014. Sarah Nogueira and Gavin Brown. Measuring the stability of feature selection. In Joint European conference on machine learning and knowledge discovery in databases, pp. 442–457. Springer, 2016. Averkiy Oliabev. heloc. Kaggle, 2021. URL https://www.kaggle.com/datasets/ averkiyoliabev/home-equity-line-of-creditheloc. Ivan Olier, Noureddin Sadawi, G. Richard Bickerton, Josie Vanschoren, Celine Griner, Luca Soldatova, and Ross D. King. Meta-qsar: a large-scale application of meta-learning to drug design and discovery. Machine Learning, 107:285–311, 2018. doi: 10.1007/s10994-017-5685-x. URL https://doi.org/10.1007/s10994-017-5685-x. Ismail Parsa. Kdd cup 1998. https://kdd.ics.uci.edu/databases/kddcup98/ kddcup98.html, 1998. Dataset, UCI Machine Learning Repository. DOI: 10.24432/C5401H. 17

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. 12:2825–2830, 2011. Hanchuan Peng, Fuhui Long, and Chris Ding. Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy. IEEE Transactions on pattern analysis and machine intelligence, 27(8):1226–1238, 2005. Keith W Penrose, Arnold G Nelson, and Arnold Garth Fisher. Generalized body composition prediction equation for men using simple measurement techniques. Medicine & Science in Sports & Exercise, 17(2):189, 1985. Simon Perkins, Kevin Lacker, and James Theiler. Grafting: Fast, incremental feature selection by gradient descent in function space. Journal of machine learning research, 3(Mar):1333–1356, 2003. Emanuel F. Petricoin, Ali M. Ardekani, Ben A. Hitt, Peter J. Levine, Vincent A. Fusaro, Seth M. Steinberg, Gordon B. Mills, Charles Simone, David A. Fishman, Elise C. Kohn, and Lance A. Liotta. Use of proteomic patterns in serum to identify ovarian cancer. The Lancet, 359(9306):572– 577, 2002a. doi: 10.1016/S0140-6736(02)07746-2. URL https://doi.org/10.1016/ S0140-6736(02)07746-2. Emanuel F. Petricoin, David K. Ornstein, Cloud P. Paweletz, Ali Ardekani, Paul S. Hackett, Ben A. Hitt, Alfredo Velassco, Christian Trucco, Laura Wiegand, Kamillah Wood, Charles B. Simone, Peter J. Levine, W. Marston Linehan, Michael R. Emmert-Buck, Seth M. Steinberg, Elise C. Kohn, and Lance A. Liotta. Serum proteomic patterns for detection of prostate cancer. JNCI: Journal of the National Cancer Institute, 94(20):1576–1578, 2002b. doi: 10.1093/jnci/94.20.1576. URL https://doi.org/10.1093/jnci/94.20.1576. Mercedes Piedra, Sohier Dane, and Soraya Jimenez. Santander customer transaction prediction. https://kaggle.com/competitions/ santander-customer-transaction-prediction, 2019. Kaggle competition. Podsyp. Is this a good customer?, 2020. URL https://www.kaggle.com/datasets/ podsyp/is-this-a-good-customer. Jingang Qu, David Holzmüller, Gaël Varoquaux, and Marine Le Morvan. Tabiclv2: A better, faster, scalable, and open tabular foundation model. arXiv preprint arXiv:2602.11139, 2026. J. Ross Quinlan. Induction of decision trees. Machine learning, 1(1):81–106, 1986. J. Ross Quinlan. Simplifying decision trees. International journal of man-machine studies, 27(3): 221–234, 1987. Juha Reunanen. Overfitting in making comparisons between variable selection methods. Journal of Machine Learning Research, 3(Mar):1371–1382, 2003. Marko Robnik-Šikonja and Igor Kononenko. An adaptation of Relief for attribute estimation in regression. In Proceedings of the Fourteenth International Conference on Machine Learning (ICML), pp. 296–304. Morgan Kaufmann, 1997. Marko Robnik-Šikonja and Igor Kononenko. Theoretical and empirical analysis of relieff and rrelieff. Machine learning, 53(1):23–69, 2003. Jorge Ruiz-Ramírez, M de La Puente, Catarina Xavier, Adrián Ambroa-Conde, J Álvarez-Dios, A Freire-Aradas, Ana Mosquera-Miguel, Arwin Ralf, Christina Amory, Maria Alexandra Katsara, et al. Development and evaluations of the ancestry informative markers of the visage enhanced tool for appearance and ancestry. Forensic Science International: Genetics, 64:102853, 2023. Yvan Saeys, Inaki Inza, and Pedro Larranaga. A review of feature selection techniques in bioinformatics. bioinformatics, 23(19):2507–2517, 2007. 18

Yvan Saeys, Thomas Abeel, and Yves Van de Peer. Robust feature selection using ensemble feature selection techniques. In Joint European conference on machine learning and knowledge discovery in databases, pp. 313–325. Springer, 2008. C Okan Sakar, S Olcay Polat, Mete Katircioglu, and Yomi Kastro. Real-time prediction of online shoppers’ purchasing intention using multilayer perceptron and lstm recurrent neural networks. Neural Computing and Applications, 31(10):6893–6908, 2019. Jeffrey Schlimmer. 1985 model import car and truck specifications - automobile, 1985. Valter Schütz, Han Wu, Reza Rezvan, Linus Aronsson, and Morteza Haghir Chehreghani. Afabench: A generic framework for benchmarking active feature acquisition. arXiv preprint arXiv:2508.14734, 2025. Razieh Sheikhpour, Mehdi Agha Sarram, Sajjad Gharaghani, and Mohammad Ali Zare Chahooki. A survey on semi-supervised feature selection methods. Pattern recognition, 64:141–158, 2017. Daniel Siegismund, Matthias Fassler, Stephan Heyse, and Stephan Steigele. Benchmarking feature selection methods for compressing image information in high-content screening. SLAS technology, 27(1):85–93, 2022. Vincent G. Sigillito, Simon Wing, Larrie V. Hutton, and K. L. Baker. Classification of radar returns from the ionosphere using neural networks. 1989. URL https://api.semanticscholar. org/CorpusID:18522381. Dinesh Singh, Phillip G. Febbo, Kenneth Ross, Donald G. Jackson, Judith Manola, Christine Ladd, Pablo Tamayo, Andrew A. Renshaw, Anthony V. D’Amico, Jerome P. Richie, et al. Gene expression correlates of clinical prostate cancer behavior. Cancer Cell, 1(2):203–209, 2002. doi: 10.1016/S1535-6108(02)00030-2. URL https://doi.org/10.1016/S1535-6108(02) 00030-2. Samuel Sithakoul, Sara Meftah, and Clément Feutry. Beexai: Benchmark to evaluate explainable ai. In World Conference on Explainable Artificial Intelligence, pp. 445–468. Springer, 2024. Gordon K Smyth. Linear models and empirical bayes methods for assessing differential expression in microarray experiments. Statistical applications in genetics and molecular biology, 3(1), 2004. Avrum Spira, Jennifer Beane, Vishal Shah, Katrina Steiling, Gang Liu, Frank Schembri, Sara Gilman, Yves-Martine Dumas, Paula Calner, Paola Sebastiani, et al. Airway epithelial gene expression in the diagnostic evaluation of smokers with suspect lung cancer. Nature Medicine, 13(3):361–366, 2007. doi: 10.1038/nm1556. URL https://doi.org/10.1038/nm1556. Gregor Stiglic, Simon Kocbek, Igor Pernek, and Peter Kokol. Comprehensive decision tree models in bioinformatics. PloS one, 7(3):e33812, 2012. Lars Ståhle, Svante Wold, et al. Analysis of variance (anova). Chemometrics and intelligent laboratory systems, 6(4):259–272, 1989. R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. Adaptive computation and machine learning. MIT Press, 2 edition, 2018. Jiliang Tang, Salem Alelyani, and Huan Liu. Feature selection for classification: A review. Data classification: Algorithms and applications, pp. 37, 2014. Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58(1):267–288, 1996. Andreas M Tillmann, Daniel Bienstock, Andrea Lodi, and Alexandra Schwartz. Cardinality minimization, constraints, and regularization: a survey. SIAM Review, 66(3):403–477, 2024. Gaurav Topre. Bank customer churn dataset, 2022. URL https://www.kaggle.com/ datasets/gauravtopre/bank-customer-churn-dataset. 19

Mubarak Albarka Umar, Zhanfang Chen, Khaled Shuaib, and Yan Liu. Effects of feature selection and normalization on network intrusion detection. Data Science and Management, 8(1):23–39, 2025. Unknown. Annealing. UCI Machine Learning Repository, 1990. URL https://doi.org/10. 24432/C5RW2F. Unknown. Qsar oral toxicity. UCI Machine Learning Repository, 2019. URL https://doi. org/10.24432/C5PS4J. Luca Urbinati, Marco Ricci, Giovanna Turvani, Jorge A. Tobon Vasquez, Francesca Vipiana, and Mario R. Casu. A machine-learning based microwave sensing approach to food contaminant detection. In 2020 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 1–5, 2020. doi: 10.1109/ISCAS45731.2020.9181293. P van der Putten and M van Someren. Coil challenge 2000: The insurance company case, sentient machine research, amsterdam and leiden institute of advanced computer science. Technical report, Leiden LIACS Technical Report 2000-09, 2000. Michel Vidal-Naquet and Shimon Ullman. Object recognition with informative features and linear classification. In ICCV, volume 3, pp. 281, 2003. Erik D VonKaenel, Lisa M Bramer, Javier E Flores, Thomas O Metz, Ernesto S Nakayasu, and Bobbie-Jo M Webb-Robertson. A benchmarking study of feature screening approaches across omics classification settings. bioRxiv, pp. 2026–02, 2026. Raphael Wallsberger, Ricardo Knauer, and Stephan Matzka. Explainable artificial intelligence beyond feature attributions: The validity and reliability of feature selection explanations. In xAI (Late-breaking Work, Demos, Doctoral Consortium), pp. 1–8, 2024. Dongjie Wang, Yanyong Huang, Wangyang Ying, Haoyue Bai, Nanxu Gong, Xinyuan Wang, Sixun Dong, Tao Zhe, Kunpeng Liu, Meng Xiao, et al. Towards data-centric ai: A comprehensive survey of traditional, reinforcement, and generative approaches for tabular data transformation. ACM Transactions on Knowledge Discovery from Data, 2025. Jialei Wang, Peilin Zhao, Steven CH Hoi, and Rong Jin. Online feature selection and its applications. IEEE Transactions on knowledge and data engineering, 26(3):698–710, 2013. Suhang Wang, Jiliang Tang, and Huan Liu. Embedded unsupervised feature selection. In Proceedings of the AAAI conference on artificial intelligence, volume 29, 2015. Tong Wang, Cynthia Rudin, Finale Doshi-Velez, Yimin Liu, Erica Klampfl, and Perry MacNeille. A bayesian framework for learning rule sets for interpretable classification. Journal of Machine Learning Research, 18(70):1–37, 2017. Bing Xue, Mengjie Zhang, Will N Browne, and Xin Yao. A survey on evolutionary computation approaches to feature selection. IEEE Transactions on evolutionary computation, 20(4):606–626, 2015. Kaggle User Yakhyojon. Customer satisfaction in airline. urlhttps://www.kaggle.com/datasets/yakhyojon/customer-satisfaction-in-airline, 2023. Kaggle. Sen Yang, Lei Yuan, Ying-Cheng Lai, Xiaotong Shen, Peter Wonka, and Jieping Ye. Feature grouping and selection over an undirected graph. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 922–930, 2012. Yi Yang, Heng Tao Shen, Zhigang Ma, Zi Huang, and Xiaofang Zhou. l 2, 1-norm regularized discriminative feature selection for unsupervised learning. In IJCAI international joint conference on artificial intelligence, 2011. I-Cheng Yeh and Che hui Lien. The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients. Expert Systems with Applications, 36(2): 2473–2480, 2009. doi: 10.1016/j.eswa.2007.12.020. URL https://doi.org/10.1016/j. eswa.2007.12.020. 20

Lei Yu and Huan Liu. Feature selection for high-dimensional data: A fast correlation-based filter solution. In Proceedings of the 20th international conference on machine learning (ICML-03), pp. 856–863, 2003. Lei Yu and Huan Liu. Efficient feature selection via analysis of relevance and redundancy. Journal of machine learning research, 5(Oct):1205–1224, 2004. Daoqiang Zhang, Songcan Chen, and Zhi-Hua Zhou. Constraint score: A new filter method for feature selection with pairwise constraints. Pattern Recognition, 41(5):1440–1451, 2008. Zheng Zhao and Huan Liu. Semi-supervised feature selection via spectral analysis. In Proceedings of the 2007 SIAM international conference on data mining, pp. 641–646. SIAM, 2007. Jing Zhou, Dean Foster, Robert Stine, and Lyle Ungar. Streaming feature selection using alphainvesting. In Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining, pp. 384–393, 2005. Maciej Zieba, Sebastian Klaudiusz Tomczak, and Jakub M. Tomczak. Ensemble boosted trees with synthetic features generation in application to bankruptcy prediction. Expert Syst. Appl., 58:93–101, 2016. URL https://api.semanticscholar.org/CorpusID:40512567. Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society Series B: Statistical Methodology, 67(2):301–320, 2005. M. Zwitter and M. Soklic. Primary Tumor. UCI Machine Learning Repository, 1987. DOI: https://doi.org/10.24432/C5WK5Q. M. Zwitter and M. Soklic. Lymphography. UCI Machine Learning Repository, 1988. DOI: https://doi.org/10.24432/C54598.

21

Appendices TABLE OF C ONTENTS . A Feature Selection Taxonomy for Practitioners

23

B Methods

23

B.1 Reviewed Surveys and Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . .

23

B.2 Method Curation Criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

23

B.3 Method Curation Insights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

24

B.4 Method Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

25

B.5 Overview of All Investigated Feature Selection Methods . . . . . . . . . . . . . .

28

C Datasets

33

C.1 Dataset Sources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

33

C.2 Dataset Curation Criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

34

C.3 Dataset Curation Insights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

35

C.4 Overview of Selected Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . .

35

C.5 Cardinality Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

38

D Benchmarking Details and Extended Results

38

D.1 Open-source Benchmarking . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

38

D.2 Predictive Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

38

D.3 Stability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

44

D.4 Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

46

D.5 Raw Predictive Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

50

22

A

F EATURE S ELECTION TAXONOMY FOR P RACTITIONERS

Feature selection methods are traditionally categorized as filters, wrappers, or embedded methods, depending on whether feature relevance is assessed independently of the downstream model M , through repeated model evaluation, or during model training (Guyon & Elisseeff, 2003). This taxonomy can be useful from an implementation perspective, for example when deciding whether cardinality-constrained linear models should be realized via exhaustive enumeration (wrapper), mixedinteger programming (embedded method), or post-hoc training on a reduced subset (filter). However, it is less informative for practitioners, who are less concerned with where feature selection occurs in a pipeline (a structural perspective) and more with how the method approaches this combinatorial optimization problem (a functional perspective). We therefore adopt a complementary categorization into search and non-search methods (Kohavi & John, 1997). We define search methods as procedures that evaluate multiple subsets using a model-dependent subset utility uM (S). Due to the exponential size of the search space, search methods differ primarily in how they explore it efficiently, for example through sequential forward or backward search. In contrast, non-search methods avoid repeated subset evaluations under uM (S) and instead rely on proxy mechanisms, such as univariate relevance or recursive partitioning. While non-search methods may still involve search procedures, they optimize a different objective than the subset utility in Section 3.1. They include correlation-based, distance-based, information-theoretic, and tree-based approaches (Duch, 2006), as well as regularization-based methods (Tibshirani, 1996; Zou & Hastie, 2005). Figure A.1 provides an overview of the categorization for SelectArena. While not exhaustive, the subgrouping can be readily extended to cover additional search methods, such as evolutionary algorithms (Eiben & Smith, 2015; Wang et al., 2025; Xue et al., 2015) and reinforcement learning (Mazyavkina et al., 2021; Sutton & Barto, 2018; Wang et al., 2025), as well as non-search methods, such as game-theoretic (Covert et al., 2021; Fryer et al., 2021; Wallsberger et al., 2024) and knowledge-driven approaches (Jeong et al., 2025; Knauer et al., 2025; Li et al., 2025). PERSPECTIVE

METHODS

STRUCTURAL Filters

Where is the method implemented in the pipeline? FUNCTIONAL

Search methods

How is the combinatorial feature selection problem tackled?

Forward

Embedded methods

Wrappers

Non- search methods

Backward

Correlation

Distance

Information theory

Tree

Regularization

Figure A.1: Taxonomy of feature selection methods for SelectArena. We complement the traditional structural perspective (Guyon & Elisseeff, 2003) with a functional categorization into search and non-search approaches (Kohavi & John, 1997). Search subgroups include forward and backward procedures, non-search subgroups are based on Duch (2006) and extended to include regularization-based methods (Tibshirani, 1996; Zou & Hastie, 2005).

B

M ETHODS

B.1

R EVIEWED S URVEYS AND B ENCHMARKS

Table B.1 lists the 33 works, predominantly surveys and benchmarks, that we reviewed to assess method popularity. For each feature selection method, the Citations column in Table 2 (Section 3.2) reports the number of these works in which the method is mentioned. We use this count to exclude methods with less than three mentions, in accordance with method curation criterion (4) (Section 3.2). B.2

M ETHOD C URATION C RITERIA

(1) Suitability for tabular data: Since SelectArena focuses on tabular datasets, we exclude methods that are primarily designed for other data modalities. For example, Information Gain for Text (Lee & Lee, 2006) is specific to textual data and therefore not included. (2) Suitability for all tasks (binary classification, multi-class classification and regression): 23

Table B.1: Reviewed survey and benchmark works. The 33 works used to determine the popularity of each feature selection method, ordered by year of publication. Jain & Zongker (1997) Kohavi & John (1997) Guyon & Elisseeff (2003) Reunanen (2003) Duch (2006) Guyon et al. (2007a) Saeys et al. (2007) Zhao & Liu (2007) Saeys et al. (2008) Meinshausen & Bühlmann (2010) Brown et al. (2012)

Bolón-Canedo et al. (2013) Wang et al. (2013) Chandrashekar & Sahin (2014) Khalid et al. (2014) Tang et al. (2014) Jović et al. (2015) Xue et al. (2015) Miao & Niu (2016) Li et al. (2017) Sheikhpour et al. (2017) Heinze et al. (2018)

Li et al. (2018) Bommert et al. (2020) Kou et al. (2020) Covert et al. (2021) Hopf & Reifenrath (2021) Bouchlaghem et al. (2022) Li et al. (2022) Siegismund et al. (2022) Cherepanova et al. (2023) Schütz et al. (2025) Bakhmach et al. (2026)

SelectArena includes datasets for binary classification, multi-class classification, and regression. Consequently, selected methods must support all target types. Methods restricted to a subset of tasks are therefore excluded. For instance, χ2 (Liu & Setiono, 1995) is only applicable to classification problems and is thus not considered. (3) Methodological independence (not merely a variant of another method): The benchmark aims to include conceptually distinct feature selection methods rather than multiple variants of the same underlying approach. Therefore, methods such as Relief (Kira & Rendell, 1992) and ReliefF (Kononenko et al., 1997) are treated as methodologically similar, since ReliefF constitutes an extension of Relief. (4) Number of citations in the relevant literature: To ensure that the selected methods of the first cohort are established and widely recognized, we require each method to be cited at least three times across prominent surveys and benchmarking studies (Section 3.2). (5) Subgroup coverage: To promote methodological diversity, we employ a functional categorization of feature selection methods (Appendix A). The final method set in SelectArena is curated to ensure representation across methodological subgroups.

B.3

M ETHOD C URATION I NSIGHTS

Below, we share general insights we obtained during method curation: • Feature selection methods are frequently applied beyond the problem settings for which they were originally designed. For example, methods such as χ2 or Gini Impurity are inherently intended for categorical targets, yet they are often applied to regression tasks, e.g., by discretizing the target variable. Such adaptations may substantially influence method behavior and evaluation outcomes. To avoid introducing methodological distortions, we exclude these methods from SelectArena and aim to raise awareness of this issue within the field. • A similar pattern can be observed with respect to data modalities. Many methods are used across heterogeneous data types, including tabular, textual, and image data, despite relying on assumptions tailored to a specific modality. Applying methods outside their intended data setting may lead to misleading conclusions and should therefore be considered carefully in empirical evaluations. • The feature selection literature is currently fragmented into several largely independent research streams, including classical statistical approaches, evolutionary approaches, and explainable AI (XAI) approaches. Despite using similar underlying techniques, these communities often evolve separately and make limited use of each other’s methodological advances. We believe that stronger integration across these research directions could accelerate progress within the field. 24

B.4

M ETHOD I MPLEMENTATION

As specified in Section 3.2, feature selection is implemented based on AutoGluon (Erickson et al., 2020) and TabArena (Erickson et al., 2025). We introduce a class called AbstractFeatureSelector, which is a specialization of AbstractFeatureGenerator and part of the preprocessing pipeline in AutoGluon. Before feature selection, AutoGluon’s default preprocessing infers feature types and removes constant, ID-like, and duplicate features. Categorical features keep the pandas category dtype but are ordinally encoded, with categories seen fewer than two times in training treated as missing. There are two different types of AbstractFeatureSelectors: (1) the feature scoring type and (2) the subset selection type. Each feature selection method inherits the properties of the AbstractFeatureSelector, which are mainly the cardinality and time constraints, as well as a fallback in case the feature selection method does not return valid results. The fallback is implemented as a random feature selection using the intermediate results of the failed or canceled method, either sampling a score for the non-scored feature or completing the subset selection in case the intermediate subset is smaller than the cardinality constraint. The implementation is in most cases based on scikit-learn (Pedregosa et al., 2011) or scikit-feature (Li et al., 2018) (see Table 2 for details). The applied changes are usually the incorporation of a regular check for the time limit and a solution for returning the intermediate results when the time limit is met, as well as the introduction of pandas DataFrames instead of numpy arrays for clean data usage in the TabArena pipeline. In the following, we provide implementation details for all selected methods, including the assumptions of the algorithms, the preprocessing, the type, a description of the approach, the criterion, and some additional information on the implementation: B.4.1

C LASSIFICATION A ND R EGRESSION T REES (CART) (B REIMAN ET AL ., 1984)

Assumptions: Numerical features (sklearn). Preprocessing: Imputation. Type: Scoring. Approach: Greedy over tree structures (best split). Criterion: Mean decrease in impurity (MDI) per feature: total impurity reduction attributed to splits using that feature, where impurity is Gini for classification and variance for regression. Implementation: sklearn’s DecisionTreeRegressor and DecisionTreeClassifier.

B.4.2

E LASTIC N ET (Z OU & H ASTIE , 2005)

Assumptions: Numerical features (sklearn). Preprocessing: Imputation, scaling. Type: Scoring. Approach: Multivariate, embedded. Criterion: Magnitude of the coefficient |β̂j | from a joint linear (or logistic) fit with combined L1 and L2 regularization. Implementation: We use sklearn’s ElasticNetCV for regression and LogisticRegressionCV with elastic-net penalty for classification. We use the default L1-ratio grid but exclude the value 1.0 so the model never becomes pure Lasso.

B.4.3

F-T EST (S TÅHLE ET AL ., 1989)

Assumptions: Numerical features (sklearn). Preprocessing: Imputation. Type: Scoring. Approach: Univariate. Criterion: F-statistic — sklearn’s f_classif for classification (ratio of between- to within-class variance) and f_regression for regression (derived from squared Pearson correlation; equivalent in ranking to |r|). Implementation: We select the top-k features based on the returned F-statistics. 25

B.4.4

I NFORMATION G AIN R ATIO (G AIN R ATIO ) (Q UINLAN , 1986)

Assumptions: Discrete features (this implementation). Preprocessing: Imputation, discretization. Type: Scoring. Approach: Univariate. M I(Y,X) Criterion: Information gain ratio IGR = IG(Y,X) SI(X) = H(X) , where X is a feature. The denominator penalizes high-cardinality features, which would otherwise dominate plain MI rankings. Implementation: We reuse sklearn’s mutual_info_classif for the numerator IG(Y, X) and our own entropy estimator for the denominator H(X).

B.4.5

L APLACIAN S CORE (H E ET AL ., 2005)

Assumptions: Numerical features. Preprocessing: Imputation, scaling (for the graph construction). Type: Scoring. Approach: Unsupervised. Criterion: Locality preservation (lower is better): a feature scores well if points that are close in the kNN graph have similar values for that feature. Implementation: We build a symmetric kNN graph with a Gaussian heat-kernel on Euclidean distances and compute the score via the Rayleigh quotient against the graph Laplacian, following the original paper. The scikit-feature version is quite outdated, so we re-implemented it. The hyperparameters k = 5 and t = median(d2 ) are arbitrary choices.

B.4.6

L EAST A BSOLUTE S HRINKAGE AND S ELECTION O PERATOR (L ASSO ) (T IBSHIRANI , 1996)

Assumptions: Numerical features (sklearn). Preprocessing: Imputation, scaling. Type: Scoring. Approach: Multivariate, embedded. Criterion: Magnitude of the coefficient |β̂j | from a joint linear (or logistic) fit with L1 regularization. Implementation: We use sklearn’s LassoCV for regression and LogisticRegressionCV with L1 penalty for classification. The CV variants pick the regularization strength internally, so no α/C tuning is needed.

B.4.7

L EAVE -O NE -C OVARIATE -O UT (LOCO) (L EI ET AL ., 2017)

Assumptions: Proxy model assumptions. Preprocessing: None. Type: Scoring. Approach: Leave-one-out. Criterion: Per-feature performance drop from omission: LOCOj = L(fˆ−j ) − L(fˆfull ), where fˆfull is the proxy trained on all features and fˆ−j is the proxy trained without feature j. Implementation: We use LGBM as the proxy (handles NaNs natively, no preprocessing required) and refit it once per feature for each split. Costly on high-dimensional data.

B.4.8

M ARKOV B LANKET (KOLLER & S AHAMI , 1996)

Assumptions: Numerical features. Preprocessing: Imputation. Type: Selection. Approach: Greedy multivariate (backward). Criterion: A feature Xi is redundant if there exists Mi ⊆ F \ {Xi } (its Markov blanket) such 26

that Y ⊥ Xi | Mi . In practice we score each feature by the cross-entropy/MSE difference under 5-fold CV between models fit on Mi and Mi ∪ {Xi }, and remove the feature whose absence hurts performance the least. Implementation: The blanket Mi is taken to be the top-k Pearson-correlated features of Xi . Models are fit and evaluated under 5-fold CV using Ridge for regression and Logistic Regression for classification. The original formulation (Koller & Sahami, 1996) searches for a blanket that renders Y conditionally independent of all remaining features; we restrict to a fixed top-k neighborhood. We use k = 5 and deviate from the original formulation by adding the regression case.

B.4.9

MINIMUM R EDUNDANCY M AXIMUM R ELEVANCE ( M RMR) (P ENG ET AL ., 2005)

Assumptions: Discrete features (this implementation). Preprocessing: Imputation, discretization. Type: Selection. Approach: Greedy multivariate (forward). Criterion: MID variant: at step t with selected set St−1 , score(Xi ) = I(Xi ; Y ) − P 1 Xj ∈St−1 I(Xi ; Xj ); the first feature is picked by max relevance I(Xi ; Y ). |St−1 | Implementation: At each step we add the candidate maximizing relevance minus average pairwise redundancy with the already selected features. This is a re-implementation following scikit-feature’s MRMR.

B.4.10

M UTUAL I NFORMATION (MI) (BATTITI , 1994)

Assumptions: Numerical features (sklearn). Preprocessing: Imputation. Type: Scoring. Approach: Univariate. Criterion: Mutual information I(X, Y ) = H(X) + H(Y ) − H(X, Y ). Implementation: We use sklearn’s mutual_info_classif for classification and mutual_info_regression for regression. Both use frequency counts for features marked as discrete and the Kraskov kNN estimator for continuous features.

B.4.11

R ANDOM

Assumptions: None. Preprocessing: None. Type: Selection. Approach: Random. Criterion: None. Implementation: Random selection of k features based on the random seed.

B.4.12

R ANDOM F OREST I MPORTANCE (RFI MPORTANCE ) (B REIMAN , 2001)

Assumptions: Numerical features (sklearn). Preprocessing: Imputation. Type: Scoring. Approach: Multivariate, embedded. Criterion: Mean decrease in impurity (MDI) per feature, averaged over all trees in the forest: Gini impurity reduction for classification, variance reduction for regression. Implementation: We use sklearn’s RandomForestRegressor and RandomForestClassifier with default hyperparameters.

B.4.13

S EQUENTIAL BACKWARD E LIMINATION (SBE) (G UYON ET AL ., 2002)

27

Assumptions: Proxy model assumptions. Preprocessing: None. Type: Selection. Approach: Greedy multivariate (backward). Criterion: Holdout performance of the proxy model. Implementation: We use LGBM as the proxy model. At each step, the feature yielding smallest drop in performance after being excluded from the current subset is eliminated. B.4.14

(R)R ELIEF F (KONONENKO ET AL ., 1997; ROBNIK -Š IKONJA & KONONENKO , 1997)

Assumptions: None. Preprocessing: Imputation, min-max scaling. Type: Scoring. Approach: Multivariate. Criterion: A feature is rewarded when it differs more on an instance’s k nearest misses (different class) than on its k nearest hits (same class), with misses weighted by the class prior. RReliefF replaces the class machinery with P (diff feature | diff prediction), computed via Bayes rule running sums over kNN pairs. Implementation: Re-implemented to match the original algorithms, with a branched per-feature difference function: |a − b| for numeric features (on [0, 1]-scaled values) and 1{a ̸= b} for nominal features. Summed across features, this gives the Manhattan-style distance used to build the kNN graph itself. For ReliefF, kNN structures are cached per class for efficiency, with small-class guards that divide by the actual number of neighbors returned (rather than by k) when a class has fewer than k + 1 samples. Neighbor weighting is uniform (the paper also allows rank-based exponential weights). RReliefF is used for regression. We use k = 5 rather than the original k = 1.

B.4.15

S EQUENTIAL F ORWARD S ELECTION (SFS) (C OVER & VAN C AMPENHOUT , 1977)

Assumptions: Proxy model assumptions. Preprocessing: None. Type: Selection. Approach: Greedy multivariate (forward). Criterion: Holdout performance of the proxy model. Implementation: We use LGBM as the proxy model. At each step, the feature with the highest importance under the LGBM proxy refit on the current subset including this feature is added.

B.5

OVERVIEW OF A LL I NVESTIGATED F EATURE S ELECTION M ETHODS

Table B.2 provides a complete overview of all investigated methods, including the source links for a clear distinction and the final decision for inclusion or exclusion. Table B.2: Overview of all investigated feature selection methods. We list all methods with their source links, provide the year, and the number of appearances across the relevant literature (Section 3.2). In the Decision column, we use Yes if the method is included in SelectArena or list the reason for exclusion (Citations if there are fewer than three citations, Variation if the method is a variation of another method, Wrong Domain if the method is not suitable for tabular data, and Not task-agnostic if the method is not suitable for all tasks). We additionally mark some methods with Yes, future work to indicate that their implementations are ongoing. In all cases, these implementations are missing differential entropy logic. Feature Selection Method Name

Year

# appearances

Decision

AACO ACO 1 ACO 2 ACO 3 ACO 4 AISCBR

2024 2004 2010 2012 2014 2011

1 1 1 1 1 1

Citations Citations Citations Citations Citations Citations

28

Table B.2: Overview of all investigated feature selection methods (continued). Feature Selection Method Name

Year

# appearances

Decision

AMIFS ANOVA F-Test AUMFS Accuracy Adaptive Lasso Alpha-Investing Attention Map Importance BAB* BRIDGE regularization BSS/WSS BW-Ratio Balanced Error Rate Ball Correlation (Bcor-SIS) Bayesian Bivariate Bolasso Boruta C4.5 CAE-S CART CAWOA-NRS CEFS+ CFS (Hall) CMIFS CMIM CPFS CXPlain Categorical Concordance Index (CI-SIS) Category-Adaptive Variable Screening (CAS) Combined Ant Colony and Differential Evolution FS Algorithm Concordance Index (CSIS) Conditional Distance Correlation (CDCSIS) Conditional Infomax Feature Extraction Conditional Perm. Test Conditional SIS Consistency Constraint score Correlation Coefficient DE 2 DIME-DG DISR DeepFS Distance Correlation (DC-SIS) Distinguishing Feature Selector EDDI-GG EHOFS EUFS Elastic net Equal Information Gain Evolution Strategy FS Exclusive Lasso Expected Cross Entropy Extremal Perturbations (EP) FCBF FIDO-CA FIGFS FOCUS FS-P FSDS FSRDE

2004 1989 2013 2006 2012 2005 2021 1993 2008 2002 2009 2006 2018 2001 2002 2008 2010 1993 2019 1984 2026 2025 1999 2011 2004 2010 2019 2023 2019 2008

1 6 1 3 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 4 0 0 8 1 5 1 1 1 1 1

Citations, Variation Yes Wrong Domain, Citations Yes Citations, Variation Citations Citations, Variation Citations Citations Citations Citations Citations Citations Citations Citations Citations, Variation Citations, Variation Citations Citations Yes Citations Citations Not task-agnostic Citations, Variation Not task-agnostic Citations Citations Citations Citations Citations

2024 2015 2005 2008 2012 2000 2008 2014 2014 2023 2006 2015 2012 2012 2019 2025 2015 2005 1966 2009 2010 2009 2019 2003 2018 2025 1994 2006 2015 2026

1 1 1 1 1 3 1 1 1 1 5 1 1 1 1 0 1 3 1 1 1 1 1 5 1 0 1 1 1 0

Citations Citations Citations Citations Citations Not task-agnostic Citations Citations Citations Citations Not task-agnostic Citations Citations Citations Citations Citations Citations, Variation Yes Citations, Variation Citations Citations, Variation Citations Citations Variation Citations Citations Citations Citations Wrong Domain, Citations Citations

29

Table B.2: Overview of all investigated feature selection methods (continued). Feature Selection Method Name

Year

# appearances

Decision

Feature Similarity (Mahalanobis) Feature Weighting k means Feature selection for sparse clustering Fisher criterion Fisher score GA 1 GA 2 GA 3 GDCOV GDFS-DG GFLasso GP 1 GP 2 GP 3 GSBE GSFS Gain Ratio Gain Ratio Cost Gamma Generalized SIS Genetic Algorithm (Siedlecki) Genetic Algorithm (Yang) Gini Grafting Graph FS (GOSCAR) GreedyFS Group Lasso HSIC Lasso Heuristic Search (GA) Hilbert-Schmidt Independence Criterion (HSIC) IAIRS ICS-RCFS ID3 IF IME (2009) IME (2010) INTERACT INVASE ISFS Impurity Impurity Euclid Impurity Hellinger Inconsistency criterion Information Gain Information Gain Text Informative Fragment Interaction Capping Interaction Gain FS JAFA-MFRL JIM JMI JMIM Jaccard Index KernelSHAP Koller-Sahami metric (KS) Kolmogorov Filter (Kfilter) Kruskal-Wallis L2X LFS (lazy feature selection LIME (Images) LIME (Tabular)

1936 2003 2010 2012 1936 1989 2010 2014 2008 2023 2009 2006 2010 2012 1997 1997 1986 2003 2001 2009 1989 2002 1912 2003 2012 2011 2020 2018 1981 2022 2014 2026 1986 2003 2009 2010 2009 2018 2026 2006 2021 2021 1996 1986 2006 2003 2011 2008 2018 2021 1999 2015 2008 2022 1996 2014 2015 2018 2009 2019 2021

1 1 1 2 6 1 1 1 1 1 1 1 1 1 1 1 3 1 1 1 2 2 6 2 2 1 3 1 1 1 0 0 1 1 1 1 2 1 0 3 1 1 1 7 1 1 2 1 1 1 5 2 1 1 1 1 2 1 1 1 1

Citations Citations Citations Citations, Variation Not task-agnostic Citations Citations Citations Wrong Domain, Citations Citations Citations, Variation Citations Citations Citations Citations Citations Yes Citations, Variation Citations Citations Citations Citations Not task-agnostic Citations, Variation Citations, Variation Citations Variation Citations, Variation Citations Citations Citations Citations Citations Citations Citations Citations Not task-agnostic Citations Citations Not task-agnostic Citations, Variation Citations, Variation Citations Not task-agnostic Wrong Domain Citations Citations Citations Citations Citations, Variation Not task-agnostic Citations Citations Citations, Variation Citations Citations Citations, Variation Citations Citations Wrong Domain, Citations Citations

30

Table B.2: Overview of all investigated feature selection methods (continued). Feature Selection Method Name

Year

# appearances

Decision

LLM-Select LS (Flip Swap) first improve LS (Flip) best improve LSSSMV-FS LUFS Laplacian score Lasso (L1) Lasso Net LinkedFS Localized Feature Selection Based on Scatter Separability (LFSBSS) Logistic Regression Coefficients LossSHAP MA-C MAFESE MBEGA MCFS MDL MDL Lamp MI (Battiti) MI (Guyon) MIFS-U MM MOPSO MRMR MVFS Markov blanket Martingale Difference Correlation (MDC-SIS) Maximum Correlation (MC-SIS) Maximum Mean Discrepancy MMD Md (measure of monotone dependence) Mean-Variance (MVSIS) Meaningful Perturbations (MP) MinRed Minimal image representation (MIR) Modified Perceptron by Truncation for OFS MsFDSs Multi-response rank canonical correlation (mRCC) Multi-viewFS Multinomial Logistic Regression Myopic ReliefF NB NDFS NJMIM NP (node pruning) NetFS Nonparametric Independence Screening (NIS) ODIN-MBRL ODIN-MFRL OFS OSCAR OSFS ObDI-FS Occlusion Odds Ratio Original (no FS) Orthogonal Matching Pursuit PAM PCA PSO 1 PSO 2

2025 / / 2026 2012 2005 1996 2019 2012 2008

0 0 0 0 1 5 8 1 1 1

Citations Citations Citations Citations Wrong Domain, Citations Yes Yes Citations, Variation Wrong Domain, Citations Citations

2002 2020 2010 2024 2007 2010 1994 2003 1994 2003 2002 2017 2025 2005 2013 1996 2013 2016 2022 2010 2015 2017 2006 2014 2014 2025 2021 2025 2006 1994 1966 2012 2015 1994 2016 2011 2019 2019 2014 2008 2010 2026 2014 2001 nan 1993 2002 2014 2010 2013

1 1 1 1 1 3 2 1 6 2 1 1 0 12 1 3 1 1 1 1 1 1 1 1 1 0 1 0 1 1 1 2 2 1 1 1 1 1 1 1 2 0 1 1 0 1 1 1 1 1

Citations Citations, Variation Citations Citations Citations Variation Citations, Variation Citations, Variation Yes Variation Citations, Variation Citations Citations Yes Wrong Domain, Citations Yes Citations Citations Citations Citations Citations Citations Citations Wrong Domain, Citations Citations Citations Citations Citations Citations Citations, Variation Citations Citations Citations, Variation Citations Wrong Domain, Citations Citations Citations Citations Wrong Domain, Citations Citations, Variation Wrong Domain, Citations Citations Citations Citations Not task-agnostic Citations Citations Citations Citations Citations

31

Table B.2: Overview of all investigated feature selection methods (continued). Feature Selection Method Name

Year

# appearances

Decision

PSO 3 PT-S Pairwise SIS (PSIS) Pearson Correlation Coefficient Permutation Test PredDiff Projection Correlation (PC-Screen) QII Quantile Correlation (QCSIS) REAL-X RF importance RISE ROC Random FS Random FS (fixed size) Random permutations Randomised Lasso Rank products Regression Relief Relief Kukar Relief-A Relief-B Relief-C Relief-D ReliefF ReliefF averaged costs ReliefF avg probablity ReliefF cost-sensitive sampling ReliefF distance ReliefF equal K ReliefF expected costs ReliefF expected probability ReliefF expected rank ReliefF squared distance Repeated Elastic Net Technique (RENT) Ridge (L2) Robust Rank Correlation (RRCS) SAGE (Shapley Additive Global importancE) SBE (sequential backward elimination) SFBE SFFS SFS (sequential forward selection) SHAP SHAPIQ SIT-BY SOA SOD-LSC SPFS-SFS SPVIM SVM-RFE Shapicant Shapley Values Sparse Group Lasso Sparse-based filter method Spearman Correlation Coefficient Spectral FS (SPEC) Stability Selection Stabl Lasso Sure Independence Screening (SIS) Sure Independent Ranking and Screening (SIRS)

2002 2001 2015 2008 2001 2017 2019 2016 2016 2021 2001 2018 2001 2026 2026 2001 2008 2004 2001 1992 1999 1994 1994 1994 1994 1997 2003 2003 2003 2003 2003 2003 2003 2003 2003 2021 2022 2012 2020 2025 1994 1994 2025 2017 2024 2022 2009 2014 2013 2020 2002 2020 1953 2014 2015 2008 2007 2008 2024 2008 2010

1 1 1 4 1 1 1 1 1 1 8 1 1 1 2 1 1 1 1 9 2 0 0 0 0 10 1 1 1 1 1 1 1 1 1 1 1 1 1 8 2 3 10 1 0 1 1 1 1 1 6 1 1 1 1 2 3 1 1 1 1

Citations Citations Citations Not task-agnostic Citations Citations Citations Citations Citations Citations Yes Citations Citations Not task-agnostic Not task-agnostic Citations Citations, Variation Citations Citations Variation Citations, Variation Citations, Variation Citations, Variation Citations, Variation Citations, Variation Yes Citations, Variation Citations, Variation Citations, Variation Citations, Variation Citations, Variation Citations, Variation Citations, Variation Citations, Variation Citations, Variation Citations, Variation Citations Citations Citations, Variation Yes Variation, Citations Variation Yes Citations, Variation Yes Citations Citations Citations Citations Citations Variation Citations, Variation Citations Citations, Variation Wrong Domain, Citations Citations Variation Citations Citations, Variation Citations Citations

32

Table B.2: Overview of all investigated feature selection methods (continued). Feature Selection Method Name

Year

# appearances

Decision

Symmetrical Uncertainty TNOM Trace Ratio Tree structured group Lasso Tree Lasso Tree-guided group Lasso TreeSHAP Ts-repso UDFS USC USFS Uniform Accuracy Uniform Gini Uniform Information Gain Univariate Predictors Weighted Leverage Score (WLS) Weighted Log Likelihood Ratio Wilcoxon rank sum WrapperSubsetEval XCSCFˆ2 auc bee colony approach FS bumble bees mating optimization FS chi squared dist angle dist euclid dist hellinger distance entropy equal gini equal hellinger f_regression limma low variance mRMR-MOBSTA max of minority oneR sam signal-to-noise (SNR) sum minority t-test term frequency–inverse document frequency true positives (TP) univariate model score variance xgb importance

1988 2000 2008 2010 2023 2010 2020 2025 2011 2003 2015 2021 2021 2021 2003 2021 2000 2001 2005 2014 2020 2015 2014 1995 2021 2021 2021 2014 2014 2021 2021 2025 2004 2018 2026 2013 1993 2001 2007 2013 2011 2020 2007 2016 2020 2016

5 1 1 1 1 1 1 0 2 1 1 1 1 1 1 1 1 1 1 1 2 1 1 6 1 2 1 1 1 1 1 2 1 1 0 1 3 1 1 1 5 1 1 1 2 1

Not task-agnostic Citations Citations, Variation Citations, Variation Citations, Variation Citations, Variation Citations, Variation Citations Citations, Variation Citations Wrong Domain, Citations Citations, Variation Citations, Variation Citations, Variation Citations Citations Citations Citations Citations Citations Citations, Variation Citations Citations Not task-agnostic Citations Citations Citations Citations Citations Citations, Variation Citations Citations Citations, Variation Citations, Variation Citations Citations Not task-agnostic Citations, Variation Citations Citations Not task-agnostic Citations Citations Citations Citations Citations, Variation

C

DATASETS

C.1

DATASET S OURCES

Table C.1 lists the 73 articles, comprising feature selection surveys and benchmarks, the papers introducing our selected methods, and related work from explainable AI, that we reviewed to assemble the pool of 368 candidate datasets. 33

Table C.1: Reviewed dataset sources. The 73 articles screened for real-world tabular datasets, in alphabetical order. Agarwal et al. (2022) Bakhmach et al. (2026) Battiti (1994) Belaid et al. (2023) Bennasar et al. (2015) Bolón-Canedo et al. (2013) Bommert et al. (2020) Bouchlaghem et al. (2022) Breiman (2001) Brown et al. (2012) Cai et al. (2010) Chandrashekar & Sahin (2014) Chen (2016) Cherepanova et al. (2023) Covert et al. (2020) Dash & Liu (2003) Erickson et al. (2025) Gao et al. (2017) Gorishniy et al. (2021) Grinsztajn et al. (2022) Guyon & Elisseeff (2003) Guyon et al. (2007a) Hall (1999) He et al. (2005) Holte (1993) Hopf & Reifenrath (2021) Khafaga et al. (2023)

C.2

Kira & Rendell (1992) Kohavi & John (1997) Koller & Sahami (1996) Kononenko et al. (1997) Kou et al. (2020) Lee & Lee (2006) Li et al. (2012) Li et al. (2017) Li et al. (2018) Li et al. (2022) Lin & Tang (2006) Liu & Setiono (1995) Liu et al. (1996) Liu et al. (2021) Liu & Motani (2025) Lundberg & Lee (2017) Ma & Huang (2008) Ma et al. (2024) Meyer & Bontempi (2006) Miao & Niu (2016) Peng et al. (2005) Perkins et al. (2003) Reunanen (2003) Heinze et al. (2018) Hooker et al. (2019) Jain & Zongker (1997)

Robnik-Šikonja & Kononenko (2003) Ruiz-Ramírez et al. (2023) Saeys et al. (2008) Schütz et al. (2025) Sithakoul et al. (2024) Smyth (2004) Stiglic et al. (2012) Umar et al. (2025) Vidal-Naquet & Ullman (2003) VonKaenel et al. (2026) Wang et al. (2013) Wang et al. (2015) Wang et al. (2017) Yang et al. (2011) Yang et al. (2012) Yu & Liu (2003) Yu & Liu (2004) Zhang et al. (2008) Zhao & Liu (2007) Zhou et al. (2005)

DATASET C URATION C RITERIA

(1) Unique predictive task definition: A task with a given name should only exist once in SelectArena. Tasks with the same or similar name and/or different shapes require careful investigation to recover the original dataset version and avoid duplicates or inconsistent variants. (2) Tabular data (e.g., no text or image data): We restrict SelectArena to genuinely tabular datasets and therefore exclude datasets derived from text, image, or other modalities. Without domainspecific expertise, such datasets cannot reliably be treated as tabular data and would compromise the comparability and validity of benchmark results. (3) Minimum 10 features for meaningful selection: Cardinality-constrained feature selection (see Section 3.1) requires a sufficient number of features to define meaningful cardinality constraints, which is why datasets with fewer than 10 features are excluded. (4) Real-world origin (no synthetic data): The goal of SelectArena is to provide guidance for practitioners. Despite advances in synthetic data generation, synthetic datasets are not representing real-world distributions and may distort the interpretation of benchmark results. (5) IID data suitable for random splitting: All datasets in SelectArena are evaluated using random splits. Datasets containing temporal dependencies, grouped observations, or other structures requiring specialized splitting strategies are excluded to avoid information leakage and invalid performance estimates. (6) Genuine predictive task: Each dataset must contain a valid predictive target. We exclude datasets where targets are artificially constructed, randomly assigned, or otherwise not intended for predictive modeling, as can occur in certain scientific data repositories. (7) No ethical concerns with the predictive task: Datasets and their associated tasks are carefully reviewed for ethical concerns before inclusion. The well-known case of the Boston Housing dataset illustrates the importance of critically examining datasets and predictive tasks prior to making them broadly accessible to the community. 34

(8) Accessible source data: Datasets referenced in the literature must be available and usable. During the curation process, we experienced several cases where datasets were not publicly available (e.g., Bakhmach et al. (2026)) or the source links outdated (e.g., Guyon & Elisseeff (2003)). We therefore exclude datasets that could not be accessed, verified, and reliably reused. C.3

DATASET C URATION I NSIGHTS • Feature selection and machine learning methods are routinely applied across a wide range of datasets, regardless of their original modality. In our opinion, tabular data derived from, e.g., texts or images cannot be safely treated as standard tabular data without domain-specific expertise. • Feature selection is typically of limited practical relevance for datasets with fewer than 10 features, as only few candidate features are available for selection. In addition, defining meaningful budgets for cardinality-constrained feature selection (see Section 3.1) requires a sufficiently large feature space. • The IID assumption receives limited attention in the literature, even if an inappropriate splitting strategy can cause information leakage and misleading results (e.g., in case of temporal dependencies or grouped observations). • Sensitive features are frequently included in benchmarks without critical evaluation. Datasets from ethically sensitive domains appear more prevalent within the XAI-focused feature selection literature. In our view, both the predictive task and the presence of sensitive attributes should be carefully considered when deciding whether a dataset is appropriate for benchmarking. • High-dimensional datasets are typically accompanied by limited sample sizes, e.g., in biomedical applications where thousands of genes may be measured for only a small number of patients. These settings are especially prone to overfitting and overtuning. • Determining whether a dataset represents real-world or synthetic data can be challenging, particularly for heavily preprocessed datasets where feature names are anonymized and the original source is no longer identifiable (e.g., AutoML challenge data). • In some cases, we observed an arbitrary choice of target variables. This issue commonly arises in scientific discovery datasets that were not originally designed for predictive modeling, where targets are later selected from existing features, or artificially constructed, resulting in diverging variants of the same dataset.

C.4

OVERVIEW OF S ELECTED DATASETS

Table C.2 provides details on the included datasets. Table C.2: Overview of selected datasets. We list all datasets included in SelectArena, the number of features, samples and classes, as well as the task type and license for the usage of the datasets. Dataset allstate-claims-severity (Ferguson et al., 2016) ancestry-study (Ruiz-Ramírez et al., 2023) anneal (Unknown, 1990) aps-failure (IDA2016Challenge, 2016) audiology-diagnosis (Bareiss et al., 1990) automobile (Schlimmer, 1985) bad-customer-detection (Podsyp, 2020) bank-customer-churn (Topre, 2022)

# Features

# Samples

# Classes

Task type

License

131

188318

/

Regression

Public

105

635

7

Multiclass

Public

39 171

898 76000

5 2

Multiclass Binary

CC-BY-4 CC-BY-4

69

199

24

Multiclass

/

25 14

205 1723

7 2

Ordinal Binary

11

10000

2

Binary

CC-BY-4 CC0: Public Domain Public

35

Table C.2: Overview of selected datasets (continued). Dataset bank-marketing (Moro et al., 2014) bioresponse (Hamner et al., 2012) body-density-prediction (Penrose et al., 1985) brasilianhouses (de Oliveira Farias & Fontes, 2020) churn (Marcoulides, 2005) clock-protein-period (Gul et al., 2021) clock-protein-toxicity (Gul et al., 2021) coil-2000-insurancepolicies (van der Putten & van Someren, 2000) colon-tumor (Alon et al., 1999) credit-approval (Quinlan, 1987) credit-card-clients-default (Yeh & hui Lien, 2009) customer-satisfaction-inairline (Yakhyojon, 2023) drug-induced-autoimmunityprediction (Huang et al., 2025) forest-fires (Cortez & Morais, 2007) framingham-heart-study (Bhardwaj, 2022) give-me-some-credit (Fusion & Cukierski, 2011) hazelnut-spread-contami/tdetection-10GHz (Urbinati et al., 2020) heart-disease-cleveland (Detrano et al., 1989) heart-disease-hungary (Detrano et al., 1989) heart-disease-switzerland (Detrano et al., 1989) heart-disease-va-long-beach (Detrano et al., 1989) heloc (Oliabev, 2021) hepatitis-survivalprediction (Efron & Gong, 1981) hiva-agnostic (Guyon et al., 2007b) homesite-quote-conversion (Darrel et al., 2015) horse-colic-survival (McLeish & Cecile, 1989) hr-analytics (Arashnic, 2021) in-vehicle-couponrecommendation (Wang et al., 2017) ionosphere (Sigillito et al., 1989) kdd-cup-09-appetency (Guyon et al., 2009) labor-negotiations (Matwin, 1988)

# Features

# Samples

# Classes

Task type

License

13

45211

2

Binary

CC-BY-4

1777 14

3751 252

2 /

Binary Regression

Competition Public

13

10692

/

Regression

Public domain

20 1092

5000 90

2 2

Binary Binary

MIT License CC-BY-4

1112

171

2

Binary

CC-BY-4

85

9822

2

Binary

/

1992 16 24

62 690 30000

2 2 2

Binary Binary Binary

Public Public /

22

129880

2

Binary

178

597

2

Binary

CC0: Public Domain CC-BY-4

13

517

/

Regression

Public

16

4240

2

Binary

Unknown

12

150000

2

Binary

Public

31

2400

2

Binary

CC-BY-SA

14

303

2

Binary

14

294

2

Binary

13

123

2

Binary

14

200

2

Binary

24 20

10459 155

2 2

Binary Binary

CC0: Public Domain CC0: Public Domain CC0: Public Domain CC0: Public Domain Public CC-BY-4

1481

4229

2

Binary

Public

296

260753

2

Binary

Competition

21

344

3

Multiclass

CC-BY-4

13

19158

2

Binary

24

12684

2

Binary

CC0: Public Domain CC-BY-4

34 212

351 50000

2 2

Binary Binary

17

57

2

Binary

36

CC-BY-4 CC0: Public Domain CC-BY-4

Table C.2: Overview of selected datasets (continued). Dataset

# Features

# Samples

# Classes

Task type

License

7072

72

2

Binary

Public

18

302

22

Multiclass

CC-BY-4

12601

203

5

Multiclass

Public

19

142

4

Multiclass

CC-BY-4

26

2240

2

Binary

Public

206

23814

2

Multiclass

Competition

24

10459

/

Regression

112 87

1700 29332

2 2

Multiclass Binary

CC-BY-NCSA CC-BY-4 CC-BY-4

15148

253

2

Binary

Public

6771

181

2

Binary

Unknown

15154

322

2

Binary

Unknown

18

12330

2

Binary

CC-BY-4

94

61878

9

Multiclass

Competition

65

5790

2

Binary

CC-BY-4

38 37

595206 4424

2 3

Binary Multiclass

Public CC-BY-4

12601 479

102 144095

2 /

Binary Regression

Public Public

42

1055

2

Binary

CC-BY-4

1025

8992

2

Binary

CC-BY-4

1025

5742

/

Regression

santander-customer-transactionprediction (Piedra et al., 2019) sdss-17 (Abazajian et al., 2009)

601

200000

2

Binary

CC0: Public Domain Competition

12

100000

3

Multiclass

smoking-lung-cancer (Spira et al., 2007) soybean-large (Michalski & Chilausky, 1980) srbct-prediction (Khan et al., 2001) superconductivity (Hamidieh, 2018) taiwanese-bankruptcyprediction (Liang et al., 2016) thyroid-discordant (Quinlan, 1987)

22216

187

2

Binary

Public Domain Public

36

306

18

Multiclass

CC-BY-4

2309

83

4

Multiclass

Public

83

21263

/

Regression

CC-BY-4

96

6819

2

Binary

CC-BY-4

27

3711

5

Multiclass

leukemia-allaml (Golub et al., 1999) ljubljana-primary-tumor (Zwitter & Soklic, 1987) lung-cancer (Bhattacharjee et al., 2001) lymphography (Zwitter & Soklic, 1988) marketing-campaign (Unknown, 1990) mechanisms-of-action (inversion et al., 2020) miami-housing (Bourassa et al., 2021) mic (Golovenkin et al., 2020) naticusdroid-androidpermissions-dataset (Mathur et al., 2021) nci-ovarian-cancer (Petricoin et al., 2002a) nci-pancreatic-cancer (Hingorani et al., 2003) nci-prostate-cancer (Petricoin et al., 2002b) online-shoppers-purchasingintention-dataset (Sakar et al., 2019) otto-group-product-classificationchallenge (Bossan et al., 2015) polish-companiesbankruptcy (Zieba et al., 2016) porto-seguro (Howard et al., 2017) predict-students-dropout-andacademic-success (Martins et al., 2021) prostate-cancer (Singh et al., 2002) pva-revenue-predictionkddcup98 (Parsa, 1998) qsar-biodeg (Mansouri et al., 2013) qsar-oral-toxicity (Unknown, 2019) qsar-tid-11 (Olier et al., 2018)

37

/

Table C.2: Overview of selected datasets (continued). Dataset

# Features

# Samples

# Classes

Task type

License

12

6497

/

Regression

CC-BY-4

wine-quality (Cortez et al., 2009)

C.5

C ARDINALITY C ONSTRAINTS

As presented in section 3.3, we calculate the cardinality constraint k based on the feature dimensionality p following one of the two formulas, conditioned on p. If p ≤ 100, we consider the dataset to have a low feature dimension and we spread the constraint k linearly across the feature set size. The datasets with up to 100 features come from very different domains, e.g. marketing, life sciences, business. Since practical tasks in these domains can range from highly sparse feature selection to settings where retaining dozens or nearly 100 features is feasible, we use a linear cardinality spacing to provide balanced coverage across this range of plausible practitioner requirements. We consider datasets with more than 100 features to have high feature dimension and we spread the constraint k in an exponential way. These datasets come primarily from biological and medical domains, where feature selection tasks often aim to identify substantially smaller subsets of relevant features rather than retain most of the original feature set. We therefore allocate more of the evaluated budgets toward smaller feature subsets, while still including larger budgets to cover less restrictive selection settings. Formally, we calculate cardinality constraint k for j ∈ {1, . . . , 5} as following: k = ⌈p × j/6⌉ if p ≤ 100, k = ⌈pj/6 ⌉ otherwise. We summarize the distribution of the features across the cardinality indices j in Table C.3. Table C.3: Number of selected features k across the 71 benchmark datasets, per cardinality index j. Cardinality idx j

1 2 3 4 5

D

Min k

Max k

Linear (p ≤ 100) min–max k

Exp. (p > 100) min–max k

Mean k

2 4 5 7 9

16 31 150 791 4190

2–16 4–31 5–47 7–62 9–78

3–6 5–29 11–150 23–791 48–4190

4.7 11.1 27.4 87.3 344.5

B ENCHMARKING D ETAILS AND E XTENDED R ESULTS

In this section, we present the benchmarking details for predictive performance (D.2), stability (D.3), and validity (D.4). Finally, we provide the raw predictive performance results (D.5). D.1

O PEN - SOURCE B ENCHMARKING

The code of our method implementations and evaluation pipeline is publicly available on GitHub (under construction). Additionally, we publish the curated versions of our datasets. D.2

P REDICTIVE P ERFORMANCE

In the following paragraphs, we present the calculation and results for the Elo, provide details on the Pareto calculation, and show the results for improvement over random and runtime per 1k features. Additionally, we provide an ablation study on the impact of tuning and discuss the statistical significance of our findings. Evaluation metrics. We evaluate the predictive performance of the downstream models using area under the receiver operating characteristic curve (ROC AUC) for binary classification, log-loss 38

for multi-class classification, and root mean squared error (RMSE) for regression, following other popular benchmarks (Erickson et al., 2025; Gijsbers et al., 2023). We do not use other metrics, that might seem to be more suited for imbalanced datasets: ROC AUC is well-suited to comparing performance across heterogeneous, often imbalanced, datasets as it is invariant to class prevalence. We do not use the area under the precision-recall curve (PR AUC), a popular alternative for imbalanced data, as it favors improvements on subpopulations with higher prevalence (McDermott et al., 2025). ROC AUC is also threshold-free, i.e., it does not require fixing a decision threshold τ for model predictions ŷ = 1[p̂(x) > τ ], where p̂(x) is the predicted probability of the positive class. This avoids the problem of defining costs for different error types (false positive or false negative), which depend on the task and application context. Log-loss is a strictly proper scoring rule which allows evaluating how well the predicted distribution matches the true distribution as measured on the held-out set. Similar to ROC AUC, it does not commit to a fixed misclassification cost, which we cannot specify in advance for a benchmark spanning heterogeneous datasets and application contexts. RMSE plays an analogous role to log-loss as it is minimized in expectation only by the true conditional mean which implies that a method cannot improve its score other than by predicting better. It additionally keeps per-dataset results directly interpretable by retaining the units of the target variable (contrary to, e.g., the mean squared error). Elo Rating. We use an Elo-like rating for comparing the included feature selection methods. The Elo rating system has originally been designed to rate chess players and therefore its nature is time-dependent, i.e. the order of the matches matters as each update uses a player’s current rating to compute the expected outcome. More formally, the rating of player A is calculated as: 1 rA ← rA + K (SA − EA ), EA = , (1) 1 + 10(rB −rA )/400 where K = 32 is a fixed scaling factor, SA ∈ {0, 0.5, 1} is the actual outcome and EA is the expected outcome. This gives a convenient interpretation of rating differences: a gap of 150 points corresponds to a win probability of about 70% for the stronger player, and a gap of 400 points to ≈ 10-to-1 odds (91%). For our benchmark setup, this system is not appropriate because 1) methods, contrary to players, are static and do not improve over time; 2) at the time of evaluation, we have access to all "matches". Therefore, following Chatbot Arena (Chiang et al., 2024), we adopt a Bradley-Terry rating system, of which Elo is a special case. We keep the "Elo" name as we scale the ratings in such a way that they can be interpreted as the Elo ratings. In the Bradley-Terry model, we calculate the outcome of a pairwise comparison of the methods between each unique combination of downstream model, and budget with scores per combination averaged over seeds. Formally, the probability of method i being better than method j is calculated as: pi eβi 1 P (i > j) = = βi = (2) pi + pj e + e βj 1 + eβj −βi where pi = eβi is the score function proposed by the model’s authors and also the one we adopt in this benchmark. The score βi is in natural-log-odds units. We rescale them to the conventional Elo scale by ri = (βi − βRandom ) · ln400 10 + 1000, which anchors random feature selection at 1000 and preserves the standard Elo interpretation. Within each dataset, we define a setting as a combination of budget and downstream model, with the metric averaged over seeds. In each setting, every pair of methods forms a match, where win is assigned to a method with better score and ties are counted as half-wins. We estimate β by maximum likelihood on all matches. To quantify uncertainty, we use a cluster bootstrap over datasets: since all settings of a dataset share dataset-specific signal, their outcomes are correlated, so we resample the 71 datasets with replacement and refit the model on the matches of the sampled datasets. Over 1000 resamples, we report the 2.5th and 97.5th percentiles of each method’s rating ri as its 95% confidence interval. Results are shown in Table D.1. Pareto Details. Comparing metric errors across datasets of different task types can either be handled by ranking methods per dataset, as we did in the Elo calculation, or by normalizing the 39

Table D.1: Elo rating. We report the score per method along with 95% confidence intervals from 1000 cluster bootstrap resamples at the dataset level. Higher is better. Method

Elo rating

95% CI

1305.8 1262.4 1249.4 1245.8 1226.7 1224.1 1217.6 1195.7 1191.5 1181.5 1176.5 1167.4 1149.5 1000.0 965.4

[1259.6, 1356.2] [1214.3, 1314.2] [1203.0, 1301.6] [1207.2, 1290.1] [1184.1, 1272.2] [1180.0, 1274.8] [1179.8, 1262.8] [1145.4, 1250.5] [1150.6, 1237.5] [1146.1, 1222.1] [1136.9, 1226.4] [1119.5, 1224.8] [1106.0, 1197.7] [1000.0, 1000.0] [ 913.9, 1009.7]

RFImportance CART ElasticNet MI F-Test Lasso MarkovBlanket (R)ReliefF mRMR GainRatio SFS SBE LOCO Random LaplacianScore

errors by a baseline. We calculate and compare the improvement I(m,d) of a method m on a dataset d in % using the metric error of a method Err(m,d) and the metric error of random feature selection Err(r,d) . The errors are averaged across budgets and models. I(m,d) =

Err(r,d) − Err(m,d) · 100 Err(r,d)

(3)

The runtime per 1k features t1k is calculated by using the needed time of a feature selection method on a dataset, budget, and model, t(d,b,m) , and the number of features of a dataset n(d) : t1k =

t(d,b,m) · 1000 n(d)

(4)

By normalizing the time and the features per dataset, the comparison across datasets is possible and we obtain the results as shown in Table D.2. Table D.2: Predictive performance vs. runtime. We report the mean improvement over random and mean runtime per 1k features for all benchmarked feature selection methods. Name

Improvement over random (%)

Time per 1k features (s)

26.23 23.80 23.78 22.76 21.88 21.67 21.24 20.90 19.53 18.57 18.18 12.98 11.40 0.00 −18.58

45.19 51.24 15.71 0.79 2400.69 56.50 21.30 2370.68 14.26 24442.87 27903.08 6304.68 27018.73 0.006 4.60

RFImportance MI CART F-Test ElasticNet mRMR MarkovBlanket Lasso GainRatio (R)ReliefF SFS LOCO SBE Random LaplacianScore

Significance analysis. We perform the significance analysis using autorank (Herbold, 2020). We consider correlations at the level of budgets, downstream models, and seeds. We prepare the 40

autorank input so that each row represents an independent measurement and columns correspond to populations we want to compare. We use rank as a task-agnostic metric to compare feature selection methods across regression and classification datasets. First, we aggregate the results over three seeds and then we rank the methods for each combination of dataset, budget, and downstream model, assigning average rank to ties. Then, for each dataset and method, we average the rank over budgets and downstream models. Cells in the resulting matrix represent average rank of a method on a given dataset. To avoid relying on parametric assumptions, autorank applies the non-parametric Friedman test followed by the post-hoc Nemenyi test, that reported significant difference between methods’ mean rank if it exceeded the critical distance of 2.546. The resulting critical difference diagram (CDD) is shown in Figure D.1a: There are four groups of non-distinguishable methods linked with the horizontal bar. We report significant difference between five pairs: RFImportance and SBE, RFImportance and LOCO, RFImportance and (R)ReliefF, CART and LOCO, ElasticNet and LOCO. Additionally, all 13 methods significantly beat the random baseline and LaplacianScore. We additionally show CDDs for five cardinality indices in Figure D.1b-D.1f. For the lowest budget index (Figure D.1b), we observe one larger group of undistinguishable methods that is significantly better only than random baseline and LaplacianScore. As the budget grows, the number of groups increases and so does the rank spread. RFImportance consistently ranks in top-3 methods, CART ranks better in smaller budgets, while F-Test ranks better in bigger budgets CD 15 14 13 12 11 10

9

8

7

6

5

4

3

2

CD

1

15 14 13 12 11 10

LaplacianScore Random LOCO SBE (R)ReliefF GainRatio mRMR SFS

RFImportance CART ElasticNet Lasso MI F-test MarkovBlanket

6

5

4

3

2

1

CART MarkovBlanket RFImportance ElasticNet MI F-test SFS

CD 9

8

7

6

5

4

3

2

15 14 13 12 11 10

1

LaplacianScore Random LOCO SBE (R)ReliefF SFS mRMR GainRatio

CART RFImportance ElasticNet Lasso MarkovBlanket MI F-test

9

8

7

6

5

4

3

2

1

LaplacianScore Random LOCO SBE mRMR SFS (R)ReliefF GainRatio

(c) Budget idx j = 2. Mean number of features k=11.1.

RFImportance ElasticNet CART Lasso MI F-test MarkovBlanket

(d) Budget idx j = 3. Mean number of features k=27.4.

CD

LaplacianScore Random LOCO SBE SFS mRMR GainRatio MarkovBlanket

7

(b) Budget idx j = 1. Mean number of features k=4.7.

CD

15 14 13 12 11 10

8

LaplacianScore Random SBE LOCO (R)ReliefF GainRatio mRMR Lasso

(a) Aggregated over budgets.

15 14 13 12 11 10

9

CD 9

8

7

6

5

4

3

2

1

15 14 13 12 11 10

RFImportance Lasso MI CART F-test ElasticNet (R)ReliefF

LaplacianScore Random LOCO GainRatio SFS SBE MarkovBlanket (R)ReliefF

(e) Budget idx j = 4. Mean number of features k=87.3.

9

8

7

6

5

4

3

2

1

RFImportance F-test MI CART ElasticNet Lasso mRMR

(f) Budget idx j = 5. Mean number of features k=344.5.

Figure D.1: Critical difference diagrams. Average ranks per dataset with critical distance= 2.546. Tuning LGBM. We assess the impact of tuning the downstream model on predictive performance. We focus on LGBM, the model with the largest tunability (Erickson et al., 2025), via a random 41

search with 10 hyperparameter configurations. For each feature selection method, we pick the configuration with the lowest metric error averaged across budgets and datasets. The tuned model is reported alongside the other downstream models in Figure D.2. We use the per-task normalized score (Erickson et al., 2025), si,t = max(0, (ẽt − ei,t )/(ẽt − e⋆t )), where e⋆t and ẽt are the best and median errors on task t. The best method on each task scores 1, the median scores 0, and worse-than-median methods are floored at 0. On average, tuning improves the normalized score by 24.7% over the default configuration. This suggests that practitioners interested in boosting predictive performance on the downstream task should consider tuning their predictors in addition to applying feature selection. Model LGBM LGBM (Tuned) LM RF RealMLPModel TabICLv2

Normalized Score

0.5 0.4 0.3 0.2 0.1

(R)

E

SF S mR MR

SB

Re li

efF

CA Ela RT sti cN et F-T e Ga st inR ati o La L O pla CO cia nS co re La sso Ma rko MI vB RF lank Im e po t rta nc Ra e nd om

0.0

Figure D.2: Impact of tuning. Normalized scores per method per model, averaged over datasets and budgets. Tuned LGBM (striped blue) outperforms the default (solid blue) by a clear margin. Random fallback. In SelectArena, besides the cardinality constraint, we impose a time limit of one hour for each: the feature selection and the downstream model training and inference. This limit reflects the practical trade-off between predictive performance and computational cost. Improvements in downstream performance may be of limited value if they require prohibitively expensive feature selection or model evaluation. We use one hour as a common computational budget to capture this trade-off consistently across methods. Where supported by the implementation, this limit is enforced by interrupting the method; otherwise, the feature selection stage is allowed to complete but is recorded as a timeout if it exceeds the limit. When feature selection is interrupted, we apply a mechanism called random fallback to keep the cardinality fixed across runs. Specifically, random fallback either fills the remaining feature subset with randomly selected features or randomly subsamples the current subset down to the cardinality constraint. The latter applies to SBE and Markov Blanket, which start with the full feature set and iteratively remove features until the target cardinality is reached. This allows timed-out runs to remain part of the evaluation. Discarding timed-out runs would instead result in different methods being evaluated on different datasets, with slower methods disproportionately evaluated on lower-dimensional and potentially easier datasets. Random fallback therefore provides a practical mechanism for maintaining comparability under both cardinality and time constraints. Table D.3 summarizes the timeout and random fallback rates. Timeouts are frequent for the search methods SBE (46.9%) and SFS (44.8%), less frequent for LOCO (23.5%) and Relief(F) (17.0%), and rare for Lasso (4.0%), mRMR (3.4%), and ElasticNet (1.4%). All other methods never hit the time limit. The discrepancy between the timeout and fallback rates has two causes. First, for SBE, SFS, LOCO, and Relief(F) some runs produce a complete set of features (no fallback) in a final step that starts within the time limit but ends after it. Second, Lasso and ElasticNet always run to completion as their scikit-learn interface does not allow interruption. Given their low timeout rates, we accept this as a minor deviation from our time-constraint framing of the benchmark. To measure the impact of random fallback on predictive performance, we rerun experiments without it on a single seed (5 budget indices × 5 downstream models × 71 datasets) and present the results in Table D.3. As expected, the ∆ (difference between the improvement over Random for no fallback and fallback cases) is largest for methods that trigger the fallback most often, i.e. SBE, SFS, and LOCO. Without fallback, SBE returns feature subsets larger than the cardinality constraint when it reaches 42

Table D.3: Impact of random fallback on run completion and improvement over R ANDOM. A run is a unique combination of dataset, budget, and downstream model; each method has 1,775 runs. Timeout counts runs exceeding the one-hour limit. Fallback counts runs completed or reduced through random selection after a timeout. Improvement over R ANDOM (Apppendix D.2) normalizes performance across tasks with different error metrics. We report the mean for the default pipeline (Fallback) and the ablated one (No fallback), with ∆ = No fallback − Fallback.

Method SBE SFS LOCO (R)ReliefF Lasso mRMR ElasticNet CART F-Test GainRatio LaplacianScore MI MarkovBlanket RFImportance Random† Total †

Timeout

Fallback

n

%

n

832 795 417 301 71 60 25 0 0 0 0 0 0 0 0

46.9 44.8 23.5 17.0 4.0 3.4 1.4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

2,501

Improvement over R ANDOM

% Fallback No fallback

825 46.5 778 43.8 134 7.5 0 0.0 0 0.0 60 3.4 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0

9.4 1,797

10.01 16.96 11.58 19.23 20.95 21.97 22.91 24.21 23.71 19.27 −18.40 23.32 21.08 25.99 –

6.8

∆ (pp)

28.06 +18.05 −1.99 −18.95 10.11 −1.47 19.22 −0.01 20.95 +0.00 21.96 −0.01 22.91 +0.00 24.21 +0.00 23.71 +0.00 19.28 +0.01 −18.40 +0.00 23.31 −0.01 21.10 +0.02 25.99 +0.00 – –

–

–

–

R ANDOM selects features at random by construction, so it is not counted as a fallback.

with random fallback

1300

no fallback

1250 1200

Elo

1150 1100 1050 1000 950 900

core

anS

ci apla

L

15 15 14 14 12 13

om LOCO

d Ran

1 12 11 11 10 10

9 9

* F o R SBE inRati )Relief mRM a R ( G

13 8

8 7

7 6

SFS lanket F-test vB arko

M

6 5

5 4

MI

et so Las lasticN E

4 3

3 2

2 1

T e CAR ortanc p RFIm

Figure D.3: Impact of random fallback on Elo ranking. New Elo ratings are calculated based on the ablation study’s results for a single seed and are presented in orange. SBE is shown in striped red to indicate that excluding random fallback has a different effect on it than other methods. Default Elo ratings are presented in blue. Numbers on the bars represent rank.

the time limit, resulting in a positive ∆ = +18.05pp. However, this reflects a budget violation rather than a real gain and is not comparable to the impact of the fallback on other methods. When SFS and LOCO time out, they output less features than the cardinality constraint assumes, resulting in a negative ∆ = −18.95pp and − 1.47pp. Without fallback, SFS even falls below Random as its search iterations are relatively long (Table D.2). As in Guyon & Elisseeff (2003), we observe that randomly selected features still carry useful signal and improve downstream model’s performance compared to passing it an incomplete feature subset. For mRMR and methods that do resort to random fallback, ∆ ≤ 0.02pp, which is attributable to run-to-run nondeterminism. 43

Figure D.3 presents the change in the Elo ranking when random fallback is removed. Without random fallback, SBE rises from rank 12 to rank 1 and SFS falls from rank 8 to rank 13, with both these shifts affecting all the other ranks as Elo rating is relative. We note, again, that SBE rise is an artifact of the random fallback’s impact on backward search methods and it reflects budget violation as described above. That is why the only meaningful change is the drop in the SFS rank. D.3

S TABILITY

We assess the stability of the feature selection methods across datasets with a different number of samples (in the minority class) per feature, i.e., events per variable (EPV). We use the EPV as a proxy for the selection difficulty (Harrell, 2015; Heinze et al., 2018). F-Test and Distance-Based Methods Dominate Stability Rankings. The dominance of F-Test and LaplacianScore is visible in Figure 5 and Figure D.4, where we additionally highlight the top-3 performing methods. This confirms that F-Test, LaplacianScore, MI, GainRatio, (R)ReliefF, and mRMR consistently rank highest in stability across EPVs. In Figure D.5, we isolate the near-optimal set of methods that are within 3% of the best-performing method. This confirms the dominance of the aforementioned 5 methods. F-Test Works Well Across Selection Difficulties. The F-Test often performs well at high and medium EPVs. Importantly, it also ranks highest at very low EPVs, where other methods struggle. In this setting, other methods are rarely within its 3% threshold. Lower EPV Reveals Robustness. Comparing the different selection difficulties, one can observe a broader diversity of methods achieving competitive stability at high and medium EPVs. Conversely, at very low EPVs, only few methods are relatively stable. Most Methods Show Intermediate to Good Stability. In Table D.4, we report the mean stability for each method. Poor stability is defined by values below 0.40, intermediate to good stability by values between 0.40 and 0.75, and values above 0.75 indicate excellent stability (Nogueira & Brown, 2016). Lasso, SBE, SFS, and LOCO only achieve low stability, showing the impact of the high number of timeouts for SBE (46.41%), SFS (45.28%), and LOCO (23.10%). The remaining methods show an intermediate to good stability.

44

F-Test LaplacianScore MI GainRatio (R)ReliefF mRMR RFImportance ElasticNet MarkovBlanket CART Lasso SFS SBE LOCO Random 2799.7

stable 1.0 0.8 0.6 0.4 0.2 0.0 unstable 217.4

15.1

3.4 0.2 Events per Variable

0.0

Figure D.4: Stability. Nogueira score (Nogueira & Brown, 2016) across datasets. We use the events per variable, i.e., the number of samples per feature for regression problems or the number of samples in the minority class per feature for classification problems, as a proxy for the selection difficulty (Harrell, 2015; Heinze et al., 2018). The top-3 methods per EPV are highlighted.

F-Test LaplacianScore MI GainRatio (R)ReliefF mRMR RFImportance ElasticNet MarkovBlanket CART Lasso SFS SBE LOCO Random 2799.7

217.4

15.1

3.4 0.2 Events per Variable

Within Threshold Outside Threshold 0.0

Figure D.5: Thresholded stability. We display all methods within a 3% performance threshold of the best-performing method.

45

D.4

VALIDITY

Like the stability, we assess the validity across different events per variable (EPV) (Appendix D.3). Additionally, we use three different noise levels relative to the original feature set size: a 50% noise level indicates that a number of noise features equal to 50% of the original feature count has been injected into the dataset. This proportional scaling applies consistently to the 75% and 100% noise levels. The 50% noise level results in one third of the features to be noise, the 100% noise level results in half of the feature to be noise. An increasing noise level represents a higher difficulty for feature selection methods to make a valid selection, because the amount of irrelevant features increases. One common strategy is to use feature removal approaches like RemOve And Retrain (ROAR), but these methods can be sensitive to feature redundancies that are prevalent in real-world data (Hooker et al., 2019). Therefore, we adopt an alternative approach to measure validity described in Chapter 3.3. F-Test and Information-Theoretic Methods Dominate Validity Rankings. In Figure 6, we can see that F-Test and information-theoretic methods dominate the validity rankings, followed by LaplacianScore. This finding is consistent across different noise levels (Figures D.6, D.7, D.8, D.9, D.10, D.11). More Noise Injection Reveals Robustness. In contrast to the stability, we can observe a higher resilience to lower EPVs for the validity. However, Figures D.6, D.8, and D.10 illustrate the challenge of increasing noise levels, as visualized by the progressively lighter coloring across all methods. This degradation pattern is quantified in Table D.4, where every method exhibits a consistent score decline with increasing noise levels. Dominant Methods Show Greater Noise Resilience. The F-Test’s validity score decreases steadily from 0.92 and 0.89 to 0.85 — a modest drop of < 4% per noise increment. In contrast, MarkovBlanket experiences substantially larger declines of 9% per step, indicating the superior noise robustness of top-performing methods. In fact, the method ranking only changes slightly at different noise levels (Figures D.6, D.7, D.8, D.9, D.10, D.11). More Noise Injection Accentuates Differences. We can observe that with increasing noise levels, the number of low-ranked methods achieving a performance within a threshold of 3% of the best method decreases (Figures D.7, D.9, D.11). This implies that the increasing difficulty emphasizes differences between high- and low-ranking methods. Table D.4: Mean validity and stability. We report the average scores across all datasets. Higher is better. Method

Stability

Validity (50%)

Validity (75%)

Validity (100%)

F-Test LaplacianScore MI GainRatio (R)ReliefF mRMR RFImportance ElasticNet MarkovBlanket CART Lasso SFS SBE LOCO Random

0.679027 0.656319 0.653849 0.648343 0.602695 0.586207 0.543465 0.442266 0.437499 0.427232 0.374202 0.276485 0.235075 0.172724 0.000598

0.922444 0.875447 0.879216 0.909595 0.836929 0.749956 0.769910 0.853544 0.621839 0.708525 0.839766 0.781449 0.726402 0.730326 0.675138

0.885531 0.837405 0.831729 0.867829 0.775536 0.698682 0.718201 0.804436 0.532242 0.650755 0.778133 0.702861 0.634586 0.639994 0.586824

0.848069 0.797492 0.785338 0.826548 0.740539 0.654502 0.672429 0.759485 0.460249 0.588315 0.731577 0.626375 0.558420 0.583660 0.499406

46

F-Test GainRatio MI LaplacianScore ElasticNet Lasso (R)ReliefF SFS RFImportance mRMR LOCO SBE CART Random MarkovBlanket 2799.7

valid 1.0 0.8 0.6 0.4 0.2 0.0 invalid 40.0

6.3 0.7 Events per Variable

0.1

0.0

Figure D.6: Validity at 50% added noise. Selection precision across datasets at a noise level of 50%. Like for the stability, we use the events per variable, i.e., the number of samples per feature for regression problems or the number of samples in the minority class per feature for classification problems, as a proxy for the selection difficulty (Harrell, 2015; Heinze et al., 2018). The top-3 methods per EPV are highlighted.

F-Test GainRatio MI LaplacianScore ElasticNet Lasso (R)ReliefF SFS RFImportance mRMR LOCO SBE CART Random MarkovBlanket 2799.7

40.0

6.3 0.7 Events per Variable

0.1

Within Threshold Outside Threshold 0.0

Figure D.7: Thresholded validity at 50% added noise. We display all methods within a 3% performance threshold of the best-performing method at a noise level of 50%.

47

F-Test GainRatio LaplacianScore MI ElasticNet Lasso (R)ReliefF RFImportance SFS mRMR CART LOCO SBE Random MarkovBlanket 2799.7

valid 1.0 0.8 0.6 0.4 0.2 0.0 invalid 40.0

6.3 0.7 Events per Variable

0.1

0.0

Figure D.8: Validity at 75% added noise. Selection precision across datasets at a noise level of 75%. Like for the stability, we use the events per variable, i.e., the number of samples per feature for regression problems or the number of samples in the minority class per feature for classification problems, as a proxy for the selection difficulty (Harrell, 2015; Heinze et al., 2018). The top-3 methods per EPV are highlighted.

F-Test GainRatio LaplacianScore MI ElasticNet Lasso (R)ReliefF RFImportance SFS mRMR CART LOCO SBE Random MarkovBlanket 2799.7

40.0

6.3 0.7 Events per Variable

0.1

Within Threshold Outside Threshold 0.0

Figure D.9: Thresholded validity at 75% added noise. We display all methods within a 3% performance threshold of the best-performing method at a noise level of 75%.

48

F-Test GainRatio LaplacianScore MI ElasticNet (R)ReliefF Lasso RFImportance mRMR SFS CART LOCO SBE Random MarkovBlanket 2799.7

valid 1.0 0.8 0.6 0.4 0.2 0.0 invalid 40.0

6.3 0.7 Events per Variable

0.1

0.0

Figure D.10: Validity at 100% added noise. Selection precision across datasets at a noise level of 100%. Like for the stability, we use the events per variable, i.e., the number of samples per feature for regression problems or the number of samples in the minority class per feature for classification problems, as a proxy for the selection difficulty (Harrell, 2015; Heinze et al., 2018). The top-3 methods per EPV are highlighted.

F-Test GainRatio LaplacianScore MI ElasticNet (R)ReliefF Lasso RFImportance mRMR SFS CART LOCO SBE Random MarkovBlanket 2799.7

40.0

6.3 0.7 Events per Variable

0.1

Within Threshold Outside Threshold 0.0

Figure D.11: Thresholded validity at 100% added noise. We display all methods within a 3% performance threshold of the best-performing method at a noise level of 100%.

49

D.5

R AW P REDICTIVE P ERFORMANCE

In the following, we show the mean and standard deviation of the metric error per method across datasets and budgets aggregated over three seeds. We break down the tables per downstream model for better legibility. Table D.5 shows the performance for LGBM, Table D.6 for Linear model, Table D.7 for RF, Table D.8 for RealMLP, and Table D.9 for TabICLv2.

CART

ElasticNet

F-test

GainRatio

LaplacianScore

Lasso

LOCO

MarkovBlanket

MI

mRMR

(R)ReliefF

Random

SBE

RFImportance

SFS

.759 ±.028

.834 ±.065

.879 ±.027

.762 ±.037

.784 ±.075

.779 ±.031

.823 ±.066

.807 ±.021

.809 ±.042

.863 ±.073

.821 ±.081

.760 ±.037

.894 ±.012

2

.739 ±.043

.748 ±.031

.805 ±.034

.889 ±.029

.738 ±.036

.763 ±.051

.761 ±.042

.796 ±.063

.790 ±.026

.767 ±.045

.802 ±.042

.774 ±.043

.739 ±.059

.887 ±.011

3

.746 ±.037

.742 ±.033

.728 ±.049

.773 ±.013

.886 ±.022

.737 ±.038

.760 ±.054

.743 ±.043

.777 ±.038

.764 ±.015

.749 ±.035

.776 ±.083

.748 ±.049

.727 ±.057

.886 ±.016

4

.730 ±.046

.719 ±.043

.723 ±.045

.731 ±.048

.793 ±.008

.720 ±.046

.757 ±.051

.730 ±.054

.746 ±.036

.732 ±.035

.743 ±.037

.759 ±.048

.737 ±.050

.718 ±.056

.852 ±.016

5

.725 ±.042

.725 ±.035

.719 ±.043

.725 ±.042

.752 ±.021

.722 ±.039

.759 ±.044

.731 ±.041

.743 ±.034

.716 ±.050

.735 ±.038

.751 ±.039

.737 ±.050

.725 ±.047

.770 ±.030

1

.698 ±.234

.292 ±.099

.521 ±.176

.556 ±.306

1.36 ±.09

.368 ±.237

1.19 ±.10

.635 ±.176

.550 ±.219

.532 ±.256

.366 ±.102

1.00 ±.37

1.09 ±.38

.405 ±.166

1.16 ±.22

2

.512 ±.184

.147 ±.091

.198 ±.085

.269 ±.180

1.06 ±.08

.149 ±.139

.799 ±.127

.207 ±.106

.229 ±.210

.211 ±.106

.103 ±.031

1.13 ±.21

.969 ±.158

.0764 ±.0675

.933 ±.316

3

.367 ±.118

.113 ±.104

.106 ±.111

.125 ±.090

.900 ±.093

.0943 ±.0808

.544 ±.183

.125 ±.094

.136 ±.087

.123 ±.095

.0935 ±.0853

.458 ±.194

.536 ±.215

.104 ±.056

.505 ±.158

4

.224 ±.156

.119 ±.093

.115 ±.096

.115 ±.069

.519 ±.164

.0980 ±.0736

.219 ±.071

.121 ±.077

.129 ±.087

.113 ±.073

.0931 ±.0818

.340 ±.061

.390 ±.084

.0954 ±.0653

.288 ±.295

5

.183 ±.108

.128 ±.082

.116 ±.081

.114 ±.069

.0963 ±.0578

.106 ±.070

.133 ±.022

.118 ±.064

.128 ±.081

.115 ±.070

.110 ±.066

.124 ±.013

.145 ±.016

.104 ±.063

.251 ±.147

1

.0968 ±.0004

.113 ±.015

.141 ±.001

.142 ±.003

.719 ±.000

.242 ±.252

.0973 ±.0011

.0973 ±.0011

.146 ±.001

.157 ±.002

.159 ±.003

.555 ±.409

.0968 ±.0004

.142 ±.003

.146 ±.001

2

.0740 ±.0010

.0762 ±.0032

.0942 ±.0106

.0754 ±.0009

.528 ±.142

.0746 ±.0018

.0739 ±.0010

.0774 ±.0063

.0829 ±.0010

.153 ±.003

.0987 ±.0031

.102 ±.008

.0741 ±.0011

.0751 ±.0011

.0983 ±.0014

3

.0795 ±.0012

.0718 ±.0009

.0869 ±.0145

.0795 ±.0012

.439 ±.023

.0718 ±.0009

.0722 ±.0007

.0795 ±.0012

.0795 ±.0012

.0939 ±.0014

.0978 ±.0115

.390 ±.038

.0722 ±.0008

.0768 ±.0051

.0809 ±.0011

4

.0787 ±.0011

.0717 ±.0007

.0866 ±.0028

.0787 ±.0012

.188 ±.146

.0760 ±.0079

.0743 ±.0049

.0787 ±.0012

.0845 ±.0011

.0912 ±.0013

.0845 ±.0012

.307 ±.181

.0720 ±.0008

.0787 ±.0012

.0788 ±.0011

5

.0841 ±.0011

.0835 ±.0013

.0837 ±.0016

.0841 ±.0011

.0844 ±.0056

.0834 ±.0013

.0786 ±.0012

.0823 ±.0040

.0841 ±.0011

.0845 ±.0011

.0843 ±.0011

.187 ±.189

.0786 ±.0012

.0841 ±.0011

.0786 ±.0012

1

.634 ±.035

.595 ±.018

.652 ±.013

.672 ±.007

.975 ±.017

.600 ±.019

.617 ±.036

.603 ±.007

.640 ±.010

.682 ±.023

.614 ±.016

.693 ±.034

.871 ±.029

.674 ±.009

.663 ±.010

2

.579 ±.016

.574 ±.018

.576 ±.018

.602 ±.006

.882 ±.031

.574 ±.021

.583 ±.013

.594 ±.012

.570 ±.016

.633 ±.015

.582 ±.012

.640 ±.023

.640 ±.053

.581 ±.012

.615 ±.028

3

.565 ±.014

.560 ±.018

.570 ±.027

.596 ±.012

.680 ±.030

.562 ±.013

.564 ±.021

.596 ±.006

.559 ±.017

.602 ±.013

.565 ±.024

.660 ±.019

.599 ±.045

.567 ±.015

.581 ±.015

4

.560 ±.017

.558 ±.017

.562 ±.018

.590 ±.005

.599 ±.014

.558 ±.012

.556 ±.018

.591 ±.005

.561 ±.015

.596 ±.014

.560 ±.024

.594 ±.016

.558 ±.015

.555 ±.014

.562 ±.017

5

.553 ±.015

.554 ±.016

.554 ±.019

.575 ±.027

.554 ±.016

.555 ±.016

.557 ±.018

.587 ±.008

.556 ±.014

.564 ±.016

.555 ±.017

.559 ±.024

.552 ±.013

.553 ±.016

.556 ±.013

1

1.74 ±.04

1.81 ±.05

1.82 ±.05

2.04 ±.14

2.12 ±.01

1.91 ±.17

1.81 ±.09

1.76 ±.06

1.75 ±.04

1.76 ±.03

1.78 ±.06

2.04 ±.12

2.08 ±.03

1.75 ±.05

1.83 ±.06

2

1.75 ±.03

1.78 ±.07

1.79 ±.08

1.85 ±.14

2.12 ±.01

1.92 ±.17

1.79 ±.07

1.78 ±.05

1.71 ±.01

1.76 ±.03

1.78 ±.06

2.11 ±.01

2.10 ±.04

1.77 ±.03

1.85 ±.08

3

1.74 ±.05

1.81 ±.08

1.79 ±.06

1.80 ±.03

2.11 ±.01

1.78 ±.07

1.81 ±.05

1.78 ±.05

1.70 ±.06

1.77 ±.05

1.79 ±.10

2.11 ±.01

2.06 ±.10

1.77 ±.05

1.82 ±.01

4

1.77 ±.03

1.78 ±.07

1.81 ±.07

1.76 ±.03

2.03 ±.01

1.78 ±.08

1.76 ±.05

1.78 ±.02

1.71 ±.02

1.77 ±.03

1.80 ±.09

1.95 ±.19

1.96 ±.05

1.71 ±.03

1.87 ±.03

5

1.70 ±.03

1.82 ±.05

1.71 ±.01

1.80 ±.03

1.79 ±.06

1.76 ±.08

1.73 ±.02

1.79 ±.03

1.72 ±.01

1.79 ±.02

1.79 ±.06

1.99 ±.14

1.78 ±.12

1.75 ±.01

1.85 ±.02

1

.711 ±.005

.783 ±.025

.909 ±.016

1.18 ±.00

.909 ±.024

.856 ±.004

.966 ±.086

.719 ±.007

.925 ±.015

.825 ±.005

.940 ±.033

1.12 ±.11

1.11 ±.08

.718 ±.006

1.12 ±.15

2

.561 ±.008

.591 ±.010

.660 ±.014

.862 ±.007

.649 ±.016

.612 ±.017

.735 ±.036

.575 ±.006

.646 ±.009

.637 ±.009

.638 ±.003

.833 ±.014

.800 ±.026

.568 ±.010

.807 ±.085

3

.508 ±.006

.523 ±.005

.559 ±.007

.604 ±.005

.532 ±.008

.527 ±.002

.670 ±.073

.517 ±.012

.573 ±.006

.560 ±.007

.553 ±.008

.645 ±.021

.636 ±.033

.520 ±.011

.644 ±.034

4

.486 ±.007

.492 ±.005

.512 ±.003

.540 ±.005

.500 ±.005

.493 ±.007

.584 ±.017

.495 ±.007

.509 ±.006

.507 ±.007

.511 ±.006

.570 ±.018

.571 ±.035

.492 ±.008

.583 ±.037

5

.479 ±.007

.482 ±.008

.486 ±.009

.491 ±.008

.484 ±.007

.482 ±.009

.511 ±.013

.481 ±.007

.487 ±.009

.486 ±.005

.486 ±.009

.510 ±.022

.528 ±.011

.480 ±.009

.520 ±.020

1

.438 ±.245

.438 ±.080

.527 ±.102

.290 ±.131

.868 ±.065

.501 ±.121

.716 ±.060

.354 ±.059

.418 ±.022

.377 ±.185

.379 ±.010

.754 ±.095

.739 ±.054

.339 ±.106

.688 ±.123

2

.390 ±.206

.241 ±.081

.374 ±.063

.266 ±.080

.851 ±.064

.337 ±.135

.413 ±.093

.263 ±.086

.265 ±.100

.227 ±.037

.284 ±.053

.507 ±.131

.404 ±.059

.200 ±.073

.427 ±.138

3

.288 ±.070

.212 ±.091

.265 ±.041

.250 ±.041

.719 ±.136

.220 ±.057

.301 ±.002

.248 ±.071

.227 ±.011

.232 ±.026

.219 ±.043

.257 ±.056

.261 ±.061

.192 ±.044

.274 ±.066

4

Dataset

.762 ±.037

.753 ±.046

.238 ±.030

.209 ±.050

.215 ±.041

.203 ±.029

.590 ±.059

.203 ±.033

.254 ±.088

.223 ±.038

.204 ±.020

.199 ±.048

.246 ±.046

.197 ±.078

.207 ±.073

.209 ±.014

.198 ±.054

.237 ±.008

.194 ±.046

.203 ±.030

.198 ±.030

.271 ±.113

.212 ±.022

.207 ±.013

.193 ±.032

.204 ±.033

.202 ±.043

.206 ±.029

.196 ±.026

.189 ±.023

.208 ±.016

.185 ±.011

Budget idx j 1

.778 ±.020

5

Table D.5: Per-cell results with downstream model LGBM. Tasks are grouped by metric. Each cell is the mean over three seeds, with the sample standard deviation as a subscript.

1

1.74 ±.19

1.15 ±.22

.971 ±.089

1.60 ±.32

1.16 ±.12

2.04 ±.18

1.12 ±.22

1.13 ±.28

.788 ±.207

2

1.01 ±.44

1.34 ±.19

.834 ±.055

.662 ±.083

.961 ±.094

.994 ±.020

1.40 ±.20

1.11 ±.20

.889 ±.249

.707 ±.017

.627 ±.139

3

.495 ±.036

.690 ±.223

1.04 ±.25

.747 ±.102

.566 ±.082

.472 ±.078

.704 ±.023

.885 ±.202

.642 ±.066

.478 ±.122

.593 ±.091

.537 ±.113

4

.520 ±.100

.621 ±.055

.623 ±.204

.441 ±.141

.543 ±.173

.680 ±.103

.633 ±.031

.453 ±.145

.462 ±.093

.537 ±.048

.682 ±.168

.534 ±.017

.477 ±.146

.458 ±.117

.462 ±.171

5

.495 ±.125

.477 ±.121

.508 ±.107

.450 ±.165

.453 ±.155

.600 ±.052

.522 ±.090

.448 ±.168

.479 ±.117

.450 ±.140

.503 ±.097

.500 ±.121

.480 ±.143

.441 ±.163

.449 ±.145

1

.551 ±.001

.574 ±.007

.541 ±.024

.556 ±.009

.695 ±.014

.546 ±.046

.635 ±.066

.520 ±.023

.553 ±.016

.562 ±.024

.554 ±.012

.698 ±.007

.696 ±.006

.548 ±.029

.534 ±.019

2

.546 ±.018

.551 ±.020

.531 ±.006

.558 ±.008

.691 ±.013

.549 ±.023

.624 ±.062

.517 ±.024

.555 ±.009

.539 ±.025

.567 ±.009

.690 ±.008

.661 ±.025

.527 ±.006

.544 ±.026

3

.542 ±.011

.531 ±.019

.543 ±.013

.536 ±.022

.685 ±.019

.525 ±.012

.581 ±.053

.537 ±.028

.524 ±.002

.522 ±.011

.573 ±.010

.584 ±.020

.618 ±.048

.525 ±.024

.549 ±.037

4

.524 ±.020

.520 ±.017

.538 ±.006

.524 ±.006

.667 ±.015

.520 ±.020

.550 ±.028

.522 ±.016

.518 ±.033

.528 ±.009

.550 ±.002

.562 ±.079

.540 ±.027

.524 ±.018

.537 ±.029

5

.511 ±.022

.512 ±.015

.507 ±.016

.520 ±.008

.634 ±.012

.515 ±.020

.501 ±.016

.517 ±.002

.499 ±.025

.519 ±.015

.516 ±.023

.537 ±.029

.527 ±.015

.506 ±.018

.516 ±.027

1

1.77 ±.14

1.91 ±.13

1.95 ±.09

1.77 ±.11

2.13 ±.06

1.90 ±.13

1.84 ±.13

1.88 ±.15

1.77 ±.14

1.83 ±.13

1.95 ±.09

1.94 ±.07

1.81 ±.14

1.83 ±.13

1.83 ±.12

2

1.67 ±.18

1.79 ±.21

1.73 ±.12

1.64 ±.22

1.97 ±.08

1.79 ±.19

1.80 ±.17

1.69 ±.21

1.55 ±.19

1.64 ±.22

1.71 ±.14

1.88 ±.18

1.75 ±.20

1.70 ±.15

1.78 ±.21

3

1.50 ±.22

1.70 ±.19

1.61 ±.12

1.57 ±.20

1.86 ±.09

1.67 ±.13

1.58 ±.13

1.56 ±.16

1.47 ±.21

1.57 ±.20

1.64 ±.12

1.70 ±.19

1.65 ±.17

1.50 ±.20

1.63 ±.23

4

1.43 ±.19

1.62 ±.16

1.49 ±.15

1.51 ±.24

1.69 ±.12

1.55 ±.20

1.46 ±.15

1.49 ±.21

1.44 ±.19

1.51 ±.24

1.55 ±.14

1.57 ±.06

1.46 ±.14

1.41 ±.17

1.51 ±.18

5

1.41 ±.17

1.50 ±.18

1.47 ±.13

1.46 ±.19

1.54 ±.14

1.45 ±.17

1.41 ±.15

1.46 ±.21

1.41 ±.18

1.46 ±.19

1.50 ±.18

1.49 ±.24

1.43 ±.17

1.41 ±.17

1.43 ±.16

1

horse_colic survival srbct prediction sdss_17 predict_students_dropout and_academic_success ancestry study otto_group_product classification_challenge lung_cancer soybean large

1.86 ±.25

.888 ±.249

.810 ±.180

.0709 ±.0278

.131 ±.056

.242 ±.020

.104 ±.019

.772 ±.072

.132 ±.041

.117 ±.011

.123 ±.033

.142 ±.008

.123 ±.042

.135 ±.017

.616 ±.101

.102 ±.045

.109 ±.048

.0900 ±.0279

2

mic

1.98 ±.32

1.02 ±.09

.890 ±.112

.0753 ±.0257

.0751 ±.0429

.107 ±.066

.0911 ±.0235

.607 ±.034

.0633 ±.0400

.0853 ±.0448

.0944 ±.0200

.0947 ±.0397

.0907 ±.0228

.0736 ±.0134

.253 ±.111

.0911 ±.0343

.0743 ±.0429

.0674 ±.0309

3

ljubljana primary_tumor

1.83 ±.21

1.25 ±.07

.623 ±.061

.0823 ±.0237

.0737 ±.0388

.0559 ±.0169

.0933 ±.0244

.381 ±.064

.0765 ±.0393

.0833 ±.0421

.0818 ±.0339

.0826 ±.0360

.0933 ±.0244

.0784 ±.0420

.149 ±.059

.0881 ±.0266

.0767 ±.0434

.0667 ±.0311

4

anneal

1.29 ±.14 .758 ±.057

.0817 ±.0311

.0752 ±.0377

.0622 ±.0311

.100 ±.036

.250 ±.039

.0756 ±.0386

.0843 ±.0423

.0796 ±.0428

.0796 ±.0420

.100 ±.036

.0651 ±.0358

.173 ±.103

.0871 ±.0269

.0746 ±.0425

.0670 ±.0285

5

log-loss (↓)

1.97 ±.05

.0770 ±.0452

.0769 ±.0444

.0783 ±.0387

.0940 ±.0341

.0481 ±.0065

.0751 ±.0425

.0837 ±.0421

.0804 ±.0424

.0788 ±.0434

.0940 ±.0341

.0722 ±.0425

.0767 ±.0413

.0867 ±.0273

.0745 ±.0424

.0646 ±.0271

50

1.68 ±.05

CART

ElasticNet

F-test

GainRatio

LaplacianScore

Lasso

LOCO

MarkovBlanket

MI

mRMR

(R)ReliefF

Random

SBE

RFImportance

SFS

.504 ±.019

.472 ±.138

.565 ±.067

.540 ±.006

.560 ±.067

.472 ±.138

.435 ±.144

.551 ±.049

.475 ±.039

.585 ±.064

.511 ±.096

.417 ±.159

.471 ±.134

2

.518 ±.015

.472 ±.037

.470 ±.134

.483 ±.078

.523 ±.018

.560 ±.067

.463 ±.148

.417 ±.163

.520 ±.074

.445 ±.050

.551 ±.134

.499 ±.080

.416 ±.155

.460 ±.117

3

.440 ±.153

.489 ±.040

.439 ±.050

.470 ±.133

.436 ±.102

.495 ±.030

.551 ±.075

.454 ±.154

.420 ±.161

.448 ±.101

.408 ±.107

.431 ±.094

.517 ±.104

.416 ±.155

.460 ±.117

4

.432 ±.139

.438 ±.088

.408 ±.143

.444 ±.131

.440 ±.105

.416 ±.102

.551 ±.075

.418 ±.162

.420 ±.161

.424 ±.126

.409 ±.117

.397 ±.126

.517 ±.104

.416 ±.155

.464 ±.123

5

.432 ±.139

.432 ±.099

.408 ±.143

.416 ±.154

.437 ±.105

.402 ±.122

.548 ±.070

.416 ±.154

.416 ±.155

.417 ±.135

.399 ±.125

.425 ±.144

.506 ±.089

.416 ±.155

.466 ±.127

1

1.34 ±.02

1.42 ±.03

1.39 ±.03

1.53 ±.02

1.40 ±.03

1.44 ±.03

1.46 ±.05

1.53 ±.01

1.36 ±.05

1.53 ±.02

1.49 ±.01

1.53 ±.01

1.53 ±.01

1.33 ±.02

1.37 ±.02

2

1.23 ±.01

1.27 ±.04

1.31 ±.03

1.50 ±.01

1.32 ±.00

1.26 ±.04

1.36 ±.06

1.47 ±.05

1.25 ±.02

1.50 ±.01

1.34 ±.06

1.50 ±.02

1.50 ±.04

1.21 ±.01

1.33 ±.04

3

1.00 ±.02

1.05 ±.01

1.05 ±.02

1.37 ±.03

1.17 ±.02

1.03 ±.02

1.17 ±.02

1.33 ±.04

1.02 ±.01

1.37 ±.03

1.04 ±.02

1.30 ±.03

1.29 ±.03

.977 ±.028

1.26 ±.04

4

.820 ±.003

.844 ±.030

.844 ±.015

1.03 ±.05

.941 ±.003

.838 ±.022

.962 ±.021

1.01 ±.03

.845 ±.010

1.03 ±.05

.815 ±.011

.976 ±.058

1.01 ±.08

.801 ±.021

.990 ±.026

5

.769 ±.014

.777 ±.023

.773 ±.014

.806 ±.033

.779 ±.002

.778 ±.021

.814 ±.023

.807 ±.034

.767 ±.015

.806 ±.033

.753 ±.022

.802 ±.027

.794 ±.024

.758 ±.013

.802 ±.032

1

1.41 ±.04

1.40 ±.02

1.44 ±.05

1.42 ±.05

1.40 ±.03

1.40 ±.02

1.40 ±.03

1.42 ±.03

1.41 ±.03

1.41 ±.03

1.43 ±.06

1.40 ±.03

1.40 ±.03

1.42 ±.04

1.40 ±.03

2

1.41 ±.02

1.41 ±.03

1.43 ±.05

1.41 ±.03

1.41 ±.04

1.41 ±.03

1.43 ±.04

1.41 ±.03

1.42 ±.05

1.42 ±.03

1.41 ±.04

1.41 ±.02

1.42 ±.05

1.42 ±.04

1.41 ±.03

3

1.41 ±.03

1.42 ±.03

1.42 ±.03

1.41 ±.03

1.41 ±.04

1.42 ±.03

1.42 ±.03

1.40 ±.02

1.41 ±.04

1.42 ±.04

1.42 ±.04

1.42 ±.03

1.41 ±.02

1.41 ±.03

1.42 ±.05

4

1.41 ±.03

1.41 ±.02

1.41 ±.02

1.41 ±.03

1.41 ±.03

1.41 ±.02

1.41 ±.02

1.42 ±.04

1.41 ±.03

1.42 ±.03

1.42 ±.04

1.42 ±.04

1.41 ±.03

1.41 ±.03

1.42 ±.05

5

1.41 ±.03

1.41 ±.02

1.41 ±.03

1.41 ±.02

1.41 ±.03

1.41 ±.02

1.41 ±.03

1.41 ±.02

1.41 ±.02

1.41 ±.02

1.41 ±.03

1.41 ±.03

1.41 ±.03

1.41 ±.02

1.41 ±.03

1

.667 ±.001

.680 ±.002

.675 ±.002

.705 ±.010

.811 ±.002

.680 ±.002

.669 ±.001

.693 ±.000

.670 ±.002

.716 ±.013

.695 ±.003

.783 ±.024

.785 ±.023

.667 ±.001

.661 ±.001

2

.637 ±.003

.651 ±.002

.658 ±.001

.648 ±.003

.808 ±.003

.651 ±.002

.633 ±.006

.637 ±.001

.646 ±.001

.660 ±.014

.650 ±.012

.784 ±.016

.793 ±.021

.639 ±.004

.644 ±.005

3

.587 ±.001

.603 ±.004

.642 ±.001

.600 ±.011

.807 ±.002

.603 ±.004

.604 ±.007

.613 ±.002

.616 ±.002

.615 ±.001

.636 ±.013

.729 ±.022

.734 ±.012

.587 ±.001

.631 ±.018

4

.558 ±.002

.562 ±.004

.612 ±.008

.569 ±.001

.778 ±.009

.562 ±.004

.564 ±.006

.613 ±.001

.565 ±.002

.602 ±.002

.593 ±.011

.669 ±.033

.672 ±.030

.558 ±.001

.605 ±.022

5

.545 ±.001

.541 ±.001

.555 ±.004

.562 ±.001

.681 ±.011

.541 ±.001

.544 ±.003

.555 ±.001

.547 ±.001

.581 ±.008

.568 ±.009

.598 ±.017

.608 ±.018

.545 ±.001

.578 ±.005

1

.248 ±.016

.261 ±.003

.323 ±.001

.272 ±.006

.330 ±.003

.261 ±.003

.340 ±.046

.194 ±.004

.203 ±.025

.313 ±.003

.367 ±.001

.451 ±.083

.198 ±.019

.254 ±.022

.261 ±.005

2

.179 ±.005

.223 ±.004

.235 ±.004

.175 ±.002

.230 ±.001

.223 ±.004

.219 ±.038

.185 ±.002

.187 ±.001

.181 ±.007

.242 ±.023

.303 ±.094

.161 ±.009

.183 ±.002

.171 ±.006

3

.152 ±.007

.172 ±.008

.171 ±.002

.170 ±.001

.225 ±.001

.172 ±.002

.156 ±.007

.176 ±.005

.170 ±.004

.168 ±.003

.178 ±.023

.206 ±.023

.147 ±.004

.153 ±.007

.146 ±.004

4

.149 ±.005

.159 ±.002

.145 ±.003

.163 ±.002

.226 ±.002

.158 ±.002

.144 ±.003

.163 ±.005

.163 ±.002

.153 ±.011

.146 ±.003

.181 ±.029

.145 ±.003

.146 ±.003

.145 ±.003

5

.144 ±.003

.143 ±.003

.143 ±.002

.144 ±.003

.197 ±.001

.143 ±.003

.143 ±.002

.144 ±.003

.144 ±.003

.144 ±.003

.144 ±.003

.160 ±.014

.142 ±.003

.144 ±.003

.142 ±.003

1

.736 ±.008

.762 ±.018

.763 ±.007

.763 ±.007

.815 ±.009

.762 ±.018

.825 ±.012

.736 ±.008

.763 ±.007

.775 ±.006

.755 ±.016

.785 ±.032

.755 ±.017

.736 ±.008

.775 ±.006

2

.692 ±.005

.704 ±.007

.713 ±.009

.719 ±.010

.741 ±.012

.704 ±.007

.769 ±.009

.711 ±.009

.726 ±.013

.728 ±.013

.713 ±.007

.756 ±.036

.696 ±.014

.692 ±.008

.707 ±.003

3

.663 ±.004

.692 ±.006

.697 ±.010

.675 ±.010

.683 ±.007

.692 ±.006

.677 ±.020

.676 ±.013

.680 ±.019

.686 ±.018

.669 ±.011

.700 ±.022

.668 ±.012

.660 ±.006

.677 ±.008

4

.643 ±.011

.661 ±.005

.662 ±.010

.648 ±.011

.665 ±.007

.661 ±.005

.651 ±.012

.645 ±.010

.649 ±.016

.657 ±.008

.643 ±.008

.670 ±.010

.644 ±.009

.642 ±.012

.656 ±.008

5

.633 ±.010

.638 ±.011

.645 ±.007

.633 ±.008

.643 ±.009

.638 ±.011

.635 ±.006

.636 ±.008

.634 ±.018

.644 ±.011

.630 ±.010

.640 ±.008

.631 ±.011

.630 ±.012

.639 ±.007

1

.0117 ±.0005

.0114 ±.0011

.0117 ±.0011

.0117 ±.0011

.0128 ±.0030

.0114 ±.0011

.0129 ±.0025

.0116 ±.0011

.0119 ±.0012

.0119 ±.0013

.0116 ±.0012

.0153 ±.0042

.0118 ±.0013

.0115 ±.0009

.0114 ±.0011

2

.0114 ±.0009

.0113 ±.0011

.0117 ±.0011

.0118 ±.0011

.0117 ±.0009

.0113 ±.0011

.0131 ±.0027

.0117 ±.0011

.0117 ±.0010

.0116 ±.0009

.0115 ±.0010

.0138 ±.0025

.0115 ±.0013

.0113 ±.0010

.0115 ±.0012

3

.0113 ±.0010

.0114 ±.0011

.0119 ±.0009

.0117 ±.0011

.0118 ±.0013

.0114 ±.0011

.0116 ±.0015

.0117 ±.0012

.0119 ±.0009

.0118 ±.0010

.0115 ±.0011

.0123 ±.0021

.0115 ±.0011

.0115 ±.0012

.0115 ±.0012

4

.0114 ±.0011

.0114 ±.0010

.0118 ±.0009

.0119 ±.0010

.0118 ±.0013

.0114 ±.0010

.0115 ±.0011

.0119 ±.0009

.0120 ±.0009

.0117 ±.0010

.0115 ±.0011

.0124 ±.0007

.0114 ±.0012

.0114 ±.0011

.0114 ±.0012

5

.0114 ±.0010

.0113 ±.0010

.0116 ±.0011

.0118 ±.0012

.0115 ±.0010

.0113 ±.0010

.0115 ±.0010

.0119 ±.0013

.0119 ±.0013

.0115 ±.0009

.0114 ±.0010

.0130 ±.0008

.0115 ±.0012

.0115 ±.0011

.0115 ±.0012

1

12.6 ±6.2

14.1 ±7.1

12.6 ±6.2

13.4 ±6.8

15.2 ±6.1

14.1 ±7.1

14.9 ±6.0

12.6 ±6.2

13.4 ±6.8

12.6 ±6.2

14.2 ±6.8

15.4 ±5.3

12.6 ±6.4

12.6 ±6.2

15.3 ±3.2

2

13.7 ±6.8

13.7 ±6.8

13.7 ±6.8

13.8 ±6.8

15.3 ±6.2

13.7 ±6.8

13.4 ±5.2

13.5 ±6.5

13.7 ±6.8

13.7 ±6.8

14.1 ±7.1

13.9 ±6.7

13.3 ±5.3

13.7 ±6.8

14.5 ±5.7

3

12.3 ±6.0

14.0 ±7.0

14.0 ±7.0

14.0 ±6.9

14.3 ±6.6

13.6 ±6.5

12.7 ±6.1

12.3 ±5.9

13.2 ±6.4

14.0 ±7.1

13.7 ±6.7

14.0 ±7.0

12.4 ±6.2

12.3 ±5.9

14.5 ±6.1

4

12.5 ±5.8

13.7 ±6.7

12.0 ±5.4

12.0 ±5.4

14.2 ±6.7

13.6 ±6.6

12.7 ±6.2

11.4 ±5.2

12.0 ±5.4

12.7 ±6.0

13.7 ±6.7

13.7 ±6.8

12.6 ±6.3

12.5 ±5.8

13.7 ±6.8

5

11.2 ±5.1

14.0 ±7.0

12.6 ±6.0

12.6 ±5.9

13.7 ±6.8

13.9 ±6.9

12.6 ±6.1

12.6 ±5.9

12.6 ±5.9

12.6 ±5.9

12.6 ±5.9

12.7 ±6.0

11.9 ±6.0

12.6 ±5.9

14.0 ±7.1

1

.738 ±.091

.817 ±.126

.732 ±.159

.802 ±.099

1.18 ±.09

.794 ±.158

.720 ±.044

1.17 ±.04

.731 ±.160

.893 ±.034

.814 ±.119

.937 ±.195

.791 ±.065

.719 ±.102

.798 ±.101

2

.685 ±.050

.761 ±.194

.783 ±.199

.828 ±.068

1.02 ±.11

.761 ±.194

.680 ±.059

.982 ±.154

.787 ±.142

.857 ±.069

.752 ±.191

.839 ±.063

.658 ±.089

.646 ±.128

.772 ±.133

3

.644 ±.108

.786 ±.187

.784 ±.190

.842 ±.132

.804 ±.012

.786 ±.187

.665 ±.076

.669 ±.081

.732 ±.206

.792 ±.122

.736 ±.198

.841 ±.234

.642 ±.084

.655 ±.113

.754 ±.139

4

.653 ±.100

.760 ±.204

.728 ±.205

.762 ±.167

.766 ±.006

.760 ±.204

.665 ±.076

.658 ±.133

.647 ±.100

.693 ±.138

.760 ±.151

.661 ±.069

.648 ±.079

.641 ±.098

.741 ±.157

5

.641 ±.107

.675 ±.093

.672 ±.080

.745 ±.183

.679 ±.089

.675 ±.093

.669 ±.070

.638 ±.104

.636 ±.101

.671 ±.132

.664 ±.066

.735 ±.041

.654 ±.077

.638 ±.102

.687 ±.048

1

.598 ±.002

.598 ±.002

.598 ±.002

.598 ±.002

.598 ±.002

.598 ±.002

.598 ±.002

.598 ±.002

.598 ±.002

.598 ±.002

.598 ±.002

.598 ±.002

.598 ±.002

.598 ±.002

.598 ±.002

2

.598 ±.002

.597 ±.002

.597 ±.001

.598 ±.002

.598 ±.002

.597 ±.002

.598 ±.002

.598 ±.002

.598 ±.002

.598 ±.002

.598 ±.002

.598 ±.002

.598 ±.002

.598 ±.002

.598 ±.002

3

.598 ±.002

.597 ±.002

.596 ±.002

.598 ±.002

.598 ±.002

.596 ±.002

.598 ±.002

.598 ±.002

.598 ±.002

.598 ±.002

.598 ±.002

.598 ±.002

.598 ±.002

.598 ±.002

.597 ±.002

4

.598 ±.002

.596 ±.002

.596 ±.002

.598 ±.002

.598 ±.002

.596 ±.002

.598 ±.002

.598 ±.002

.598 ±.002

.597 ±.002

.597 ±.002

.598 ±.002

.598 ±.002

.598 ±.002

.597 ±.002

5

.598 ±.002

.596 ±.002

.596 ±.002

.597 ±.002

.598 ±.002

.596 ±.002

.597 ±.002

.598 ±.002

.597 ±.001

.597 ±.002

.597 ±.002

.598 ±.002

.598 ±.002

.598 ±.002

.598 ±.001

1

9.75 ±.18

9.62 ±.02

10.0 ±.1

9.72 ±.03

10.5 ±.2

9.83 ±.04

9.71 ±.08

9.66 ±.13

10.7 ±.4

9.74 ±.11

9.82 ±.11

9.75 ±.09

9.81 ±.07

9.71 ±.15

9.63 ±.11

2

9.53 ±.07

9.48 ±.05

9.54 ±.06

9.47 ±.07

9.51 ±.05

9.52 ±.05

9.48 ±.05

9.50 ±.10

9.48 ±.06

9.50 ±.14

9.54 ±.06

9.61 ±.08

9.56 ±.06

9.46 ±.05

9.50 ±.01

3

9.41 ±.05

9.48 ±.03

9.47 ±.03

9.42 ±.05

9.42 ±.05

9.44 ±.04

9.36 ±.05

9.43 ±.10

9.38 ±.03

9.38 ±.11

9.43 ±.03

9.45 ±.03

9.41 ±.05

9.42 ±.05

9.39 ±.05

4

9.35 ±.04

9.38 ±.05

9.40 ±.02

9.37 ±.04

9.36 ±.02

9.36 ±.02

9.37 ±.06

9.38 ±.03

9.32 ±.07

9.37 ±.06

9.36 ±.03

9.38 ±.05

9.45 ±.10

9.35 ±.04

9.35 ±.02

5

9.32 ±.06

9.33 ±.04

9.34 ±.03

9.32 ±.03

9.31 ±.04

9.32 ±.07

9.31 ±.05

9.34 ±.05

9.29 ±.02

9.33 ±.03

9.33 ±.04

9.33 ±.02

9.33 ±.07

9.33 ±.06

9.36 ±.05

1

.739 ±.014

.719 ±.006

.697 ±.008

.718 ±.012

.584 ±.017

.720 ±.006

.737 ±.009

.727 ±.010

.722 ±.023

.715 ±.016

.737 ±.012

.606 ±.038

.734 ±.015

.742 ±.009

.742 ±.009

2

.789 ±.008

.724 ±.008

.724 ±.007

.763 ±.012

.623 ±.005

.725 ±.009

.797 ±.023

.765 ±.010

.762 ±.011

.737 ±.011

.778 ±.016

.663 ±.019

.804 ±.010

.794 ±.008

.761 ±.018

3

.802 ±.003

.768 ±.004

.738 ±.005

.800 ±.004

.675 ±.002

.768 ±.004

.815 ±.007

.802 ±.007

.803 ±.011

.762 ±.010

.801 ±.004

.735 ±.047

.826 ±.010

.800 ±.007

.798 ±.038

4

.826 ±.006

.797 ±.009

.809 ±.021

.817 ±.006

.732 ±.003

.798 ±.010

.823 ±.006

.815 ±.005

.823 ±.007

.800 ±.006

.816 ±.011

.809 ±.009

.833 ±.008

.832 ±.007

.818 ±.024

5

.833 ±.005

.827 ±.002

.825 ±.007

.828 ±.003

.811 ±.006

.827 ±.002

.834 ±.005

.824 ±.012

.830 ±.010

.824 ±.004

.830 ±.006

.826 ±.003

.833 ±.005

.832 ±.004

.829 ±.008

1

.815 ±.042

.818 ±.038

.769 ±.016

.780 ±.022

.709 ±.020

.817 ±.038

.787 ±.027

.801 ±.011

.761 ±.023

.808 ±.043

.736 ±.013

.753 ±.034

.828 ±.029

.824 ±.025

.793 ±.048

2

.850 ±.024

.869 ±.027

.871 ±.022

.822 ±.026

.808 ±.016

.874 ±.032

.858 ±.008

.849 ±.011

.824 ±.024

.853 ±.037

.814 ±.038

.770 ±.036

.874 ±.024

.848 ±.030

.850 ±.010

3

Dataset

Budget idx j 1

.521 ±.018

.447 ±.165

.875 ±.020

.889 ±.022

.872 ±.019

.870 ±.011

.858 ±.017

.891 ±.020

.883 ±.021

.868 ±.009

.859 ±.018

.865 ±.024

.873 ±.026

.829 ±.019

.885 ±.022

.865 ±.023

.887 ±.030

4

audiology diagnosis

.452 ±.160

.888 ±.020

.893 ±.024

.881 ±.017

.876 ±.012

.856 ±.019

.897 ±.027

.895 ±.025

.877 ±.008

.889 ±.012

.884 ±.024

.885 ±.024

.865 ±.040

.897 ±.019

.886 ±.019

.900 ±.025

5

(continued from previous page)

.895 ±.017

.900 ±.018

.899 ±.016

.884 ±.017

.876 ±.037

.898 ±.019

.897 ±.025

.896 ±.010

.898 ±.014

.893 ±.021

.890 ±.018

.882 ±.036

.898 ±.020

.895 ±.019

.900 ±.022

superconductivity

pva_revenue prediction_kddcup98

automobile

brasilian houses (×103 )

body_density prediction

wine quality

miami housing

allstate claims_severity

forest fires

qsar_tid_11

RMSE (↓)

marketing campaign

in_vehicle_coupon recommendation

AUC (↑)

51

CART

ElasticNet

F-test

GainRatio

LaplacianScore

Lasso

LOCO

MarkovBlanket

MI

mRMR

(R)ReliefF

Random

SBE

RFImportance

SFS

2

.940 ±.019

3

.952 ±.016

.897 ±.042

.879 ±.022

.941 ±.017

.863 ±.011

.900 ±.038

.943 ±.017

.942 ±.014

.938 ±.016

.946 ±.015

.933 ±.022

.889 ±.051

.937 ±.033

.953 ±.018

.943 ±.019

4

.954 ±.012

.907 ±.032

.913 ±.037

.940 ±.017

.873 ±.014

.907 ±.032

.945 ±.012

.949 ±.014

.942 ±.014

.952 ±.015

.940 ±.019

.943 ±.005

.955 ±.013

.955 ±.016

.949 ±.015

5

.954 ±.014

.933 ±.026

.940 ±.017

.945 ±.014

.884 ±.004

.934 ±.026

.947 ±.013

.950 ±.015

.952 ±.013

.954 ±.015

.946 ±.011

.943 ±.008

.954 ±.014

.954 ±.015

.955 ±.009

1

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

2

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

3

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

4

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

5

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

1

.943 ±.005

.939 ±.010

.940 ±.009

.939 ±.011

.933 ±.007

.942 ±.016

.907 ±.027

.935 ±.002

.939 ±.009

.938 ±.006

.887 ±.054

.916 ±.015

.934 ±.019

.936 ±.013

.939 ±.009

2

.941 ±.009

.946 ±.010

.935 ±.010

.938 ±.008

.941 ±.011

.945 ±.010

.936 ±.004

.939 ±.010

.939 ±.012

.945 ±.010

.945 ±.008

.933 ±.004

.936 ±.004

.941 ±.005

.937 ±.011

3

.942 ±.005

.943 ±.016

.947 ±.014

.940 ±.010

.941 ±.011

.940 ±.010

.940 ±.010

.944 ±.013

.944 ±.011

.946 ±.008

.941 ±.009

.942 ±.012

.943 ±.008

.936 ±.010

.943 ±.009

4

.948 ±.006

.940 ±.006

.943 ±.012

.946 ±.008

.946 ±.008

.942 ±.012

.945 ±.017

.944 ±.011

.942 ±.011

.947 ±.016

.943 ±.011

.948 ±.010

.941 ±.007

.945 ±.020

.944 ±.007

5

.942 ±.009

.943 ±.007

.945 ±.008

.946 ±.004

.943 ±.006

.943 ±.004

.945 ±.017

.946 ±.008

.945 ±.006

.944 ±.005

.946 ±.008

.943 ±.010

.947 ±.004

.946 ±.009

.944 ±.013

1

.629 ±.007

.630 ±.006

.632 ±.004

.632 ±.004

.501 ±.001

.630 ±.006

.587 ±.020

.632 ±.004

.502 ±.002

.632 ±.004

.503 ±.008

.510 ±.005

.530 ±.007

.632 ±.004

.575 ±.011

2

.697 ±.002

.700 ±.002

.702 ±.003

.701 ±.003

.501 ±.001

.700 ±.002

.631 ±.021

.701 ±.004

.508 ±.002

.701 ±.003

.505 ±.002

.574 ±.027

.578 ±.032

.700 ±.001

.578 ±.007

3

.783 ±.005

.786 ±.004

.786 ±.005

.781 ±.004

.515 ±.002

.786 ±.004

.637 ±.021

.784 ±.003

.522 ±.002

.787 ±.003

.509 ±.003

.602 ±.033

.598 ±.016

.785 ±.003

.612 ±.021

4

.860 ±.004

.868 ±.003

.867 ±.003

.863 ±.005

.525 ±.002

.868 ±.003

.677 ±.010

.859 ±.003

.545 ±.004

.861 ±.004

.525 ±.004

.651 ±.012

.637 ±.018

.866 ±.004

.671 ±.023

5

.896 ±.003

.896 ±.003

.897 ±.003

.897 ±.003

.543 ±.007

.896 ±.003

.779 ±.013

.891 ±.005

.684 ±.003

.888 ±.002

.565 ±.028

.771 ±.019

.764 ±.010

.895 ±.003

.780 ±.014

1

.948 ±.023

.937 ±.003

.927 ±.010

.917 ±.029

.543 ±.026

.909 ±.050

.746 ±.087

.912 ±.056

.923 ±.015

.949 ±.015

.877 ±.058

.786 ±.077

.746 ±.112

.933 ±.020

.757 ±.054

2

.961 ±.012

.960 ±.019

.953 ±.028

.969 ±.014

.685 ±.089

.981 ±.011

.880 ±.051

.977 ±.006

.972 ±.005

.973 ±.007

.952 ±.020

.857 ±.062

.863 ±.033

.956 ±.024

.875 ±.041

3

.966 ±.007

.983 ±.002

.968 ±.010

.978 ±.009

.799 ±.075

.987 ±.002

.940 ±.011

.983 ±.008

.991 ±.005

.984 ±.003

.975 ±.014

.957 ±.007

.949 ±.017

.991 ±.002

.946 ±.025

4

.983 ±.011

.990 ±.006

.982 ±.008

.988 ±.001

.917 ±.038

.989 ±.001

.952 ±.016

.990 ±.005

.990 ±.002

.986 ±.003

.980 ±.012

.974 ±.012

.972 ±.008

.989 ±.002

.969 ±.015

5

.986 ±.009

.989 ±.004

.989 ±.003

.989 ±.003

.956 ±.014

.991 ±.003

.991 ±.009

.989 ±.004

.989 ±.004

.986 ±.004

.989 ±.006

.979 ±.009

.983 ±.006

.990 ±.003

.985 ±.004

1

.741 ±.010

.767 ±.012

.767 ±.012

.816 ±.003

.597 ±.011

.767 ±.012

.816 ±.003

.816 ±.003

.816 ±.003

.816 ±.003

.749 ±.060

.540 ±.060

.816 ±.003

.741 ±.010

.816 ±.003

2

.758 ±.008

.784 ±.006

.784 ±.006

.848 ±.007

.766 ±.003

.784 ±.006

.846 ±.007

.848 ±.007

.832 ±.011

.832 ±.007

.814 ±.004

.683 ±.068

.831 ±.012

.758 ±.008

.836 ±.004

3

.830 ±.001

.799 ±.004

.833 ±.026

.847 ±.008

.784 ±.008

.799 ±.004

.855 ±.015

.847 ±.008

.855 ±.014

.832 ±.007

.829 ±.003

.708 ±.092

.841 ±.003

.830 ±.001

.843 ±.005

4

.844 ±.002

.843 ±.032

.864 ±.005

.847 ±.008

.813 ±.011

.843 ±.032

.862 ±.004

.847 ±.008

.859 ±.014

.844 ±.003

.842 ±.006

.798 ±.052

.864 ±.006

.842 ±.002

.858 ±.014

5

.864 ±.005

.842 ±.033

.864 ±.006

.853 ±.011

.813 ±.012

.842 ±.033

.864 ±.006

.859 ±.003

.864 ±.006

.846 ±.002

.851 ±.008

.837 ±.034

.864 ±.006

.863 ±.005

.864 ±.005

1

.775 ±.083

.776 ±.072

.800 ±.013

.807 ±.046

.779 ±.091

.800 ±.044

.769 ±.029

.832 ±.031

.780 ±.023

.833 ±.032

.813 ±.067

.816 ±.043

.791 ±.087

.762 ±.024

.745 ±.098

2

.800 ±.013

.821 ±.063

.833 ±.029

.822 ±.004

.811 ±.038

.821 ±.033

.752 ±.035

.844 ±.022

.811 ±.038

.834 ±.009

.838 ±.035

.776 ±.025

.797 ±.093

.794 ±.050

.744 ±.068

3

.817 ±.030

.804 ±.035

.811 ±.021

.813 ±.027

.825 ±.043

.806 ±.031

.766 ±.051

.828 ±.042

.823 ±.020

.812 ±.024

.795 ±.059

.803 ±.008

.815 ±.070

.816 ±.034

.754 ±.074

4

.812 ±.015

.796 ±.033

.813 ±.031

.802 ±.033

.845 ±.011

.806 ±.022

.793 ±.060

.810 ±.034

.815 ±.028

.816 ±.023

.810 ±.048

.814 ±.041

.837 ±.037

.812 ±.024

.797 ±.050

5

.817 ±.010

.805 ±.027

.804 ±.023

.808 ±.025

.813 ±.012

.805 ±.022

.799 ±.046

.812 ±.025

.807 ±.025

.808 ±.031

.824 ±.037

.811 ±.044

.808 ±.015

.809 ±.023

.814 ±.066

1

.957 ±.000

.920 ±.001

.935 ±.002

.924 ±.000

.940 ±.001

.920 ±.001

.941 ±.002

.961 ±.000

.959 ±.000

.913 ±.000

.956 ±.000

.858 ±.052

.968 ±.002

.959 ±.000

.971 ±.001

2

.979 ±.000

.977 ±.001

.956 ±.001

.976 ±.000

.979 ±.000

.977 ±.001

.976 ±.000

.977 ±.000

.977 ±.000

.973 ±.000

.976 ±.005

.888 ±.019

.985 ±.001

.974 ±.001

.984 ±.001

3

.989 ±.001

.989 ±.001

.981 ±.008

.985 ±.000

.989 ±.000

.989 ±.001

.990 ±.001

.987 ±.001

.986 ±.000

.983 ±.000

.989 ±.000

.977 ±.011

.992 ±.000

.985 ±.001

.991 ±.001

4

.992 ±.000

.990 ±.000

.990 ±.001

.989 ±.000

.992 ±.000

.990 ±.000

.993 ±.000

.989 ±.000

.990 ±.000

.990 ±.000

.992 ±.000

.990 ±.003

.993 ±.000

.990 ±.001

.993 ±.000

5

.994 ±.000

.993 ±.000

.993 ±.000

.992 ±.000

.993 ±.000

.993 ±.000

.994 ±.000

.992 ±.000

.993 ±.001

.993 ±.000

.994 ±.000

.993 ±.001

.994 ±.000

.994 ±.000

.994 ±.000

1

.772 ±.008

.765 ±.007

.771 ±.008

.768 ±.012

.665 ±.003

.768 ±.005

.756 ±.048

.784 ±.010

.769 ±.012

.772 ±.014

.729 ±.014

.723 ±.051

.781 ±.009

.772 ±.010

.765 ±.008

2

.776 ±.008

.777 ±.007

.784 ±.012

.784 ±.009

.768 ±.009

.780 ±.006

.782 ±.017

.789 ±.010

.787 ±.011

.789 ±.009

.790 ±.007

.752 ±.018

.789 ±.010

.780 ±.009

.787 ±.008

3

.792 ±.007

.785 ±.008

.793 ±.007

.793 ±.005

.768 ±.011

.787 ±.006

.786 ±.017

.793 ±.007

.794 ±.006

.792 ±.008

.794 ±.007

.772 ±.008

.792 ±.009

.792 ±.007

.790 ±.010

4

.794 ±.008

.794 ±.007

.794 ±.007

.794 ±.007

.778 ±.007

.794 ±.007

.793 ±.009

.794 ±.007

.794 ±.008

.794 ±.008

.794 ±.007

.784 ±.010

.793 ±.009

.795 ±.007

.793 ±.009

5

.795 ±.008

.795 ±.008

.795 ±.008

.795 ±.007

.791 ±.012

.794 ±.008

.794 ±.009

.795 ±.008

.795 ±.007

.794 ±.008

.795 ±.008

.792 ±.005

.793 ±.009

.795 ±.008

.793 ±.009

1

.577 ±.035

.657 ±.054

.562 ±.030

.528 ±.057

.522 ±.014

.580 ±.034

.543 ±.098

.506 ±.032

.604 ±.078

.528 ±.057

.606 ±.068

.589 ±.029

.574 ±.042

.577 ±.064

.683 ±.060

2

.670 ±.030

.706 ±.039

.624 ±.079

.651 ±.016

.557 ±.025

.727 ±.032

.651 ±.023

.625 ±.043

.615 ±.086

.651 ±.016

.661 ±.075

.630 ±.040

.596 ±.073

.696 ±.055

.673 ±.075

3

.706 ±.059

.776 ±.032

.674 ±.090

.697 ±.042

.604 ±.061

.755 ±.019

.740 ±.050

.724 ±.052

.717 ±.073

.697 ±.042

.730 ±.032

.732 ±.047

.714 ±.056

.774 ±.031

.738 ±.038

4

.755 ±.023

.774 ±.022

.783 ±.038

.730 ±.003

.698 ±.044

.759 ±.022

.727 ±.041

.776 ±.027

.764 ±.054

.730 ±.003

.779 ±.010

.775 ±.021

.780 ±.019

.789 ±.031

.759 ±.023

5

.780 ±.011

.793 ±.025

.793 ±.013

.771 ±.024

.785 ±.007

.777 ±.023

.755 ±.028

.788 ±.013

.777 ±.022

.771 ±.024

.774 ±.035

.791 ±.022

.800 ±.009

.790 ±.030

.785 ±.021

1

.677 ±.013

.711 ±.010

.711 ±.010

.683 ±.014

.624 ±.007

.711 ±.010

.702 ±.008

.698 ±.007

.674 ±.013

.668 ±.007

.676 ±.008

.673 ±.021

.708 ±.006

.666 ±.007

.714 ±.009

2

.717 ±.009

.730 ±.010

.729 ±.008

.725 ±.012

.650 ±.007

.730 ±.010

.733 ±.011

.733 ±.009

.732 ±.013

.696 ±.007

.700 ±.009

.704 ±.026

.730 ±.005

.714 ±.013

.732 ±.011

3

.726 ±.011

.741 ±.009

.741 ±.010

.737 ±.010

.734 ±.011

.741 ±.009

.744 ±.015

.739 ±.010

.750 ±.009

.733 ±.006

.717 ±.011

.737 ±.015

.746 ±.016

.726 ±.011

.737 ±.008

4

.746 ±.017

.757 ±.010

.753 ±.011

.748 ±.011

.747 ±.010

.755 ±.012

.759 ±.011

.748 ±.011

.759 ±.010

.753 ±.011

.746 ±.009

.749 ±.006

.755 ±.016

.731 ±.010

.747 ±.012

5

Dataset

Budget idx j 1

.933 ±.017

.952 ±.020

.762 ±.010

.761 ±.013

.759 ±.010

.753 ±.008

.759 ±.010

.762 ±.011

.761 ±.012

.760 ±.013

.761 ±.011

.759 ±.010

.759 ±.010

.754 ±.011

.760 ±.011

.762 ±.010

.755 ±.010

1

polish_companies bankruptcy

.947 ±.013

.920 ±.040

.887 ±.031

.982 ±.016

.997 ±.005

.990 ±.012

.728 ±.049

.948 ±.045

.646 ±.130

.985 ±.000

.990 ±.009

.980 ±.018

.982 ±.025

.714 ±.310

.746 ±.268

.990 ±.009

.554 ±.065

2

colon_tumor

.875 ±.009

.888 ±.057

.909 ±.059

.998 ±.004

.993 ±.013

.997 ±.005

.836 ±.151

.992 ±.007

.743 ±.062

.952 ±.049

.997 ±.005

.995 ±.009

1.00 ±.00

.634 ±.132

.792 ±.153

.997 ±.005

.741 ±.131

3

mechanisms of_action

.842 ±.009

.928 ±.027

.960 ±.036

.997 ±.005

.988 ±.015

.997 ±.005

.878 ±.152

.997 ±.005

.923 ±.036

.980 ±.034

.997 ±.005

.997 ±.005

.997 ±.005

.936 ±.054

.950 ±.033

1.00 ±.00

.939 ±.020

4

santander_customer transaction_prediction

.844 ±.006

.938 ±.012

.972 ±.037

.997 ±.005

.997 ±.005

.997 ±.005

.974 ±.033

.997 ±.005

.975 ±.011

.990 ±.011

.997 ±.005

.997 ±.005

.997 ±.005

.957 ±.027

.954 ±.040

.997 ±.005

.982 ±.009

5

nci_prostate cancer

.928 ±.015

.936 ±.016

.977 ±.020

.997 ±.005

.997 ±.005

.997 ±.005

1.00 ±.00

.995 ±.009

.992 ±.014

1.00 ±.00

.997 ±.005

.997 ±.005

.997 ±.005

.974 ±.025

.990 ±.012

.997 ±.005

.997 ±.005

1

bank_customer churn

.922 ±.019

.943 ±.014

.772 ±.032

.818 ±.018

.794 ±.064

.842 ±.061

.756 ±.020

.814 ±.015

.859 ±.090

.817 ±.027

.798 ±.040

.789 ±.017

.787 ±.053

.818 ±.038

.840 ±.045

.777 ±.024

.830 ±.037

2

hepatitis_survival prediction

.936 ±.016

.944 ±.014

.829 ±.073

.848 ±.007

.848 ±.004

.870 ±.055

.796 ±.016

.855 ±.043

.862 ±.076

.828 ±.015

.856 ±.024

.831 ±.001

.866 ±.037

.832 ±.029

.833 ±.071

.867 ±.053

.828 ±.044

3

customer_satisfaction in_airline

.936 ±.008

.859 ±.030

.860 ±.038

.882 ±.023

.864 ±.008

.879 ±.052

.818 ±.026

.875 ±.034

.865 ±.040

.838 ±.024

.877 ±.055

.851 ±.011

.888 ±.025

.819 ±.028

.880 ±.040

.872 ±.053

.859 ±.056

4

heloc

.822 ±.014

.858 ±.012

.873 ±.033

.881 ±.027

.888 ±.014

.880 ±.055

.855 ±.040

.877 ±.032

.875 ±.051

.880 ±.055

.876 ±.053

.889 ±.034

.884 ±.017

.893 ±.039

.872 ±.032

.879 ±.040

.883 ±.051

5

hiva agnostic

.833 ±.017

.931 ±.022

.886 ±.033

.895 ±.029

.897 ±.030

.882 ±.042

.883 ±.028

.885 ±.040

.888 ±.043

.892 ±.055

.884 ±.063

.884 ±.063

.889 ±.018

.872 ±.063

.889 ±.043

.894 ±.041

.884 ±.038

1

bank marketing

.902 ±.025

.878 ±.022

.568 ±.104

.626 ±.136

.569 ±.065

.504 ±.008

.631 ±.075

.626 ±.136

.573 ±.142

.561 ±.128

.491 ±.073

.496 ±.008

.526 ±.120

.651 ±.200

.649 ±.168

.503 ±.077

.610 ±.040

2

leukemia allaml

.863 ±.022

.855 ±.032

.635 ±.051

.684 ±.093

.617 ±.160

.538 ±.065

.614 ±.063

.684 ±.093

.519 ±.035

.543 ±.090

.582 ±.104

.532 ±.094

.586 ±.114

.507 ±.083

.520 ±.165

.598 ±.166

.638 ±.061

3

heart_disease cleveland

.827 ±.005

.951 ±.014

.683 ±.076

.646 ±.116

.647 ±.146

.578 ±.109

.698 ±.106

.646 ±.116

.557 ±.037

.593 ±.052

.631 ±.051

.578 ±.145

.538 ±.158

.674 ±.110

.690 ±.143

.603 ±.173

.656 ±.074

4

clock_protein period

.949 ±.016

.665 ±.057

.670 ±.048

.646 ±.197

.619 ±.147

.725 ±.104

.670 ±.048

.554 ±.032

.600 ±.122

.584 ±.105

.578 ±.149

.621 ±.186

.598 ±.050

.624 ±.146

.612 ±.127

.637 ±.056

5

(continued from previous page)

.543 ±.069

.543 ±.067

.651 ±.171

.644 ±.174

.633 ±.054

.543 ±.067

.593 ±.170

.668 ±.158

.635 ±.147

.635 ±.147

.631 ±.171

.651 ±.122

.665 ±.141

.638 ±.140

.601 ±.069

52

CART

ElasticNet

F-test

GainRatio

LaplacianScore

Lasso

LOCO

MarkovBlanket

MI

mRMR

(R)ReliefF

Random

SBE

RFImportance

SFS

2

.908 ±.001

3

.921 ±.003

.910 ±.002

.912 ±.013

.878 ±.008

.825 ±.010

.917 ±.006

.841 ±.062

.857 ±.009

.900 ±.013

.915 ±.007

.903 ±.001

.754 ±.122

.919 ±.003

.913 ±.006

.911 ±.007

4

.920 ±.005

.920 ±.004

.920 ±.005

.871 ±.009

.821 ±.013

.920 ±.003

.877 ±.063

.861 ±.002

.908 ±.009

.918 ±.006

.910 ±.008

.847 ±.028

.916 ±.005

.919 ±.003

.915 ±.003

5

.913 ±.004

.919 ±.003

.915 ±.003

.867 ±.002

.910 ±.003

.918 ±.003

.915 ±.004

.860 ±.001

.911 ±.008

.914 ±.003

.909 ±.006

.880 ±.056

.914 ±.004

.918 ±.002

.916 ±.002

1

.980 ±.006

.958 ±.041

.977 ±.005

.963 ±.014

.928 ±.014

.977 ±.001

.949 ±.024

.980 ±.005

.982 ±.004

.972 ±.005

.942 ±.003

.966 ±.017

.942 ±.016

.984 ±.004

.976 ±.007

2

.987 ±.003

.981 ±.011

.981 ±.004

.977 ±.004

.960 ±.007

.984 ±.003

.977 ±.008

.983 ±.004

.983 ±.003

.985 ±.002

.967 ±.001

.979 ±.008

.973 ±.003

.987 ±.003

.982 ±.006

3

.989 ±.003

.990 ±.002

.985 ±.003

.979 ±.005

.981 ±.003

.988 ±.001

.987 ±.004

.988 ±.003

.985 ±.004

.987 ±.003

.984 ±.003

.987 ±.004

.986 ±.003

.990 ±.003

.987 ±.003

4

.991 ±.002

.991 ±.002

.989 ±.003

.990 ±.003

.988 ±.002

.991 ±.001

.990 ±.003

.990 ±.004

.987 ±.003

.990 ±.002

.988 ±.004

.989 ±.002

.989 ±.004

.991 ±.002

.989 ±.003

5

.992 ±.002

.991 ±.003

.991 ±.003

.991 ±.003

.991 ±.002

.991 ±.002

.991 ±.003

.991 ±.003

.991 ±.003

.992 ±.002

.991 ±.002

.992 ±.003

.992 ±.003

.991 ±.003

.991 ±.003

1

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

2

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

3

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

4

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

5

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

.500 ±.000

1

.655 ±.076

.522 ±.019

.762 ±.016

.746 ±.017

.539 ±.029

.522 ±.019

.799 ±.009

.765 ±.011

.620 ±.027

.750 ±.020

.653 ±.021

.515 ±.010

.505 ±.004

.748 ±.017

.798 ±.015

2

.709 ±.070

.542 ±.056

.769 ±.016

.745 ±.018

.558 ±.040

.542 ±.056

.804 ±.013

.795 ±.016

.625 ±.019

.762 ±.014

.644 ±.041

.549 ±.009

.530 ±.025

.756 ±.014

.800 ±.017

3

.782 ±.021

.578 ±.035

.774 ±.017

.786 ±.013

.558 ±.037

.578 ±.035

.801 ±.010

.793 ±.014

.709 ±.092

.764 ±.023

.767 ±.008

.602 ±.033

.623 ±.019

.770 ±.002

.793 ±.017

4

.795 ±.010

.606 ±.055

.782 ±.023

.794 ±.017

.602 ±.015

.606 ±.055

.798 ±.009

.785 ±.008

.771 ±.003

.784 ±.019

.787 ±.011

.714 ±.036

.683 ±.044

.772 ±.008

.782 ±.010

5

.793 ±.011

.719 ±.107

.792 ±.014

.796 ±.018

.713 ±.014

.719 ±.107

.809 ±.011

.780 ±.001

.772 ±.004

.786 ±.014

.796 ±.008

.724 ±.040

.726 ±.040

.796 ±.010

.787 ±.013

1

.720 ±.016

.693 ±.042

.693 ±.025

.568 ±.062

.562 ±.016

.696 ±.035

.553 ±.051

.552 ±.043

.686 ±.021

.568 ±.062

.568 ±.022

.578 ±.014

.570 ±.010

.693 ±.028

.736 ±.013

2

.772 ±.007

.748 ±.033

.717 ±.034

.622 ±.027

.586 ±.007

.781 ±.023

.624 ±.035

.635 ±.035

.727 ±.020

.622 ±.027

.677 ±.009

.634 ±.045

.661 ±.044

.736 ±.043

.745 ±.022

3

.833 ±.016

.827 ±.029

.799 ±.036

.739 ±.028

.729 ±.008

.839 ±.019

.750 ±.022

.759 ±.032

.806 ±.017

.739 ±.028

.827 ±.013

.757 ±.018

.751 ±.004

.820 ±.037

.799 ±.038

4

.876 ±.016

.862 ±.028

.876 ±.021

.844 ±.014

.844 ±.022

.870 ±.026

.846 ±.020

.853 ±.032

.867 ±.013

.844 ±.014

.893 ±.021

.847 ±.020

.844 ±.018

.889 ±.027

.850 ±.028

5

.895 ±.026

.888 ±.027

.895 ±.024

.884 ±.020

.882 ±.023

.886 ±.026

.885 ±.013

.891 ±.026

.896 ±.014

.884 ±.020

.899 ±.024

.881 ±.028

.877 ±.021

.896 ±.029

.884 ±.024

1

.932 ±.011

.927 ±.003

.928 ±.006

.932 ±.011

.933 ±.009

.923 ±.012

.922 ±.017

.929 ±.001

.932 ±.007

.932 ±.006

.917 ±.009

.936 ±.009

.927 ±.003

.937 ±.003

.936 ±.001

2

.934 ±.007

.934 ±.002

.931 ±.004

.937 ±.011

.939 ±.010

.941 ±.004

.936 ±.015

.934 ±.003

.936 ±.008

.941 ±.010

.929 ±.003

.934 ±.008

.939 ±.009

.937 ±.013

.941 ±.004

3

.935 ±.009

.939 ±.008

.928 ±.002

.940 ±.007

.938 ±.009

.945 ±.011

.943 ±.010

.937 ±.004

.938 ±.009

.939 ±.011

.932 ±.005

.939 ±.005

.944 ±.011

.942 ±.011

.942 ±.012

4

.942 ±.010

.946 ±.009

.934 ±.007

.939 ±.010

.936 ±.011

.947 ±.008

.942 ±.011

.940 ±.009

.940 ±.008

.941 ±.006

.942 ±.008

.941 ±.010

.942 ±.007

.942 ±.013

.942 ±.007

5

.942 ±.010

.945 ±.013

.939 ±.006

.939 ±.012

.942 ±.011

.946 ±.011

.942 ±.010

.943 ±.010

.944 ±.008

.940 ±.011

.941 ±.011

.946 ±.007

.940 ±.014

.941 ±.013

.944 ±.007

1

.969 ±.014

.778 ±.214

.892 ±.078

.974 ±.007

.533 ±.001

.778 ±.214

.967 ±.017

.965 ±.015

.923 ±.026

.911 ±.037

.935 ±.016

.706 ±.221

.976 ±.006

.971 ±.012

.927 ±.074

2

.967 ±.016

.923 ±.020

.907 ±.071

.966 ±.015

.568 ±.006

.923 ±.020

.970 ±.018

.955 ±.021

.921 ±.027

.931 ±.027

.906 ±.068

.742 ±.173

.982 ±.005

.969 ±.015

.923 ±.083

3

.969 ±.018

.942 ±.030

.939 ±.048

.968 ±.017

.551 ±.037

.942 ±.030

.970 ±.014

.955 ±.021

.936 ±.035

.935 ±.026

.914 ±.092

.952 ±.024

.983 ±.003

.969 ±.015

.929 ±.075

4

.968 ±.017

.945 ±.015

.946 ±.036

.969 ±.018

.580 ±.035

.945 ±.015

.970 ±.014

.956 ±.022

.940 ±.009

.928 ±.035

.966 ±.016

.931 ±.074

.984 ±.003

.970 ±.016

.927 ±.074

5

.969 ±.018

.951 ±.025

.969 ±.016

.969 ±.018

.908 ±.090

.951 ±.025

.970 ±.014

.970 ±.016

.958 ±.020

.955 ±.033

.965 ±.016

.971 ±.017

.984 ±.003

.970 ±.016

.946 ±.065

1

.739 ±.037

.711 ±.066

.743 ±.016

.744 ±.015

.534 ±.003

.663 ±.090

.735 ±.027

.737 ±.028

.733 ±.006

.717 ±.034

.645 ±.019

.698 ±.045

.675 ±.045

.736 ±.032

.746 ±.032

2

.751 ±.025

.736 ±.041

.747 ±.020

.743 ±.022

.563 ±.035

.722 ±.043

.743 ±.026

.738 ±.021

.742 ±.021

.726 ±.044

.645 ±.021

.744 ±.029

.727 ±.033

.741 ±.033

.759 ±.024

3

.752 ±.024

.738 ±.040

.749 ±.024

.744 ±.022

.643 ±.025

.726 ±.043

.747 ±.023

.742 ±.023

.748 ±.019

.742 ±.030

.700 ±.028

.717 ±.013

.700 ±.047

.741 ±.023

.752 ±.026

4

.752 ±.023

.738 ±.040

.752 ±.022

.748 ±.023

.652 ±.021

.734 ±.040

.752 ±.020

.749 ±.016

.750 ±.014

.753 ±.009

.728 ±.018

.754 ±.024

.740 ±.019

.753 ±.022

.752 ±.026

5

.754 ±.021

.750 ±.030

.753 ±.022

.752 ±.026

.651 ±.024

.751 ±.027

.755 ±.024

.752 ±.022

.753 ±.021

.759 ±.020

.728 ±.017

.750 ±.026

.741 ±.037

.752 ±.022

.755 ±.021

1

.922 ±.077

.951 ±.051

.960 ±.042

.956 ±.064

.520 ±.070

.943 ±.044

.681 ±.092

.948 ±.081

.956 ±.049

.935 ±.081

.949 ±.060

.663 ±.144

.664 ±.159

.947 ±.029

.854 ±.110

2

.932 ±.070

.956 ±.047

.961 ±.051

.969 ±.043

.567 ±.156

.949 ±.054

.745 ±.095

.953 ±.070

.965 ±.054

.958 ±.060

.964 ±.057

.784 ±.154

.781 ±.162

.958 ±.036

.871 ±.040

3

.941 ±.043

.955 ±.058

.955 ±.066

.957 ±.071

.676 ±.041

.964 ±.062

.791 ±.095

.955 ±.078

.963 ±.064

.961 ±.068

.961 ±.059

.851 ±.015

.810 ±.064

.962 ±.052

.910 ±.108

4

.953 ±.035

.957 ±.060

.967 ±.058

.964 ±.062

.841 ±.023

.969 ±.054

.926 ±.061

.949 ±.076

.962 ±.066

.957 ±.071

.963 ±.064

.930 ±.083

.925 ±.062

.961 ±.056

.931 ±.081

5

.960 ±.042

.960 ±.070

.964 ±.062

.963 ±.064

.926 ±.041

.971 ±.050

.925 ±.073

.958 ±.072

.962 ±.066

.962 ±.066

.964 ±.062

.963 ±.061

.962 ±.054

.961 ±.059

.965 ±.057

1

.895 ±.011

.915 ±.008

.915 ±.006

.834 ±.004

.642 ±.111

.909 ±.007

.901 ±.019

.838 ±.018

.911 ±.006

.826 ±.014

.880 ±.011

.777 ±.066

.908 ±.010

.909 ±.007

.911 ±.013

2

.915 ±.010

.914 ±.008

.926 ±.015

.854 ±.008

.757 ±.051

.912 ±.002

.916 ±.012

.847 ±.002

.914 ±.008

.862 ±.054

.895 ±.015

.844 ±.111

.916 ±.013

.912 ±.003

.915 ±.017

3

.920 ±.012

.918 ±.004

.920 ±.015

.867 ±.013

.899 ±.005

.923 ±.004

.919 ±.008

.863 ±.009

.918 ±.011

.894 ±.037

.917 ±.007

.862 ±.054

.919 ±.006

.929 ±.012

.907 ±.012

4

.927 ±.005

.927 ±.009

.930 ±.011

.886 ±.034

.916 ±.015

.931 ±.005

.927 ±.012

.869 ±.014

.930 ±.015

.923 ±.008

.926 ±.012

.878 ±.047

.925 ±.008

.928 ±.011

.921 ±.012

5

.928 ±.005

.930 ±.009

.932 ±.008

.929 ±.007

.920 ±.010

.929 ±.008

.925 ±.010

.915 ±.034

.929 ±.009

.930 ±.010

.930 ±.006

.899 ±.037

.926 ±.008

.930 ±.008

.922 ±.007

1

.671 ±.127

.831 ±.101

.818 ±.092

.816 ±.095

.612 ±.165

.836 ±.063

.711 ±.194

.790 ±.086

.805 ±.078

.811 ±.096

.811 ±.150

.689 ±.163

.827 ±.045

.802 ±.089

.840 ±.036

2

.767 ±.045

.857 ±.076

.838 ±.067

.813 ±.089

.855 ±.053

.857 ±.064

.722 ±.200

.783 ±.067

.847 ±.093

.824 ±.048

.867 ±.064

.800 ±.147

.841 ±.051

.757 ±.038

.864 ±.050

3

.800 ±.008

.837 ±.083

.843 ±.074

.796 ±.089

.861 ±.062

.836 ±.064

.756 ±.151

.800 ±.079

.844 ±.072

.839 ±.054

.835 ±.059

.818 ±.033

.840 ±.050

.797 ±.079

.864 ±.050

4

.825 ±.027

.848 ±.075

.843 ±.076

.796 ±.074

.851 ±.071

.846 ±.074

.853 ±.077

.795 ±.074

.845 ±.074

.848 ±.074

.835 ±.073

.792 ±.036

.844 ±.050

.824 ±.061

.866 ±.052

5

Dataset

Budget idx j 1

.891 ±.003

.901 ±.002

.829 ±.061

.844 ±.073

.844 ±.075

.827 ±.057

.842 ±.074

.838 ±.074

.844 ±.072

.808 ±.079

.839 ±.067

.840 ±.068

.839 ±.072

.827 ±.078

.836 ±.064

.835 ±.065

.836 ±.064

1

churn

.866 ±.010

.911 ±.008

.626 ±.002

.732 ±.001

.732 ±.005

.732 ±.005

.608 ±.021

.732 ±.001

.758 ±.002

.732 ±.005

.759 ±.003

.732 ±.005

.658 ±.009

.702 ±.035

.751 ±.012

.626 ±.002

.759 ±.003

2

aps_failure

.891 ±.003

.714 ±.101

.772 ±.003

.807 ±.012

.813 ±.003

.784 ±.002

.703 ±.012

.807 ±.012

.813 ±.013

.787 ±.003

.796 ±.003

.809 ±.008

.687 ±.003

.796 ±.019

.798 ±.023

.703 ±.007

.799 ±.003

3

labor negotiations

.677 ±.125

.815 ±.012

.779 ±.004

.824 ±.004

.826 ±.003

.788 ±.002

.740 ±.040

.824 ±.004

.826 ±.009

.799 ±.005

.826 ±.003

.826 ±.003

.788 ±.002

.797 ±.039

.825 ±.006

.779 ±.004

.811 ±.003

4

kdd_cup_09 appetency

.794 ±.011

.892 ±.012

.822 ±.012

.829 ±.004

.833 ±.002

.796 ±.004

.804 ±.023

.829 ±.004

.841 ±.003

.804 ±.004

.836 ±.003

.829 ±.004

.818 ±.020

.823 ±.012

.842 ±.005

.838 ±.010

.835 ±.004

5

qsar_oral toxicity

.858 ±.034

.881 ±.011

.845 ±.004

.840 ±.003

.842 ±.004

.833 ±.009

.838 ±.013

.840 ±.003

.845 ±.004

.829 ±.003

.846 ±.004

.840 ±.003

.843 ±.005

.846 ±.004

.845 ±.004

.845 ±.004

.844 ±.007

1

taiwanese_bankruptcy prediction

.866 ±.009

.859 ±.011

.520 ±.100

.514 ±.126

.460 ±.028

.591 ±.106

.470 ±.058

.514 ±.126

.486 ±.062

.403 ±.089

.568 ±.176

.554 ±.102

.490 ±.043

.493 ±.126

.497 ±.059

.532 ±.114

.519 ±.044

2

thyroid discordant

.821 ±.007

.810 ±.084

.489 ±.049

.501 ±.052

.482 ±.041

.558 ±.125

.470 ±.034

.501 ±.052

.547 ±.020

.517 ±.036

.516 ±.163

.576 ±.132

.530 ±.106

.543 ±.151

.542 ±.053

.571 ±.033

.480 ±.077

3

coil_2000 insurance_policies

.717 ±.011

.852 ±.088

.509 ±.069

.536 ±.036

.538 ±.041

.571 ±.137

.449 ±.136

.536 ±.036

.466 ±.055

.519 ±.057

.472 ±.127

.557 ±.087

.549 ±.045

.471 ±.088

.516 ±.067

.542 ±.063

.558 ±.121

4

prostate cancer

.769 ±.203

.732 ±.081

.509 ±.058

.539 ±.036

.524 ±.073

.521 ±.097

.523 ±.077

.539 ±.036

.488 ±.054

.542 ±.059

.518 ±.054

.548 ±.110

.505 ±.024

.543 ±.136

.581 ±.094

.516 ±.033

.513 ±.060

5

credit approval

.690 ±.008

.841 ±.015

.549 ±.111

.519 ±.071

.476 ±.019

.519 ±.080

.557 ±.094

.519 ±.071

.496 ±.076

.516 ±.070

.491 ±.052

.511 ±.078

.519 ±.066

.485 ±.078

.518 ±.078

.495 ±.049

.540 ±.104

1

heart_disease hungary

.812 ±.012

.904 ±.001

.894 ±.001

.897 ±.004

.891 ±.007

.884 ±.010

.678 ±.012

.897 ±.004

.920 ±.004

.908 ±.011

.894 ±.008

.918 ±.004

.708 ±.032

.693 ±.020

.922 ±.005

.897 ±.004

.881 ±.010

2

give_me some_credit

.866 ±.009

.903 ±.001

.902 ±.007

.929 ±.003

.905 ±.006

.911 ±.021

.712 ±.012

.929 ±.003

.926 ±.004

.932 ±.004

.909 ±.019

.925 ±.004

.862 ±.109

.708 ±.035

.928 ±.003

.902 ±.007

.927 ±.004

3

clock_protein toxicity

.875 ±.005

.905 ±.023

.930 ±.003

.930 ±.002

.933 ±.004

.933 ±.004

.751 ±.003

.931 ±.002

.927 ±.003

.932 ±.004

.931 ±.003

.926 ±.006

.927 ±.007

.905 ±.009

.930 ±.002

.930 ±.004

.930 ±.003

4

online_shoppers_purchasing intention_dataset

.828 ±.011

.931 ±.003

.931 ±.002

.931 ±.004

.931 ±.004

.929 ±.005

.931 ±.002

.929 ±.003

.932 ±.003

.931 ±.004

.933 ±.004

.933 ±.004

.924 ±.009

.931 ±.002

.931 ±.004

.930 ±.003

5

(continued from previous page)

.931 ±.004

.932 ±.004

.931 ±.005

.931 ±.004

.931 ±.004

.932 ±.004

.932 ±.003

.931 ±.004

.932 ±.004

.933 ±.004

.934 ±.004

.867 ±.083

.932 ±.003

.931 ±.004

.930 ±.004

53

CART

ElasticNet

F-test

GainRatio

LaplacianScore

Lasso

LOCO

MarkovBlanket

MI

mRMR

(R)ReliefF

Random

SBE

RFImportance

SFS

2

.768 ±.007

3

.777 ±.006

.776 ±.009

.778 ±.006

.779 ±.006

.758 ±.020

.775 ±.008

.780 ±.005

.782 ±.008

.779 ±.003

.777 ±.005

.782 ±.005

.770 ±.021

.779 ±.007

.776 ±.005

.775 ±.002

4

.781 ±.005

.781 ±.004

.781 ±.005

.781 ±.003

.775 ±.004

.780 ±.004

.782 ±.003

.782 ±.006

.781 ±.005

.779 ±.005

.784 ±.004

.772 ±.019

.782 ±.007

.781 ±.005

.779 ±.003

5

.782 ±.004

.784 ±.004

.784 ±.005

.784 ±.004

.784 ±.005

.783 ±.004

.783 ±.004

.784 ±.006

.783 ±.005

.784 ±.005

.783 ±.004

.773 ±.023

.784 ±.005

.784 ±.005

.781 ±.003

1

.806 ±.012

.760 ±.007

.778 ±.003

.670 ±.012

.507 ±.004

.771 ±.004

.615 ±.057

.774 ±.029

.805 ±.007

.741 ±.014

.763 ±.004

.545 ±.024

.550 ±.027

.796 ±.016

.760 ±.022

2

.829 ±.018

.799 ±.017

.815 ±.016

.735 ±.013

.506 ±.007

.814 ±.006

.717 ±.040

.832 ±.025

.842 ±.008

.776 ±.024

.801 ±.006

.721 ±.038

.664 ±.076

.835 ±.017

.785 ±.014

3

.843 ±.016

.844 ±.011

.839 ±.013

.826 ±.014

.514 ±.001

.840 ±.013

.808 ±.017

.847 ±.020

.849 ±.010

.812 ±.012

.836 ±.007

.760 ±.039

.717 ±.007

.850 ±.009

.823 ±.019

4

.857 ±.010

.860 ±.012

.854 ±.011

.852 ±.017

.514 ±.007

.859 ±.013

.831 ±.035

.855 ±.017

.856 ±.011

.846 ±.016

.850 ±.011

.825 ±.037

.823 ±.029

.858 ±.011

.840 ±.010

5

.862 ±.009

.863 ±.013

.861 ±.010

.860 ±.018

.768 ±.021

.861 ±.010

.859 ±.016

.860 ±.018

.861 ±.010

.860 ±.020

.861 ±.012

.856 ±.014

.854 ±.013

.865 ±.012

.856 ±.011

1

.879 ±.050

.833 ±.110

.838 ±.061

.830 ±.034

.518 ±.032

.842 ±.117

.843 ±.127

.788 ±.018

.882 ±.058

.900 ±.044

.864 ±.096

.773 ±.169

.847 ±.093

.886 ±.054

.851 ±.087

2

.924 ±.014

.858 ±.112

.889 ±.059

.871 ±.021

.722 ±.167

.856 ±.110

.880 ±.084

.835 ±.015

.901 ±.043

.869 ±.063

.894 ±.064

.710 ±.235

.901 ±.033

.927 ±.024

.860 ±.076

3

.916 ±.024

.877 ±.085

.891 ±.063

.896 ±.036

.853 ±.111

.874 ±.083

.877 ±.114

.885 ±.005

.902 ±.033

.874 ±.055

.899 ±.070

.865 ±.041

.905 ±.049

.914 ±.033

.857 ±.098

4

.913 ±.031

.908 ±.034

.913 ±.025

.902 ±.048

.900 ±.070

.908 ±.033

.910 ±.057

.905 ±.029

.896 ±.046

.872 ±.065

.919 ±.039

.882 ±.066

.893 ±.069

.921 ±.037

.857 ±.099

5

.918 ±.040

.912 ±.034

.916 ±.047

.895 ±.054

.896 ±.086

.909 ±.031

.905 ±.069

.896 ±.038

.902 ±.047

.896 ±.077

.917 ±.037

.913 ±.076

.886 ±.069

.916 ±.052

.866 ±.098

1

.987 ±.014

.985 ±.023

.978 ±.030

.980 ±.032

.558 ±.084

1.00 ±.00

.830 ±.067

.980 ±.035

.975 ±.028

.998 ±.001

.978 ±.030

.795 ±.180

.779 ±.127

.995 ±.009

.993 ±.002

2

.987 ±.013

1.00 ±.00

.999 ±.001

.999 ±.001

.755 ±.134

1.00 ±.00

.952 ±.023

.998 ±.003

.999 ±.001

.999 ±.001

.998 ±.002

.961 ±.032

.956 ±.059

.999 ±.001

.997 ±.004

3

.999 ±.001

1.00 ±.00

1.00 ±.00

1.00 ±.00

.805 ±.134

1.00 ±.00

.994 ±.003

1.00 ±.00

1.00 ±.00

1.00 ±.00

1.00 ±.00

.989 ±.011

.997 ±.001

1.00 ±.00

.998 ±.002

4

1.00 ±.00

1.00 ±.00

1.00 ±.00

1.00 ±.00

.943 ±.030

1.00 ±.00

.999 ±.001

1.00 ±.00

1.00 ±.00

1.00 ±.00

1.00 ±.00

.999 ±.001

.999 ±.001

1.00 ±.00

.999 ±.001

5

1.00 ±.00

1.00 ±.00

1.00 ±.00

1.00 ±.00

.970 ±.010

1.00 ±.00

1.00 ±.00

1.00 ±.00

1.00 ±.00

1.00 ±.00

1.00 ±.00

1.00 ±.00

1.00 ±.00

1.00 ±.00

1.00 ±.00

1

.578 ±.010

.609 ±.045

.629 ±.058

.712 ±.024

.664 ±.051

.609 ±.045

.651 ±.066

.606 ±.060

.681 ±.038

.682 ±.021

.613 ±.041

.615 ±.035

.689 ±.019

.578 ±.010

.706 ±.030

2

.690 ±.033

.578 ±.050

.643 ±.020

.709 ±.022

.658 ±.049

.578 ±.050

.633 ±.086

.668 ±.013

.715 ±.032

.700 ±.012

.628 ±.003

.567 ±.039

.706 ±.032

.690 ±.033

.715 ±.029

3

.696 ±.031

.644 ±.011

.638 ±.026

.700 ±.024

.653 ±.034

.644 ±.011

.698 ±.019

.698 ±.029

.716 ±.021

.699 ±.038

.616 ±.023

.685 ±.054

.686 ±.036

.689 ±.025

.701 ±.023

4

.707 ±.018

.653 ±.009

.654 ±.028

.700 ±.027

.676 ±.030

.653 ±.009

.700 ±.018

.699 ±.030

.722 ±.020

.713 ±.033

.653 ±.010

.680 ±.060

.701 ±.029

.701 ±.024

.698 ±.023

5

.709 ±.020

.710 ±.019

.710 ±.028

.731 ±.026

.685 ±.037

.710 ±.019

.706 ±.021

.709 ±.023

.718 ±.019

.717 ±.031

.670 ±.016

.718 ±.031

.701 ±.029

.730 ±.024

.706 ±.021

1

.931 ±.017

.917 ±.011

.916 ±.010

.916 ±.012

.881 ±.010

.914 ±.014

.921 ±.019

.922 ±.013

.917 ±.011

.916 ±.015

.927 ±.007

.861 ±.008

.925 ±.007

.928 ±.017

.916 ±.013

2

.956 ±.017

.950 ±.010

.946 ±.012

.950 ±.012

.913 ±.016

.950 ±.010

.959 ±.014

.951 ±.015

.949 ±.013

.947 ±.009

.953 ±.007

.923 ±.020

.959 ±.009

.953 ±.011

.953 ±.016

3

.967 ±.008

.960 ±.007

.964 ±.010

.957 ±.010

.939 ±.018

.961 ±.007

.964 ±.010

.966 ±.010

.957 ±.011

.957 ±.006

.966 ±.010

.949 ±.022

.969 ±.010

.963 ±.009

.969 ±.012

4

.970 ±.007

.967 ±.009

.965 ±.010

.967 ±.010

.954 ±.012

.967 ±.009

.968 ±.010

.969 ±.010

.967 ±.013

.965 ±.007

.971 ±.010

.965 ±.017

.971 ±.011

.970 ±.009

.972 ±.012

5

.971 ±.009

.972 ±.009

.967 ±.010

.970 ±.011

.963 ±.011

.972 ±.009

.971 ±.009

.972 ±.008

.970 ±.012

.972 ±.009

.972 ±.009

.968 ±.012

.971 ±.010

.971 ±.009

.974 ±.010

1

.657 ±.042

.646 ±.049

.594 ±.076

.636 ±.031

.564 ±.057

.651 ±.080

.670 ±.025

.689 ±.028

.679 ±.044

.670 ±.039

.605 ±.078

.646 ±.066

.584 ±.030

.681 ±.019

.648 ±.034

2

.707 ±.041

.701 ±.041

.622 ±.008

.717 ±.010

.605 ±.032

.652 ±.015

.704 ±.028

.723 ±.027

.682 ±.040

.701 ±.024

.635 ±.026

.697 ±.035

.692 ±.026

.693 ±.040

.703 ±.047

3

.745 ±.032

.752 ±.009

.726 ±.010

.779 ±.026

.628 ±.021

.729 ±.070

.754 ±.014

.798 ±.040

.727 ±.034

.742 ±.047

.745 ±.025

.721 ±.041

.759 ±.028

.785 ±.037

.739 ±.008

4

.787 ±.014

.791 ±.013

.760 ±.031

.773 ±.024

.720 ±.027

.775 ±.035

.769 ±.015

.796 ±.032

.784 ±.054

.769 ±.012

.749 ±.021

.737 ±.019

.740 ±.009

.780 ±.031

.763 ±.042

5

.803 ±.005

.806 ±.031

.773 ±.022

.784 ±.025

.766 ±.025

.788 ±.016

.763 ±.012

.777 ±.035

.778 ±.027

.772 ±.020

.786 ±.028

.773 ±.010

.783 ±.030

.787 ±.020

.792 ±.039

1

.491 ±.219

.459 ±.172

.488 ±.221

.435 ±.118

.625 ±.126

.459 ±.172

.373 ±.031

.535 ±.267

.500 ±.000

.476 ±.135

.623 ±.058

.449 ±.277

.434 ±.183

.358 ±.074

.396 ±.120

2

.380 ±.177

.520 ±.333

.515 ±.259

.574 ±.230

.572 ±.049

.520 ±.333

.459 ±.169

.563 ±.252

.615 ±.055

.656 ±.261

.625 ±.165

.459 ±.191

.449 ±.145

.365 ±.261

.394 ±.140

3

.540 ±.228

.440 ±.311

.548 ±.267

.507 ±.243

.489 ±.171

.440 ±.311

.478 ±.214

.446 ±.270

.511 ±.267

.526 ±.122

.521 ±.285

.557 ±.215

.439 ±.130

.376 ±.180

.441 ±.217

4

.432 ±.281

.494 ±.287

.520 ±.277

.424 ±.191

.471 ±.258

.494 ±.287

.537 ±.232

.483 ±.237

.561 ±.193

.573 ±.233

.552 ±.346

.480 ±.163

.401 ±.143

.483 ±.158

.480 ±.198

5

.487 ±.195

.493 ±.301

.466 ±.182

.497 ±.219

.485 ±.217

.493 ±.301

.475 ±.247

.439 ±.184

.493 ±.241

.583 ±.287

.475 ±.192

.554 ±.235

.490 ±.267

.514 ±.197

.511 ±.242

1

.721 ±.001

.785 ±.006

.785 ±.006

.768 ±.025

.645 ±.006

.785 ±.006

.687 ±.002

.785 ±.006

.719 ±.005

.785 ±.006

.720 ±.002

.580 ±.046

.785 ±.006

.721 ±.001

.783 ±.004

2

.784 ±.006

.793 ±.007

.791 ±.007

.783 ±.009

.670 ±.004

.793 ±.007

.695 ±.004

.787 ±.007

.787 ±.008

.794 ±.006

.765 ±.042

.721 ±.023

.794 ±.008

.728 ±.005

.792 ±.004

3

.789 ±.006

.795 ±.006

.793 ±.010

.791 ±.007

.696 ±.019

.795 ±.006

.734 ±.049

.792 ±.004

.791 ±.007

.794 ±.007

.793 ±.013

.794 ±.006

.799 ±.009

.789 ±.006

.795 ±.008

4

.796 ±.005

.798 ±.009

.799 ±.007

.800 ±.007

.707 ±.014

.798 ±.009

.766 ±.043

.800 ±.007

.801 ±.007

.797 ±.006

.799 ±.010

.793 ±.010

.801 ±.009

.795 ±.003

.799 ±.008

5

.800 ±.006

.801 ±.005

.801 ±.010

.802 ±.007

.803 ±.007

.801 ±.005

.799 ±.011

.802 ±.007

.803 ±.007

.801 ±.006

.801 ±.010

.799 ±.008

.802 ±.007

.800 ±.006

.803 ±.006

1

.609 ±.004

.617 ±.003

.606 ±.004

.606 ±.002

.549 ±.003

.607 ±.013

.602 ±.010

.616 ±.003

.596 ±.002

.600 ±.006

.600 ±.001

.596 ±.007

.594 ±.009

.606 ±.001

.587 ±.003

2

.612 ±.001

.628 ±.006

.623 ±.005

.625 ±.001

.598 ±.002

.619 ±.013

.614 ±.001

.626 ±.006

.618 ±.002

.617 ±.004

.610 ±.001

.614 ±.005

.614 ±.003

.612 ±.001

.605 ±.008

3

.617 ±.001

.634 ±.005

.630 ±.002

.629 ±.007

.611 ±.003

.628 ±.007

.619 ±.003

.622 ±.001

.627 ±.001

.620 ±.003

.615 ±.002

.616 ±.004

.622 ±.009

.622 ±.001

.618 ±.007

4

.628 ±.001

.637 ±.002

.634 ±.001

.625 ±.002

.624 ±.004

.632 ±.005

.622 ±.007

.625 ±.002

.628 ±.000

.629 ±.002

.620 ±.000

.627 ±.007

.624 ±.009

.626 ±.001

.624 ±.001

5

.630 ±.000

.636 ±.005

.631 ±.000

.629 ±.001

.624 ±.002

.633 ±.004

.628 ±.001

.628 ±.002

.630 ±.001

.640 ±.001

.629 ±.000

.629 ±.000

.628 ±.003

.631 ±.000

.630 ±.008

1

.814 ±.071

.751 ±.157

.816 ±.074

.804 ±.027

.431 ±.068

.814 ±.071

.683 ±.154

.829 ±.066

.804 ±.029

.796 ±.031

.610 ±.191

.577 ±.191

.675 ±.137

.844 ±.051

.698 ±.156

2

.822 ±.077

.741 ±.143

.827 ±.065

.798 ±.015

.479 ±.063

.808 ±.049

.679 ±.152

.801 ±.021

.790 ±.018

.778 ±.060

.839 ±.043

.707 ±.069

.667 ±.126

.819 ±.040

.693 ±.165

3

.828 ±.069

.794 ±.013

.816 ±.042

.799 ±.022

.528 ±.087

.820 ±.040

.689 ±.162

.794 ±.014

.789 ±.038

.793 ±.046

.795 ±.039

.685 ±.045

.677 ±.109

.800 ±.022

.693 ±.165

4

.813 ±.016

.798 ±.023

.828 ±.059

.790 ±.017

.713 ±.076

.786 ±.002

.686 ±.160

.796 ±.027

.795 ±.017

.806 ±.035

.790 ±.032

.794 ±.031

.688 ±.114

.794 ±.015

.685 ±.137

5

Dataset

Budget idx j 1

.748 ±.011

.770 ±.007

.796 ±.010

.784 ±.019

.805 ±.027

.792 ±.012

.837 ±.067

.790 ±.028

.702 ±.125

.792 ±.013

.799 ±.019

.787 ±.013

.791 ±.033

.781 ±.022

.702 ±.125

.800 ±.025

.702 ±.125

1

credit_card clients_default

.763 ±.004

.774 ±.005

.955 ±.022

.929 ±.060

.951 ±.022

.963 ±.019

.913 ±.055

.921 ±.070

.937 ±.049

.946 ±.025

.945 ±.034

.955 ±.031

.952 ±.025

.923 ±.013

.945 ±.032

.962 ±.008

.962 ±.015

2

bioresponse

.762 ±.002

.740 ±.041

.956 ±.020

.947 ±.035

.968 ±.023

.960 ±.026

.927 ±.054

.946 ±.037

.943 ±.043

.959 ±.031

.964 ±.029

.966 ±.016

.960 ±.027

.954 ±.039

.952 ±.033

.962 ±.029

.959 ±.029

3

lymphography

.696 ±.030

.774 ±.005

.962 ±.022

.960 ±.031

.959 ±.023

.968 ±.016

.939 ±.051

.961 ±.033

.947 ±.035

.962 ±.033

.967 ±.025

.966 ±.023

.964 ±.021

.959 ±.030

.951 ±.044

.963 ±.030

.959 ±.027

4

nci_ovarian cancer

.750 ±.003

.763 ±.004

.965 ±.020

.968 ±.025

.957 ±.026

.970 ±.018

.955 ±.030

.965 ±.030

.955 ±.037

.967 ±.029

.967 ±.026

.965 ±.026

.966 ±.023

.960 ±.028

.951 ±.046

.964 ±.024

.958 ±.028

5

bad_customer detection

.749 ±.004

.770 ±.007

.964 ±.025

.968 ±.024

.964 ±.025

.966 ±.027

.963 ±.024

.966 ±.025

.965 ±.024

.970 ±.027

.965 ±.026

.969 ±.023

.966 ±.023

.968 ±.020

.967 ±.026

.965 ±.024

.963 ±.022

1

hazelnut_spread contaminant_detection_10GHz

.743 ±.004

.777 ±.005

.664 ±.042

.684 ±.044

.688 ±.064

.663 ±.063

.623 ±.035

.688 ±.035

.656 ±.071

.660 ±.107

.694 ±.090

.672 ±.065

.688 ±.033

.559 ±.027

.507 ±.021

.716 ±.055

.559 ±.076

2

drug_induced autoimmunity_prediction

.763 ±.007

.771 ±.009

.737 ±.025

.752 ±.031

.694 ±.056

.731 ±.044

.708 ±.041

.728 ±.050

.670 ±.030

.685 ±.029

.733 ±.078

.744 ±.018

.735 ±.039

.661 ±.072

.654 ±.084

.720 ±.052

.673 ±.061

3

heart_disease switzerland

.761 ±.005

.769 ±.009

.737 ±.021

.764 ±.015

.728 ±.026

.719 ±.035

.731 ±.027

.733 ±.059

.710 ±.034

.728 ±.009

.715 ±.061

.728 ±.049

.757 ±.037

.695 ±.066

.707 ±.021

.747 ±.063

.711 ±.055

4

hr analytics

.756 ±.008

.729 ±.004

.729 ±.034

.764 ±.024

.744 ±.027

.744 ±.018

.722 ±.021

.751 ±.014

.724 ±.088

.742 ±.027

.716 ±.061

.743 ±.027

.752 ±.016

.748 ±.040

.741 ±.065

.755 ±.024

.742 ±.061

5

porto seguro

.682 ±.003

.778 ±.007

.730 ±.062

.741 ±.026

.735 ±.037

.745 ±.021

.725 ±.034

.740 ±.032

.717 ±.041

.745 ±.036

.735 ±.036

.740 ±.029

.742 ±.027

.739 ±.052

.715 ±.040

.752 ±.038

.715 ±.032

1

japanese_credit screening

.742 ±.003

.777 ±.006

.588 ±.055

.674 ±.045

.593 ±.131

.572 ±.036

.501 ±.120

.582 ±.132

.512 ±.036

.541 ±.070

.512 ±.097

.531 ±.052

.579 ±.056

.509 ±.010

.494 ±.028

.579 ±.026

.490 ±.084

2

ionosphere

.742 ±.003

.773 ±.010

.591 ±.039

.731 ±.035

.638 ±.039

.580 ±.031

.512 ±.108

.690 ±.027

.529 ±.023

.597 ±.038

.565 ±.039

.663 ±.029

.626 ±.078

.559 ±.038

.577 ±.016

.630 ±.044

.513 ±.034

3

smoking lung_cancer

.756 ±.008

.772 ±.003

.610 ±.024

.719 ±.039

.650 ±.030

.558 ±.053

.542 ±.076

.707 ±.056

.574 ±.072

.586 ±.036

.561 ±.075

.655 ±.028

.638 ±.044

.587 ±.042

.653 ±.046

.637 ±.054

.657 ±.072

4

nci_pancreatic cancer

.758 ±.005

.631 ±.036

.705 ±.066

.683 ±.025

.614 ±.062

.540 ±.077

.701 ±.074

.615 ±.011

.602 ±.028

.595 ±.055

.677 ±.070

.613 ±.041

.597 ±.007

.597 ±.019

.660 ±.044

.589 ±.038

5

(continued from previous page)

.651 ±.028

.695 ±.066

.655 ±.067

.597 ±.010

.517 ±.041

.659 ±.053

.628 ±.023

.578 ±.022

.596 ±.066

.651 ±.051

.620 ±.065

.616 ±.064

.609 ±.059

.661 ±.022

.612 ±.021

54

CART

ElasticNet

F-test

GainRatio

LaplacianScore

Lasso

LOCO

MarkovBlanket

MI

mRMR

(R)ReliefF

Random

SBE

RFImportance

SFS

.799 ±.045

.880 ±.019

.891 ±.001

.902 ±.001

.862 ±.037

.603 ±.038

.605 ±.022

.903 ±.001

.848 ±.015

2

.929 ±.001

.856 ±.048

.920 ±.001

.909 ±.001

.908 ±.001

.920 ±.001

.725 ±.062

.725 ±.052

.912 ±.001

.871 ±.008

3

.957 ±.002

.951 ±.001

.910 ±.001

.916 ±.000

.676 ±.001

.950 ±.000

.924 ±.036

.932 ±.000

.928 ±.002

.923 ±.001

.939 ±.001

.774 ±.056

.780 ±.054

.956 ±.001

.904 ±.003

4

.962 ±.001

.962 ±.000

.955 ±.000

.932 ±.000

.702 ±.001

.959 ±.003

.944 ±.019

.936 ±.002

.941 ±.000

.939 ±.005

.946 ±.001

.867 ±.025

.855 ±.009

.961 ±.001

.905 ±.008

5

.965 ±.001

.966 ±.000

.961 ±.000

.941 ±.000

.754 ±.006

.966 ±.000

.931 ±.014

.945 ±.002

.965 ±.000

.964 ±.000

.960 ±.001

.947 ±.010

.926 ±.026

.962 ±.001

.944 ±.006

1

.575 ±.067

.646 ±.128

.647 ±.141

.611 ±.128

.671 ±.115

.646 ±.128

.620 ±.075

.628 ±.112

.618 ±.118

.609 ±.111

.605 ±.115

.575 ±.055

.618 ±.115

.564 ±.061

.619 ±.075

2

.611 ±.110

.666 ±.130

.617 ±.065

.654 ±.119

.648 ±.104

.666 ±.130

.605 ±.067

.611 ±.075

.651 ±.109

.609 ±.111

.582 ±.114

.578 ±.070

.663 ±.098

.557 ±.031

.614 ±.068

3

.604 ±.071

.642 ±.130

.621 ±.068

.630 ±.101

.696 ±.142

.642 ±.130

.640 ±.099

.659 ±.103

.660 ±.113

.639 ±.171

.631 ±.111

.659 ±.057

.665 ±.099

.621 ±.065

.628 ±.083

4

.660 ±.093

.639 ±.134

.649 ±.082

.647 ±.089

.702 ±.138

.639 ±.134

.665 ±.097

.646 ±.101

.667 ±.089

.666 ±.176

.650 ±.111

.630 ±.105

.675 ±.099

.648 ±.089

.668 ±.094

5

.648 ±.087

.633 ±.076

.666 ±.098

.644 ±.092

.694 ±.099

.633 ±.076

.652 ±.081

.644 ±.092

.675 ±.083

.652 ±.090

.645 ±.092

.635 ±.097

.673 ±.101

.644 ±.092

.664 ±.097

1

.899 ±.018

.885 ±.014

.856 ±.017

.888 ±.011

.660 ±.054

.873 ±.011

.880 ±.026

.910 ±.007

.889 ±.028

.908 ±.009

.902 ±.003

.874 ±.024

.899 ±.015

.873 ±.015

.911 ±.023

2

.921 ±.005

.920 ±.006

.910 ±.014

.909 ±.006

.884 ±.012

.906 ±.008

.912 ±.009

.918 ±.004

.913 ±.011

.910 ±.008

.921 ±.011

.896 ±.016

.917 ±.013

.910 ±.011

.916 ±.019

3

.920 ±.008

.922 ±.015

.921 ±.008

.923 ±.011

.910 ±.011

.917 ±.014

.918 ±.008

.921 ±.004

.922 ±.010

.917 ±.011

.919 ±.009

.919 ±.006

.924 ±.004

.922 ±.009

.923 ±.016

4

.921 ±.006

.925 ±.014

.925 ±.012

.925 ±.007

.915 ±.008

.924 ±.015

.920 ±.008

.924 ±.005

.925 ±.008

.929 ±.007

.926 ±.008

.926 ±.010

.924 ±.006

.923 ±.007

.925 ±.009

5

.924 ±.008

.925 ±.008

.926 ±.008

.925 ±.007

.922 ±.012

.925 ±.011

.925 ±.007

.926 ±.007

.925 ±.008

.930 ±.010

.926 ±.008

.924 ±.007

.925 ±.007

.926 ±.008

.928 ±.007

1

.979 ±.000

.977 ±.004

.978 ±.001

.849 ±.052

.581 ±.001

.969 ±.011

.973 ±.005

.782 ±.092

.974 ±.006

.849 ±.052

.972 ±.001

.885 ±.069

.849 ±.075

.979 ±.001

.973 ±.005

2

.982 ±.001

.982 ±.002

.981 ±.002

.936 ±.002

.633 ±.003

.980 ±.004

.978 ±.002

.877 ±.047

.981 ±.001

.936 ±.002

.979 ±.000

.886 ±.012

.932 ±.019

.983 ±.001

.975 ±.006

3

.983 ±.001

.984 ±.002

.983 ±.002

.956 ±.009

.724 ±.010

.983 ±.001

.980 ±.003

.954 ±.005

.983 ±.002

.956 ±.009

.982 ±.001

.950 ±.008

.949 ±.022

.984 ±.001

.980 ±.002

4

Dataset

Budget idx j 1

.816 ±.002

.611 ±.091

.984 ±.001

.985 ±.002

.984 ±.002

.978 ±.005

.822 ±.002

.984 ±.001

.981 ±.003

.967 ±.001

.984 ±.002

.978 ±.005

.984 ±.001

.957 ±.029

.960 ±.033

.985 ±.001

.982 ±.001

5

homesite_quote conversion

.502 ±.000

.871 ±.030

.985 ±.001

.985 ±.001

.984 ±.002

.983 ±.001

.954 ±.007

.985 ±.001

.982 ±.002

.981 ±.003

.984 ±.001

.983 ±.001

.984 ±.001

.977 ±.005

.979 ±.008

.985 ±.001

.984 ±.003

1

heart_disease va_long_beach

.711 ±.091

.885 ±.001

.600 ±.059

.676 ±.033

.658 ±.012

.666 ±.016

.558 ±.021

.674 ±.033

.662 ±.013

.648 ±.022

.661 ±.017

.671 ±.008

.659 ±.019

.574 ±.021

.665 ±.004

.625 ±.031

.673 ±.008

2

qsar_biodeg

.875 ±.001

.929 ±.001

.656 ±.025

.686 ±.022

.663 ±.015

.664 ±.016

.589 ±.017

.686 ±.022

.683 ±.021

.654 ±.023

.660 ±.017

.688 ±.016

.674 ±.018

.630 ±.018

.690 ±.015

.673 ±.009

.677 ±.015

3

naticusdroid_android permissions_dataset

.816 ±.002

.944 ±.005

.688 ±.022

.691 ±.016

.689 ±.014

.683 ±.024

.642 ±.020

.696 ±.017

.696 ±.013

.682 ±.022

.670 ±.017

.695 ±.019

.681 ±.020

.657 ±.020

.696 ±.014

.688 ±.022

.685 ±.004

4

framingham heart_study

.902 ±.001

.697 ±.021

.694 ±.016

.696 ±.017

.689 ±.021

.689 ±.021

.694 ±.016

.695 ±.015

.689 ±.021

.681 ±.020

.693 ±.020

.689 ±.021

.690 ±.021

.695 ±.019

.699 ±.022

.693 ±.013

5

(continued from previous page)

.703 ±.022

.698 ±.022

.701 ±.022

.703 ±.021

.695 ±.021

.696 ±.020

.700 ±.020

.695 ±.016

.696 ±.026

.696 ±.020

.698 ±.019

.682 ±.031

.697 ±.019

.703 ±.022

.697 ±.018

CART

ElasticNet

F-test

GainRatio

LaplacianScore

Lasso

LOCO

MarkovBlanket

MI

mRMR

(R)ReliefF

Random

SBE

RFImportance

SFS

.741 ±.061

.804 ±.050

.836 ±.053

.737 ±.075

.796 ±.070

.770 ±.030

.762 ±.088

.786 ±.061

.791 ±.050

.838 ±.068

.826 ±.119

.765 ±.067

.865 ±.040

2

.791 ±.093

.766 ±.049

.769 ±.044

.861 ±.106

.769 ±.066

.790 ±.068

.731 ±.054

.803 ±.100

.763 ±.040

.774 ±.047

.841 ±.112

.808 ±.105

.720 ±.050

.865 ±.085

3

.786 ±.059

.814 ±.085

.796 ±.016

.826 ±.048

.874 ±.080

.797 ±.085

.811 ±.121

.783 ±.045

.817 ±.108

.778 ±.041

.827 ±.030

.771 ±.047

.795 ±.023

.787 ±.018

.901 ±.085

4

.868 ±.080

.878 ±.078

.870 ±.032

.857 ±.053

.812 ±.053

.878 ±.065

.889 ±.184

.843 ±.066

.840 ±.018

.848 ±.024

.867 ±.013

.889 ±.037

.803 ±.017

.859 ±.014

.949 ±.109

5

.899 ±.033

.904 ±.048

.920 ±.034

.876 ±.018

.855 ±.042

.908 ±.069

.929 ±.142

.882 ±.017

.896 ±.047

.871 ±.041

.906 ±.037

.893 ±.107

.816 ±.015

.904 ±.019

.970 ±.100

1

.659 ±.134

.273 ±.134

.496 ±.202

.561 ±.111

1.33 ±.13

.441 ±.189

1.28 ±.13

.730 ±.098

.538 ±.017

.465 ±.243

.362 ±.108

.958 ±.411

1.12 ±.36

.420 ±.149

1.15 ±.23

2

.618 ±.237

.119 ±.054

.267 ±.131

.233 ±.139

1.02 ±.28

.150 ±.040

.682 ±.272

.235 ±.112

.207 ±.104

.236 ±.088

.161 ±.020

1.53 ±.23

1.11 ±.25

.130 ±.012

1.09 ±.17

3

.396 ±.020

.0647 ±.0476

.0779 ±.0459

.0636 ±.0368

.713 ±.122

.0865 ±.0745

.423 ±.164

.0987 ±.0800

.0787 ±.0455

.0696 ±.0426

.0542 ±.0247

.374 ±.169

.398 ±.182

.0672 ±.0087

.391 ±.036

4

.160 ±.032

.0458 ±.0184

.0424 ±.0118

.0483 ±.0121

.428 ±.144

.0983 ±.0523

.193 ±.074

.0492 ±.0222

.0362 ±.0074

.0503 ±.0104

.0417 ±.0190

.250 ±.076

.189 ±.091

.0590 ±.0184

.280 ±.034

5

.133 ±.050

.0618 ±.0056

.0674 ±.0090

.0589 ±.0059

.186 ±.026

.0917 ±.0277

.132 ±.022

.0528 ±.0105

.0571 ±.0119

.0558 ±.0100

.0607 ±.0168

.120 ±.033

.112 ±.020

.0734 ±.0034

.120 ±.020

1

.170 ±.003

.187 ±.014

.223 ±.004

.228 ±.003

.732 ±.001

.352 ±.311

.172 ±.001

.172 ±.001

.207 ±.002

.232 ±.003

.221 ±.002

.606 ±.397

.170 ±.003

.228 ±.003

.207 ±.003

2

.158 ±.005

.161 ±.004

.171 ±.003

.159 ±.002

.624 ±.098

.159 ±.004

.159 ±.005

.162 ±.006

.147 ±.004

.219 ±.003

.171 ±.003

.181 ±.032

.157 ±.003

.160 ±.003

.158 ±.003

3

.142 ±.003

.156 ±.002

.163 ±.008

.142 ±.003

.581 ±.035

.156 ±.002

.155 ±.002

.142 ±.003

.142 ±.003

.170 ±.005

.157 ±.002

.641 ±.094

.155 ±.003

.147 ±.008

.145 ±.004

4

.142 ±.003

.154 ±.002

.160 ±.005

.141 ±.003

.279 ±.213

.154 ±.002

.150 ±.009

.141 ±.003

.142 ±.003

.152 ±.005

.143 ±.002

.415 ±.227

.154 ±.003

.142 ±.003

.142 ±.003

5

.142 ±.003

.155 ±.002

.151 ±.009

.142 ±.003

.143 ±.004

.155 ±.002

.141 ±.003

.142 ±.002

.142 ±.003

.142 ±.003

.142 ±.002

.275 ±.231

.141 ±.003

.142 ±.003

.141 ±.003

1

.658 ±.031

.610 ±.017

.704 ±.015

.692 ±.010

.977 ±.022

.615 ±.018

.646 ±.037

.616 ±.003

.664 ±.019

.734 ±.019

.661 ±.020

.731 ±.045

.727 ±.036

.725 ±.030

.713 ±.020

2

.588 ±.006

.579 ±.020

.587 ±.014

.611 ±.015

.891 ±.022

.580 ±.021

.619 ±.025

.602 ±.010

.576 ±.024

.689 ±.012

.640 ±.006

.669 ±.034

.660 ±.077

.607 ±.013

.669 ±.020

3

.575 ±.010

.566 ±.013

.575 ±.023

.603 ±.009

.740 ±.019

.569 ±.009

.582 ±.025

.601 ±.004

.564 ±.018

.634 ±.015

.572 ±.017

.705 ±.015

.621 ±.058

.589 ±.015

.611 ±.029

4

.569 ±.023

.565 ±.013

.570 ±.014

.602 ±.009

.624 ±.018

.567 ±.014

.575 ±.029

.602 ±.009

.566 ±.020

.623 ±.011

.569 ±.017

.615 ±.017

.574 ±.021

.564 ±.021

.594 ±.052

5

.566 ±.019

.566 ±.018

.564 ±.016

.586 ±.024

.570 ±.016

.566 ±.020

.576 ±.031

.599 ±.013

.571 ±.018

.595 ±.017

.572 ±.015

.582 ±.036

.569 ±.026

.565 ±.022

.575 ±.012

1

1.61 ±.06

1.74 ±.10

1.71 ±.07

2.03 ±.15

2.12 ±.03

1.85 ±.23

1.78 ±.11

1.72 ±.05

1.69 ±.06

1.73 ±.04

1.71 ±.05

2.07 ±.14

2.11 ±.04

1.72 ±.05

1.84 ±.08

2

1.61 ±.10

1.66 ±.06

1.67 ±.09

1.83 ±.15

2.12 ±.03

1.80 ±.14

1.75 ±.10

1.73 ±.06

1.62 ±.09

1.75 ±.07

1.65 ±.07

2.13 ±.05

2.14 ±.06

1.74 ±.04

1.85 ±.08

3

1.76 ±.20

1.74 ±.07

1.63 ±.05

1.74 ±.06

2.04 ±.10

1.69 ±.10

1.80 ±.08

1.78 ±.04

1.67 ±.09

1.87 ±.06

1.68 ±.08

2.17 ±.10

2.06 ±.11

1.82 ±.09

1.82 ±.02

4

1.93 ±.10

1.86 ±.09

1.78 ±.07

1.91 ±.12

1.99 ±.03

1.75 ±.07

1.82 ±.07

2.08 ±.12

1.90 ±.12

2.11 ±.16

1.85 ±.20

2.05 ±.28

2.13 ±.05

1.91 ±.09

1.97 ±.04

5

2.03 ±.03

2.04 ±.08

1.89 ±.04

2.22 ±.15

1.95 ±.15

1.93 ±.03

2.00 ±.08

2.23 ±.17

1.98 ±.11

2.21 ±.14

2.03 ±.05

2.38 ±.13

2.12 ±.23

2.07 ±.08

2.19 ±.04

1

1.07 ±.00

1.08 ±.01

1.16 ±.02

1.34 ±.01

1.25 ±.02

1.13 ±.01

1.25 ±.05

1.07 ±.00

1.19 ±.02

1.11 ±.01

1.26 ±.05

1.40 ±.06

1.38 ±.06

1.07 ±.00

1.36 ±.12

2

.899 ±.007

.909 ±.002

.964 ±.018

1.10 ±.01

1.01 ±.02

.924 ±.019

1.12 ±.10

.910 ±.003

.981 ±.003

.970 ±.001

1.02 ±.00

1.14 ±.02

1.11 ±.03

.911 ±.003

3

.828 ±.007

.818 ±.001

.847 ±.003

.881 ±.003

.852 ±.005

.825 ±.002

1.01 ±.07

.830 ±.002

.884 ±.005

.873 ±.004

.905 ±.009

.976 ±.021

.971 ±.051

.828 ±.005

.987 ±.035

.790 ±.004

.780 ±.001

.796 ±.001

.817 ±.003

.804 ±.004

.787 ±.001

.878 ±.014

.791 ±.002

.798 ±.003

.798 ±.002

.828 ±.004

.889 ±.016

.890 ±.045

.783 ±.002

.895 ±.032

.766 ±.002

.764 ±.001

.769 ±.006

.769 ±.001

.768 ±.003

.764 ±.002

.795 ±.005

.765 ±.003

.766 ±.002

.770 ±.001

.786 ±.003

.813 ±.021

.835 ±.003

.765 ±.002

.826 ±.005

Budget idx j

.737 ±.075

.763 ±.070

4

Dataset

1

.767 ±.068

5

Table D.6: Per-cell results with downstream model Linear. Tasks are grouped by metric. Each cell is the mean over three seeds, with the sample standard deviation as a subscript.

otto_group_product classification_challenge

ancestry study

predict_students_dropout and_academic_success

sdss_17

srbct prediction

horse_colic survival

log-loss (↓)

55

1.17 ±.06

CART

ElasticNet

F-test

GainRatio

LaplacianScore

Lasso

LOCO

MarkovBlanket

MI

mRMR

(R)ReliefF

Random

SBE

RFImportance

SFS

.348 ±.085

.411 ±.088

.316 ±.054

.378 ±.034

.761 ±.080

.737 ±.106

.340 ±.101

.615 ±.149

2 3

.185 ±.057

4

.176 ±.017

5

.154 ±.026

1

.917 ±.048

2

.598 ±.034

3

.595 ±.034

4

.535 ±.039

5

.496 ±.068

1

.489 ±.007

2

.486 ±.019

3

.504 ±.022

4

.532 ±.023

5

.557 ±.055

1

1.74 ±.09

2

1.63 ±.14

3

1.46 ±.15

4

1.41 ±.19

5

1.41 ±.16

1

.144 ±.008

2

.110 ±.010

3

.106 ±.017

4

.102 ±.018

.100 ±.020

.104 ±.026

.104 ±.027

.251 ±.046

.0924 ±.0173

.103 ±.031

.0942 ±.0177

.107 ±.026

.104 ±.027

.0997 ±.0176

.212 ±.072

.0931 ±.0216

.0906 ±.0176

.111 ±.025

5

.104 ±.023

.0913 ±.0267

.0900 ±.0182

.0968 ±.0207

.0854 ±.0216

.0847 ±.0228

.0941 ±.0365

.0928 ±.0226

.0934 ±.0216

.0968 ±.0207

.0921 ±.0265

.0932 ±.0224

.0915 ±.0178

.0869 ±.0221

.101 ±.028

1

.443 ±.157

.477 ±.130

.454 ±.122

.444 ±.147

.536 ±.077

.538 ±.073

.533 ±.049

.463 ±.117

.407 ±.130

.489 ±.107

.430 ±.120

.520 ±.061

.491 ±.137

.403 ±.132

.428 ±.092

2

.418 ±.189

.427 ±.139

.370 ±.159

.427 ±.184

.427 ±.130

.540 ±.028

.526 ±.093

.416 ±.137

.359 ±.123

.435 ±.133

.409 ±.124

.461 ±.215

.449 ±.098

.393 ±.213

.369 ±.128

3

.352 ±.246

.367 ±.135

.333 ±.138

.403 ±.165

.372 ±.093

.414 ±.103

.472 ±.117

.408 ±.195

.375 ±.176

.358 ±.098

.355 ±.154

.416 ±.125

.385 ±.048

.316 ±.180

.367 ±.143

4

.303 ±.187

.325 ±.149

.325 ±.141

.324 ±.138

.358 ±.126

.337 ±.137

.443 ±.128

.345 ±.103

.355 ±.190

.312 ±.105

.359 ±.184

.338 ±.109

.376 ±.054

.293 ±.185

.373 ±.163

5

.300 ±.212

.317 ±.161

.319 ±.157

.318 ±.196

.336 ±.162

.308 ±.158

.385 ±.171

.300 ±.138

.343 ±.229

.306 ±.187

.332 ±.194

.354 ±.160

.339 ±.129

.293 ±.187

.358 ±.169

1

1.35 ±.02

1.44 ±.03

1.39 ±.03

1.53 ±.02

1.41 ±.03

1.45 ±.02

1.46 ±.05

1.53 ±.01

1.40 ±.04

1.53 ±.02

1.50 ±.01

1.53 ±.01

1.53 ±.01

1.35 ±.03

1.39 ±.01

2

1.29 ±.02

1.34 ±.02

1.37 ±.04

1.51 ±.01

1.38 ±.03

1.34 ±.02

1.41 ±.05

1.49 ±.05

1.35 ±.03

1.51 ±.01

1.43 ±.05

1.51 ±.02

1.52 ±.00

1.28 ±.01

1.37 ±.01

3

1.24 ±.03

1.23 ±.02

1.26 ±.02

1.45 ±.03

1.31 ±.04

1.21 ±.02

1.36 ±.02

1.42 ±.02

1.29 ±.03

1.45 ±.03

1.38 ±.04

1.42 ±.03

1.41 ±.05

1.21 ±.01

1.36 ±.02

4

1.14 ±.01

1.07 ±.02

1.14 ±.02

1.31 ±.02

1.20 ±.02

1.06 ±.02

1.27 ±.03

1.30 ±.01

1.18 ±.02

1.31 ±.02

1.21 ±.03

1.28 ±.06

1.30 ±.06

1.10 ±.02

1.24 ±.01

5

1.04 ±.00

.962 ±.022

1.02 ±.02

1.09 ±.03

1.04 ±.02

.963 ±.024

1.12 ±.01

1.13 ±.02

1.04 ±.04

1.09 ±.03

1.04 ±.00

1.10 ±.01

1.09 ±.02

1.00 ±.02

1.11 ±.01

1

1.41 ±.02

1.41 ±.02

1.41 ±.02

1.40 ±.02

1.41 ±.05

1.41 ±.02

1.41 ±.03

1.41 ±.02

1.41 ±.05

1.40 ±.03

1.40 ±.02

1.40 ±.02

1.41 ±.02

1.41 ±.02

1.40 ±.03

2

1.40 ±.02

1.42 ±.02

1.43 ±.04

1.40 ±.02

1.41 ±.05

1.42 ±.02

1.41 ±.02

1.40 ±.02

1.41 ±.06

1.41 ±.05

1.41 ±.01

1.41 ±.02

1.41 ±.02

1.41 ±.02

1.42 ±.06

3

1.41 ±.02

1.41 ±.02

1.42 ±.04

1.42 ±.05

1.42 ±.05

1.41 ±.02

1.43 ±.04

1.41 ±.03

1.40 ±.05

1.42 ±.07

1.40 ±.04

1.42 ±.03

1.42 ±.02

1.40 ±.01

1.42 ±.08

4

1.41 ±.02

1.41 ±.03

1.43 ±.03

1.42 ±.04

1.42 ±.05

1.41 ±.03

1.41 ±.05

1.41 ±.03

1.40 ±.05

1.42 ±.05

1.41 ±.04

1.42 ±.04

1.44 ±.03

1.42 ±.02

1.42 ±.07

5

1.42 ±.06

1.42 ±.03

1.43 ±.04

1.42 ±.05

1.42 ±.05

1.42 ±.03

1.42 ±.05

1.41 ±.04

1.41 ±.04

1.42 ±.06

1.42 ±.04

1.43 ±.05

1.43 ±.05

1.43 ±.03

1.42 ±.06

1

.679 ±.002

.681 ±.002

.677 ±.002

.712 ±.006

.811 ±.002

.681 ±.002

.672 ±.001

.702 ±.003

.672 ±.002

.722 ±.010

.695 ±.003

.784 ±.024

.787 ±.024

.679 ±.002

.668 ±.004

2

.655 ±.001

.655 ±.001

.662 ±.002

.662 ±.006

.808 ±.003

.655 ±.001

.648 ±.011

.654 ±.001

.654 ±.001

.685 ±.014

.656 ±.010

.789 ±.012

.797 ±.018

.658 ±.004

.656 ±.008

3

.608 ±.002

.619 ±.004

.649 ±.001

.619 ±.012

.807 ±.002

.619 ±.004

.625 ±.012

.643 ±.001

.630 ±.001

.644 ±.002

.643 ±.013

.740 ±.027

.746 ±.018

.609 ±.002

.639 ±.010

4

.585 ±.002

.592 ±.002

.628 ±.008

.598 ±.003

.778 ±.009

.592 ±.002

.590 ±.008

.634 ±.001

.596 ±.005

.633 ±.004

.606 ±.009

.688 ±.037

.686 ±.031

.585 ±.001

.635 ±.011

5

.572 ±.000

.571 ±.001

.583 ±.004

.584 ±.002

.699 ±.008

.571 ±.001

.571 ±.005

.579 ±.001

.571 ±.001

.616 ±.009

.588 ±.006

.623 ±.014

.634 ±.022

.572 ±.001

.602 ±.003

1

.322 ±.020

.304 ±.003

.353 ±.004

.356 ±.004

.536 ±.007

.304 ±.003

.393 ±.014

.369 ±.004

.366 ±.027

.387 ±.006

.400 ±.002

.529 ±.055

.379 ±.002

.322 ±.021

.321 ±.004

2

.295 ±.003

.296 ±.004

.300 ±.003

.317 ±.004

.460 ±.002

.296 ±.004

.344 ±.025

.350 ±.014

.330 ±.003

.331 ±.027

.356 ±.039

.508 ±.018

.357 ±.027

.295 ±.003

.316 ±.006

3

.283 ±.005

.284 ±.005

.288 ±.003

.292 ±.003

.444 ±.001

.286 ±.000

.311 ±.007

.300 ±.013

.292 ±.002

.311 ±.006

.307 ±.004

.387 ±.091

.299 ±.011

.283 ±.005

.309 ±.006

4

.277 ±.002

.276 ±.002

.276 ±.004

.289 ±.003

.431 ±.003

.277 ±.001

.296 ±.021

.282 ±.007

.289 ±.003

.306 ±.004

.274 ±.005

.338 ±.060

.285 ±.015

.277 ±.004

.286 ±.014

5

.269 ±.005

.271 ±.005

.270 ±.005

.269 ±.005

.387 ±.004

.271 ±.005

.270 ±.006

.269 ±.005

.269 ±.005

.273 ±.005

.269 ±.005

.306 ±.055

.283 ±.017

.269 ±.005

.283 ±.017

1

.750 ±.005

.790 ±.031

.783 ±.002

.783 ±.002

.832 ±.010

.793 ±.029

.843 ±.011

.750 ±.005

.783 ±.002

.777 ±.002

.771 ±.020

.795 ±.032

.777 ±.030

.750 ±.005

.777 ±.002

2

.744 ±.004

.742 ±.005

.746 ±.005

.765 ±.012

.771 ±.004

.742 ±.005

.840 ±.010

.747 ±.007

.772 ±.001

.764 ±.011

.758 ±.017

.822 ±.039

.759 ±.031

.742 ±.005

.746 ±.002

3

.735 ±.007

.736 ±.009

.744 ±.005

.743 ±.005

.738 ±.006

.736 ±.009

.781 ±.041

.742 ±.009

.749 ±.019

.757 ±.011

.744 ±.007

.774 ±.034

.750 ±.019

.732 ±.005

.742 ±.007

4

.734 ±.007

.734 ±.004

.742 ±.002

.736 ±.008

.738 ±.007

.734 ±.004

.754 ±.032

.738 ±.006

.736 ±.005

.748 ±.017

.740 ±.005

.762 ±.021

.745 ±.012

.731 ±.004

.737 ±.005

5

.730 ±.005

.730 ±.006

.734 ±.006

.732 ±.005

.735 ±.006

.730 ±.006

.735 ±.008

.735 ±.003

.733 ±.007

.737 ±.005

.731 ±.005

.746 ±.012

.735 ±.008

.731 ±.004

.735 ±.005

1

.0114 ±.0014

.0108 ±.0015

.0112 ±.0011

.0110 ±.0013

.0122 ±.0032

.0108 ±.0015

.0121 ±.0025

.0109 ±.0013

.0110 ±.0013

.0112 ±.0015

.0112 ±.0017

.0143 ±.0041

.0112 ±.0014

.0113 ±.0015

.0109 ±.0013

2

.0112 ±.0018

.0113 ±.0011

.0107 ±.0011

.0107 ±.0011

.0109 ±.0009

.0113 ±.0011

.0123 ±.0026

.0107 ±.0011

.0108 ±.0011

.0111 ±.0013

.0109 ±.0011

.0127 ±.0017

.0111 ±.0015

.0108 ±.0015

.0113 ±.0011

3

Dataset

Budget idx j 1

.618 ±.061

.0109 ±.0012

.0112 ±.0010

.0108 ±.0010

.0107 ±.0012

.0110 ±.0008

.0112 ±.0010

.0111 ±.0010

.0108 ±.0013

.0108 ±.0010

.0112 ±.0014

.0110 ±.0010

.0118 ±.0023

.0110 ±.0014

.0107 ±.0014

.0110 ±.0008

4

lung_cancer

.440 ±.217

.0110 ±.0015

.0113 ±.0011

.0110 ±.0011

.0108 ±.0009

.0110 ±.0007

.0113 ±.0011

.0110 ±.0011

.0110 ±.0011

.0109 ±.0014

.0111 ±.0013

.0109 ±.0009

.0123 ±.0007

.0106 ±.0010

.0110 ±.0016

.0107 ±.0008

5

soybean large

.816 ±.098

.0112 ±.0012

.0111 ±.0012

.0111 ±.0010

.0111 ±.0010

.0109 ±.0007

.0111 ±.0012

.0110 ±.0010

.0112 ±.0010

.0112 ±.0010

.0111 ±.0013

.0110 ±.0009

.0125 ±.0005

.0108 ±.0008

.0109 ±.0017