AutoMatBench: An Automatic Optimization Toolkit for the Acceleration of Material Properties Prediction Benchmarking Hongxiao Lia,b , Wanling Gaoa,b a Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China
arXiv:2607.11526v1 [cs.LG] 13 Jul 2026
b University of Chinese Academy of Sciences, Beijing, China
ARTICLE INFO
ABSTRACT
Keywords: Artificial intelligence for science Artificial intelligence for materials Material properties prediction Benchmark Evaluation Automatic optimization
Material property prediction (MPP) infers key properties from chemical composition and structure, accelerating the discovery and optimization of novel materials. In the realm of MPP, MatBench is a widely accepted benchmarking tool that defines over ten significant problems and provides the paradigm of performance evaluation for AI prediction models. Even though MatBench works well in benchmarking the performances of prediction models on in-distribution (ID) tasks and datasets, it lacks the ability to reflect their performances on out-of-distribution (OOD) material data, resulting failure in new material discovery. By combining the pipelines of MatBench and the existing researches on OOD performance evaluation, this study enables a huge space of benchmarking configurations, comprehensively reflecting the performances, abilities, and disadvantages of various AI prediction models. This work reports that the discrepancy of performances at different configuration values is huge and can be illustrated with prior knowledge and novel insights, therefore consideration of causal effect of configurations on performance results is necessary. In case of the impossibility of enumerative benchmarking at every configuration, this work further proposes AutoMatBench, an automatic toolkit with Bayesian optimization. Experiments with AutoMatBench reports that, within twelve steps of optimization, the similar results with MatBench and former OOD research can be accessed while more than half of the cost are saved. Besides, this tool also yields more essential findings on MPP benchmarking, positively contributing to the cost and efficiency of new material discovery.
1. Introduction Material property prediction (MPP), as a signficant subfield of Artificial Intelligence for Science (AI4S), aims at predicting or inferring the physical, chemical, or functional properties of materials directly from their composition and structure data. By learning relationships of compositional or structural information from experimental and computational data, MPP models enable high-efficiency screening of demanded materials, guide target-oriented synthesis, and reduces reliance on high-cost experiments or ab initio simulations. MatBench [1] emerges as a widely accepted benchmarking suite that defines over ten diverse representative prediction tasks for both classification and regression problems, curated from solid datasets. MatBench provides a consistent training-test splitting and cross-fold validation, establishing a generalized and comparable paradigm for evaluating the prediction accuracies of AI models across tasks and datasets. It has become the most adopted de facto reference for developing and validating new material informatics methods. Unfortunately, benchmarking results with MatBench only represent the performances on in-distribution (ID) tasks and datasets, while the performances on out-of-distribution (OOD) data are totally different. Research from Omee et al. [2] and Fung et al. [3] also support such conclusion. Experiments of this work also report similar outcomes. This study aims at establishing an evaluation methodology the abilities of AI prediction models that works both on ID and OOD data, referring the pipelines of MatBench [1] and the existing OOD performance evaluation. There are two main challenges. On the one hand, the existing OOD First Author et al.: Preprint submitted to Elsevier
performance evaluation [4, 5, 6, 2] are neither completely undergone under the same or similar paradigm of MatBench, nor including the majority of material prediction tasks that MatBench owns. On the other hand, the results of both MatBench and OOD research relies on the specific benchmarking configurations of experiments, which fluctuate heavily when changed. As a result, asserting which performance result is the actual ability of an AI model is non-trivial. To solve such problems, we basically adopt the workflow of Omee et al. [2] and the dataset source of MatBench [1], but with a larger space of hundreds of common configurations and a larger data scope. We select six tasks: band gap [7, 8, 9], bulk modulus [7, 8, 10], shear modulus [7, 8, 10], formation energy [7, 8, 11], refractive index [7, 8, 12], and metallicity [9], which can be aligned to MatBench [1] and is the superset of Omee et al. [2]’s research scope. By evaluation of five general-purpose prediction models that can be aligned to the officially announced results of MatBench [1]: ALIGNN [13], SchNet [14, 15], CrabNet [16], RF-SCM [1, 17, 18, 19], and MEGNet [20], we report the discrepancy of performances at different configuration values, where configuration refers to three factors: dataset ratio 𝑟, clustering number of OOD material data 𝑛, and the number of centroid and nearest neighbor 𝑠, which are the same semantics as in those of OOD research. Our results report that the performance for a specific model on a certain task has a huge discrepancy between different configurations. Merely deciding the result of any single configuration or a few configurations as the ability of the model may involve heavy biases and misunderstanding of the model.
Page 1 of 16
AutoMatBench: An Automatic Optimization Toolkit
This issue splits into two occasions. First, if the groundtruth performance exists, how to find the closest configuration at low cost; second, if not (or cannot be well-defined), which set of configurations still makes sense in benchmarking the models. For the first problem, our answer is developing an automatic optimization toolkit, AutoMatBench; for the second one, our aim is to search for the configurations that discriminate performances of different models. We propose AutoMatBench, an automatic toolkit with Bayesian optimization [21]. The experiment results report that, within twelve steps of optimization, AutoMatBench manages to find the configuration (i.e., aforementioned (𝑟, 𝑛, 𝑠) tuples) that meets the goal for all tasks and AI models, in accordance of the existing research outcome, but saves more than half of the cost. When the steps are set to more than twelve, improvement is limited. Moreover, AutoMatBench also delivers more findings for MPP benchmarking that are not reported in the existing studies, and can be well-illustrated with prior knowledge in MPP field and novel findings of this work. We hope the study of AutoMatBench guide the selection of high value-for-money algorithmic configurations, reduce experimental and computational costs, and support end-toend optimization from method selection and data governance to deployment, informing model design and feature engineering for not only MPP but also other AI4S issues.
2. Background and related work 2.1. Background This study focuses on the evaluation and automatization in materials informatics field. At the foundation of materials informatics lie ab initio calculations, most notably density functional theory (DFT) (e.g., DFT-GGA [22], DFT-vDWDF [23]), the gold standard for generating accurate, atomiclevel materials property data with well-defined physical consistency started from physical theorems. The most prominent material datasets are the Materials Project (MP) [7] (over 154,000 inorganic materials with DFT-calculated thermodynamic, electronic, and mechanical properties), Open Quantum Materials Database (OQMD) [24] (crystalline materials’ structural and energetic data), and the JARVIS infrastructure [25] (over 40,000 materials’ standardized DFT datasets and integrated machinelearning (ML) tools). These databases have democratized high-quality material data access and established a unified foundation for property prediction ML model training and validation. Our proposal of AutoMatBench is based on MatBench [1] – analogous to ImageNet [26] – offering thirteen standardized classification and regression tasks with DFT and experimental data, covering a variety of materials, serving as the de facto benchmarking suite for validating materials ML models like graph neural networks (GNNs). Although SOTA ML models perform well on ID benchmarks like MatBench [1], real-world materials discovery requires generalization to OOD samples. Traditional random First Author et al.: Preprint submitted to Elsevier
splitting strategies overestimate performance and ignore real extrapolation challenges. Omee et al. [2] first systematically benchmarked GNNs on OOD materials property prediction with five realistic tasks. They found a large generalization gap that most top MatBench models degraded severely under OOD settings, while only a few (e.g., ALIGNN [13]) remained relatively robust. This work revealed the weakness of conventional ID benchmarks and spurred research on OODrobust methods for materials informatics. The framework of AutoMatBench follows the majority of principles of Omee et al. [2]. Beyond domain-specific benchmarks, Evaluatology [27] has been established as a universal theoretical framework for rigorous, consistent evaluation across disciplines, addressing fundamental challenges in evaluation. It focuses on distinguishing the causal effect of a particular object from the total effect of an entire system. The problem definition of our work is in accordance with Evalautology [27]. Our study also involves automatic optimization of benchmarking techniques. Existing optimization methods include gradient-based optimizers (i.e., SGD [28], Adam [29]), nongradient algorithms, and model compression techniques. We only adopt Bayesian optimization [21], a sample-efficient black-box optimization approach that uses a surrogate model and acquisition function to balance exploration and exploitation.
2.2. Other related work Besides, there are other studies of OOD materials. Segal et al. [4] specifically addresses the issue of OOD attribute value prediction (Y-extrapolation) by introducing transduction learning methods. Tan et al. proposes MatUQ [5], aiming to jointly evaluate the prediction accuracy and Uncertainty Quantification (UQ) quality of GNNS in OOD scenarios, emphasizing the need for prediction reliability in actual material discovery. Li et al. proposes the Crystal Adversarial Learning (CAL) algorithm [6], specifically for the small sample problem. Before our research, there is the study of SAIBench [30, 31], which is based on Evaluatology [27] and the physical symmetricity of AI4S tasks, an AI4S benchmark suite that includes the tasks of machine learning for force field (MLFF), jet tagging, and precipitation nowcasting, provides a unified benchmarking system for scientific AI, decoupling scientific research problems, AI models, evaluation metrics, and computational configurations into reusable modules to standardize benchmarking across diverse scientific domains.
3. Motivation Figure 1 presents the overall motivation of this work.
Page 2 of 16
AutoMatBench: An Automatic Optimization Toolkit
3.1. MatBench has limited ability on benchmarking the OOD performance of ML models MatBench [1] evaluative capacity is predominantly restricted to ID scenarios. According to its dataset split strategy, the test dataset adheres to the same statistical distribution as the training dataset. In practical materials science research and applications, ML models frequently encounter OOD challenges, that is, predicting the properties of materials whose compositions, crystal structures, or properties (Y-values) are distant from the distribution center of the training dataset. Omee et al. [2] reports that all GNN models for material property prediction exhibit a substantial performance gap between OOD and ID tasks: their OOD MAEs are significantly higher than ID baselines with degradation ranging from −0.83% to −5568.31%. Only CGCNN [32], ALIGNN [13] and DeeperGATGNN [33] achieve slight ID performance surpassing on a few OOD targets. This inherent limitation tends to result in severe overestimation of model performance in real-world applications, thereby impeding the efficiency of AI models in materials discovery and rational design.
ML. Automatic optimization offers a promising solution to this pressing challenge. By leveraging an optimization algorithm, it is feasible to automatically search for optimal OOD evaluation configurations – including material dataset ratio, data splitting strategy, and OOD data scale decided by the number of nearest neighbor by clustering – that strike a balance between evaluation cost and result reliability. This study focuses on three concrete problems. First, whether the evaluated performances of different configurations vary, which influences the reliability of a single evaluation. Second, if there is a ground-truth performance of a certain AI model, how to find the evaluation configuration that yields the closest result at low lost. Third, without defining the ground-truth performance, what configurations discriminates most AI models’ performances. We managed to formalize, solve, and implement all of the above with the Bayesian optimization algorithm [21] as AutoMatBench. The experiment results of this research reports that within twelve steps of optimization, the adequate configurations of each problem are found (even though not the real optimum). This study further finds better configurations with less than half (< 50%) of the cost compared to Omee et al.’s method [2].
3.2. The existing OOD benchmarking methods has a huge cost and poor generalizability To address the aforementioned gap in OOD evaluation, several novel models and OOD benchmarking approaches have been proposed in recent years [13, 33, 20, 34]. However, these methods are plagued by two critical limitations: high implementation cost and poor generalizability. On the one hand, due to the high cost of OOD material discovery from the laboratories, only limited scale if OOD data are obtained. For example, the MP [7] and JARVIS [25] dataset include the majority of known material data. However, the OOD data ratio is around 15% or 20%. Therefore, conducting OOD evaluations across multiple model architectures and task types demands considerable computational resources, as it involves retraining models on diverse data splits and performing repeated evaluative experiments. On the other hand, the majority of existing OOD benchmarking methods are task-specific – they are designed for a particular materials property (e.g., band gap, formation energy, etc.), which renders them difficult to extend to other materialsrelated tasks or OOD settings. This lack of generalizability significantly limits their utility as a universal evaluative tool for assessing OOD performance across the broader material ML community. According to our estimation, the cost of a complete evaluation on six tasks is up to orders of magnitude than benchmarking with MatBench [1].
3.3. Automatic optimization finds cost-effective evaluation configurations with near-reliable results Given the inherent limitations of MatBench [1] and the critical shortcomings of existing OOD benchmarking methods, there exists an urgent demand for a cost-effective and generalizable approach to OOD evaluation in materials First Author et al.: Preprint submitted to Elsevier
Page 3 of 16
AutoMatBench: An Automatic Optimization Toolkit
Figure 1: The motivation: this study provide a novel, costeffective, and near-reliable benchmarking method for material that solves both problems of failure in OOD evaluation and the lack of generalizability.
First Author et al.: Preprint submitted to Elsevier
Page 4 of 16
AutoMatBench: An Automatic Optimization Toolkit
A dataset with the better discrimination ability on models is more acceptable.
4. Problem definition 4.1. Evaluation configuration space This study focuses on the performances of material prediction models on certain material datasets. All the prediction tasks in this research are classification problems. The index of models’ performance is chosen as the mean average error (MAE). Section 6 in late text provides detailed description of OOD benchmarking pipeline. Briefly, sampling from two datasets, material clustering into groups, and retrieving nearest neighbors are three critical steps that includes parameters. For formalization, we note the total set of AI models as 𝑂 and a certain model as 𝑜𝑖 ∈ 𝑂 (𝑖 for a certain model). The ratio 𝑟 denotes the ratio of material data sampled from JARVIS dataset [25], while the rest are sampled from MP dataset [7]. The letter 𝑛 denotes the number of folds in clustering data with K-Means algorithm [35]. The letter 𝑠 denotes the number of nearest neighbors of every cluster centroid plus the centroid itself. The details are provided in Section 6. We use capital letters 𝑅, 𝑁, and 𝑆 to represent the value range of 𝑟, 𝑛, and 𝑠, which includes multiple choices, provided in Section 6. We note the Cartesian product of them as the evaluation configuration space 𝐶 = 𝑅 × 𝑁 × 𝑆, which includes every probable setting in our work. This definition enables to evaluate an AI model with any dataset generated with a given configuration 𝑐𝑗 = (𝑟, 𝑛, 𝑠) (𝑗 for a certain choice), which is the basis of the following analysis. The performance of a model 𝑜𝑖 on a certain dataset 𝑐𝑗 is noted as 𝑚(𝑜𝑖 , 𝑐𝑗 ). This work studies the following problems: firstly, quantify the disparity between models’ performances with different configuration; secondly, find the best configuration at low cost when ground-truth exists; lastly, choose a rational configuration when ground-truth is not defined. They are formalized as follows respectively.
4.2. The first-class evaluation problem This problem is the most common consideration of material science researchers. It aims at finding the model of best average performance on a certain material dataset. Formally, we have: arg max mean 𝑚(𝑜𝑖 , 𝑐𝑗 ) 𝑜𝑖 ∈𝑂 𝑐𝑗 ∈𝐶
(1)
Especially, if top-K performance rather than the average is considered, the mean operator can be changed to top𝐾 correspondingly. This work studies the stability of this value when the evaluation configuration is changed.
4.3. The dual of first-class evaluation problem This problem is the mathematical dual of the above problem, if the mean operator is changed to range. It aims at solving such problem: among all the configurations, which one has the maximum quantity discrimination ability among AI models, where “quantity” refers to its performance (MAE). arg max range 𝑚(𝑜𝑖 , 𝑐𝑗 ) 𝑐𝑗 ∈𝐶 𝑜 ∈𝑂 𝑖
First Author et al.: Preprint submitted to Elsevier
(2)
4.4. The second-class evaluation problem This problem is another essential problem that focuses on the validity and cost of benchmarking. In this context, we suppose the ground-truth quantity of an AI model is formally definite. It can refer to a certain reliable benchmarking research or a conventionally consensus of the AI4S community. Otherwise it can be defined as the aggregation (average, top-K, or other representations) of AI performances in the entire configuration space 𝐶 as follows, according to the concrete research requirement: def
gt(𝑜𝑖 , 𝐶) = agg (𝑜, 𝑐𝑗 )
(3)
𝑐𝑗 ∈𝐶
Based on the definition of gt, the problem is defined as finding the best configuration set that yields results close to the gt value at low cost: arg min mean dist(𝑚(𝑜𝑖 , 𝑐𝑗 ), gt(𝑜𝑖 , 𝐶)) 𝑜𝑖 ∈𝑂 𝑐𝑗 ∈𝐶
(4)
A result that has a huge disparity with the ground-truth result is considered as invalid or flawed.
5. Methodology 5.1. The existing methods Based on the problem definitions, the MatBench [1]’s benchmarking problem can be reformed as follows: arg max 𝑚(𝑜𝑖 , 𝑐0 ) 𝑜𝑖 ∈𝑂
(5)
where 𝑐0 is a certain dataset for a given task, and different 𝑜𝑖 ’s are AI models to be evaluated. For example, suppose the task is the band gap prediction problem [7, 8, 9], then the dataset 𝑐0 is derived from Zhuo et al. [9]. According to the rules defined by MatBench [1], the dataset is separated into several folds that cover training set and test set, following the Nested CV scheme [36]. An 𝑜𝑖 may refer to a certain model (e.g., ALIGNN [13]). For OOD benchmarking, the method of Omee et al. [2] is similar but for several differences. First, the dataset includes only screened OOD materials, rather than the entire primitive dataset. Second, the training-test set splitting is in accordance with the clustering of materials, that is, every fold belongs to a certain cluster, representing materials that have similar characteristics or properties. The formalized problem remains unchanged.
5.2. Our novelty As defined in Section 4, our research focuses on three major problems different from the existing. According to Section 3, the performances of models vary to orders of magnitude from ID problems to OOD problems. In addition, using only a mere fixed OOD dataset, the results with different configurations (𝑟, 𝑛, 𝑠) fluctuates heavily. Therefore, this Page 5 of 16
AutoMatBench: An Automatic Optimization Toolkit
study does not only consider the performance results on any fixed dataset with its separation. We first define the total dataset that contains the majority of important ID and OOD material data. The ideal goal is to evaluate every model’s performance on the entire dataset. Unfortunately, it is impossible for cost concerns. As a substitute, we evaluate on different samples with the dataset to approximate the ground-truth performances of models. Two main challenges exist with this approach. First, how the ground-truth performance is defined. This issue is explained in the second-class problem in late text. Second, what algorithm approximates the ground-truth performance at low cost. For all the three problems, the Bayesian optimizer [21] works well. Figure 2 is the overall view of our methodology.
5.3. The algorithm for the first-class problem To address the first-class evaluation optimization problem under a limited evaluation budget, we adopt Bayesian optimizer [21] to efficiently identify the optimal AI model that maximizes average performance (MAE) across all evaluation configurations. The core objective is to obtain the optimal model 𝑜̂ by maximizing the mean evaluation metric over the full configuration set, defined with formula 1 where 𝑜 denotes a candidate AI model, 𝑐 denotes an evaluation configuration, and 𝑚(𝑜, 𝑐) measures the performance of model 𝑜 under configuration 𝑐. Exhaustive evaluation of all modelconfiguration pairs is computationally infeasible, making Bayesian optimization an ideal sample-efficient solution. Algorithm 1 operates iteratively within a fixed evaluation budget 𝑇 , split into two sequential phases: initial random exploration and adaptive optimization. It first initializes the dataset with a small set of random (𝑜, 𝑐) samples to exhaust an initial portion of the budget, prioritizing broad exploration of unobserved (𝑜, 𝑐) pairs to map the latent performance landscape. Once the random exploration phase is complete, this procedure iteratively updates a probabilistic surrogate model using the collected data and leverages an acquisition function to select the most informative (𝑜, 𝑐) pair for evaluation – this optimization phase balances targeted exploration of under-sampled regions and exploitation of highperformance regions to maximize evaluation efficiency. After fully exhausting the total evaluation budget, we estimate ̂ the average performance 𝑀(𝑜) for each AI model across all evaluated configurations, and select the model with the highest estimated average performance as the optimal solution. This algorithm avoids redundant evaluations, adapts dynamically to the underlying performance landscape, and reliably locates the optimal AI model under strict resource constraints.
5.4. The algorithm for the dual of first-class problem Similar to the first-class problem, its dual problem aims to obtain the optimal configuration 𝑐̂ by maximizing the range evaluation metric over the full configuration set, defined with formula 2. After fully exhausting the total evaluation budget, we estimate the range of models’ performances First Author et al.: Preprint submitted to Elsevier
Algorithm 1 Bayesian optimization for the first-class evaluation problem Require: Option set 𝑂 = {𝑜1 , 𝑜2 , ⋯}, Context set 𝐶 = {𝑐1 , 𝑐2 , ⋯}, Evaluation function 𝑚(𝑜, 𝑐), Evaluation budget 𝑇 , Initial sample size 𝑇0 . ̂ Ensure: Optimal option 𝑜̂ ∈ 𝑂 that maximizes 𝑀(𝑜) = 1 ∑ 𝑚(𝑜, 𝑐 ) 𝑗 𝑐𝑗 ∈𝐶 |𝐶| 1: Initialize dataset 𝐷 ← ∅ 2: Random sample and evaluate 𝑇0 (𝑜, 𝑐) pairs into 𝐷 3: for 𝑡 = 1 to 𝑇 − 𝑇0 do 4: Update surrogate model with 𝐷 5: Select (𝑜∗ , 𝑐 ∗ ) by acquisition function 6: Evaluate and update dataset 𝐷 7: end for ̂ for all 𝑜 ∈ 𝑂 8: Estimate 𝑀(𝑜) ̂ 9: Return 𝑜̂ = arg max𝑜 𝑀(𝑜) Algorithm 2 Bayesian optimization for the dual of first-class evaluation problem Require: Option set 𝑂 = {𝑜1 , 𝑜2 , ⋯}, Context set 𝐶 = {𝑐1 , 𝑐2 , ⋯}, Evaluation function 𝑚(𝑜, 𝑐), Evaluation budget 𝑇 , Initial sample size 𝑇0 . ̂ Ensure: Optimal context 𝑐̂ ∈ 𝐶 that maximizes 𝑅(𝑐) = min |𝑚(𝑜𝑖1 , 𝑐) − 𝑚(𝑜𝑖2 , 𝑐)| 𝑜𝑖 ,𝑜𝑖 ∈𝑂 1
2
1: Initialize dataset 𝐷 ← ∅ 2: Random sample and evaluate 𝑇0 (𝑜, 𝑐) pairs into 𝐷 3: for 𝑡 = 1 to 𝑇 − 𝑇0 do 4: Update surrogate model with 𝐷 5: Select (𝑜∗ , 𝑐 ∗ ) by acquisition function 6: Evaluate and update dataset 𝐷 7: end for
̂ for all 𝑐 ∈ 𝐶 8: Estimate 𝑅(𝑐) ̂ 9: Return 𝑐̂ = arg max𝑐 𝑅(𝑐) ̂ 𝑅(𝑐) under the configuration 𝑐 instead for each AI model across all evaluated configurations, and select the configuration with the maximum estimated range as the optimal solution. The corresponding algorithm is Algorithm 2.
5.5. The algorithm for the second-class problem For the second-class problem, the target is to obtain the optimal configuration 𝑐̂ by minimizing the disparity of performances between its result and the ground-truth over the full configuration set, defined with formula 4. The groundtruth function gt(𝑜, 𝐶) denotes the ground-truth performance value of option 𝑜 measured on the complete context set 𝐶, according to Formula 3. We estimate and select the configuration with the minimum distance to the groundtruth as the optimal solution. The corresponding algorithm is Algorithm 3. Page 6 of 16
AutoMatBench: An Automatic Optimization Toolkit
Algorithm 3 Bayesian optimization for the second-class evaluation problem Require: Option set 𝑂 = {𝑜1 , 𝑜2 , ⋯}, Context set 𝐶 = {𝑐1 , 𝑐2 , ⋯}, Evaluation function 𝑚(𝑜, 𝑐), Ground-truth function gt(𝑜, 𝐶), Distance function dist(⋅, ⋅), Ground-truth gt(𝑜𝑖 , 𝐶) = agg (𝑜, 𝑐𝑗 ), 𝑐𝑗 ∈𝐶
Evaluation budget 𝑇 , Initial sample size 𝑇0 . ̂ Ensure: Optimal option 𝑜̂ ∈ 𝑂 that minimizes 𝐿(𝑜) = ( ) 1 ∑ 𝑐𝑗 ∈𝐶 dist 𝑚(𝑜, 𝑐𝑗 ), gt(𝑜, 𝐶) |𝐶| 1: Initialize dataset 𝐷 ← ∅ 2: Random sample and evaluate 𝑇0 (𝑜, 𝑐) pairs into 𝐷 3: for 𝑡 = 1 to 𝑇 − 𝑇0 do 4: Update surrogate model with 𝐷 5: Select (𝑜∗ , 𝑐 ∗ ) by acquisition function 6: Evaluate and update dataset 𝐷 7: end for ̂ for all 𝑜 ∈ 𝑂 8: Estimate 𝐿(𝑜) ̂ 9: Return 𝑜̂ = arg min𝑜 𝐿(𝑜)
Figure 2: The methodology of this research. The other existing benchmarking methods do not include configuration adjustment and optimization parts. The optimization part is the essence of our method.
First Author et al.: Preprint submitted to Elsevier
Page 7 of 16
AutoMatBench: An Automatic Optimization Toolkit
6. Framework 6.1. The evaluation pipeline Step 1 (Sampling): Sample 𝑛𝑡𝑜𝑡𝑎𝑙 = 1, 600 materials from the MP [7] and JARVIS [25] databases according to a prescribed proportional ratio 𝑟. For every task, all material data from MP [7] and JARVIS [25] that include the corresponding property values are downloaded. After that, 𝑛𝑡𝑜𝑡𝑎𝑙 data are randomly sampled from the database. The savefile of every material includes the original CIF file (generated using pymatgen or JARVIS-Tools), the 𝑛𝑑𝑖𝑚 = 1, 024 dimensional OFM characteristic, and the property (Y-value). Step 2 (Dimensionality reduction): Embed the highdimensional structural descriptors into a 2D space using tdistributed stochastic neighbor embedding (t-SNE, using sklearn.manifold.TSNE) [37]. This step preserves local neighborhood structure to reveal clusters and similarities in the high-dimensional data, making latent structural patterns more readily identifiable in the 2D visualization. Step 3 (Density estimation): Estimate local sample densities on the 2D embedding via Gaussian kernel density estimation (KDE, using scipy.stats.gaussian_kde) [38]. This step approximates the continuous probability density of points on the 2D manifold, highlighting high-density regions (clusters) and low-density gaps by smoothing with a Gaussian kernel. Step 4 (Sparse screening): Select the 𝑛𝑠𝑎𝑚𝑝𝑙𝑒 = 480 samples with the lowest KDE values and designate them as out-of-distribution (OOD) candidates. The selected candidates that have the lowest density represent the materials of 2D OOD characteristics. Step 5 (Sparse clustering): Cluster the 𝑛𝑠𝑎𝑚𝑝𝑙𝑒 OOD candidates into 𝑛 groups using K-means (𝑛-fold) [35]. This step partitions the 𝑛𝑠𝑎𝑚𝑝𝑙𝑒 OOD candidates into 𝑛 compact, non-overlapping clusters by minimizing within-cluster variance, yielding representative groups for next-step analysis. Step 6 (Target generation and split): For each cluster, choose its centroid as an anchor and, in the original highdimensional space, retrieve the 𝑠 nearest neighbors of the corresponding sample; the union of these neighbors forms the test set, and the remaining samples constitute the training set. For clarity, if 𝑠 = 1, then no nearest neighbors are taken. This measure corresponds to the SparseX-Single strategy in Omee et al.’s benchmark [2]. If 𝑠 > 1, then there will be 𝑠 − 1 nearest neighbors, corresponding to the SparseXCluster strategy instead. The final 𝑛-fold data for crossvalidation is as follows: each fold corresponds to a certain cluster, representing an unknown or known type of materials. In each fold, a proportion 𝑝 = 0.1 of data are considered as the validation set. For the rest, the test set includes only 𝑠 data. The training set consists of all samples except this cluster.
6.2. The optimization settings Figure 3 presents the entire procedure. The cycle on the bottom describes a one-step evaluation under the configuration (𝑟, 𝑛, 𝑠). As for optimization algorithms (1,2, and 3), the evaluation budget is set as 𝑇 = 12, and the exploration First Author et al.: Preprint submitted to Elsevier
budget of the initial dataset 𝐷 is set as 𝑇0 = 4. To ensure a fair comparison across methods, all algorithms are initialized with the same random seed (42) and the identical initial dataset 𝐷. Additionally, all reported results are totally replicable and traceable, as the random factors are under control. All the configurations included in the full configuration space are as follows: 𝑟 ∈ 𝑅 = {0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1}, 𝑛 ∈ 𝑁 = {4, 6, 8, 10, 12, 24, 48, 60}, 𝑠 ∈ 𝑆 = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}, especially 𝑠 ≤ 8 when 𝑛 = 60. Altogether 858 configuration values in total.
6.3. Material datasets This subsection introduces the two datasets of our work. The materials are sampled from the two datasets with a given ratio.
6.3.1. Materials Project The Materials Project (MP) [7] is a large-scale, openaccess repository of computed inorganic materials properties generated primarily via high-throughput ab initio density functional theory workflows. Crystal structures are sourced from experimental databases and hypothetical prototypes, then standardized through space-group symmetrization and conventional cell selection before property evaluation following well-documented protocols adopting generalized gradient approximation. The Materials Project provides a semantically rich schema that links each material identifier to full provenance covering input structures, calculation parameters and code versions, computed thermodynamics including formation energies and phase stability from convex hull analyses, electronic structure information for band structures and density of states, mechanical and transport descriptors, and defect and surface models for selected systems. Data are accessible through a versioned REST API and Python client tools, with ontology-aligned fields capturing synthesis relevant information including energy above hull and decomposition pathways, while computed entries support reproducible and queryable workflows across consistent calculation sets. 6.3.2. Joint Automated Repository for Various Integrated Simulations Joint Automated Repository for Various Integrated Simulations [25] is a comprehensive material data ecosystem combining first principles calculations classical force field simulations and machine learning surrogates to deliver standardized property benchmarks across two dimensional bulk and molecular systems. ab initio data largely derive from DFT calculations for layered materials and PBE methods [39] paired with spin orbit coupling in applicable cases while further supplemented by many body perturbation related workflows tight binding calculations and phonon analyses to support optoelectronic mechanical and vibrational property research. Semantically JARVIS organizes all research content as defined tasks with clear metadata Page 8 of 16
AutoMatBench: An Automatic Optimization Toolkit
recording relevant computational details exchange correlation functional selections k point mesh arrangements pseudopotential configurations convergence standards and post processing procedures. It supplies abundant characteristic parameters including exfoliation energies dielectric tensors elastic constants effective masses and topological invariants through stable unique identifiers and dedicated application programming interfaces. The framework prioritizes standardized benchmark evaluations and guaranteed result reproducibility facilitating direct model comparisons and clear traceability across different computational approaches.
6.4. Property prediction tasks This subsection introduces the six prediction tasks used in AutoMatBench.
6.4.1. Band gap Band gap [7, 8, 9] task is defined as a regression problem to predict the electronic band gap in eV of inorganic crystalline materials. The dataset’s inputs are crystal structures referenced to Materials Project entries along with structure derived descriptors while the target label is a continuous band gap value computed through standard DFT workflows adopted by the Materials Project following conventional GGA or PBE approaches that tend to produce values lower than experimental measurements. Labels reflect the fundamental gap obtained from calculated electronic band structure information. Metallic systems are assigned values close to zero and numerical thresholds support metallic classification in preprocessing workflows. Detailed metadata records original material identifiers and calculation configurations from the Materials Project ensuring full reproducibility and objective evaluation for models trained directly on structural information or predefined features. 6.4.2. Bulk modulus Bulk modulus [7, 8, 10] task aims to predict the bulk modulus in GPa for inorganic crystals reflecting inherent resistance toward volumetric compression. Input content mainly adopts crystal structure information and can integrate feature descriptions generated from component composition and structural characteristics. The corresponding label maintains continuous numerical form obtained from energy volume equation of state results calculated through standard elasticity workflows based on density functional theory. All data entries follow unified unit standards and consistent computational settings while relevant uncertainty information exists in implicit form without direct display. Fixed grouping modes support stable benchmark verification during model evaluation and every sample connects with exclusive material identification content as well as original elastic tensor data used for calculating bulk modulus values. 6.4.3. Formation energy Formation energy [7, 8, 11] task targets the formation energy per atom in eV per atom a thermodynamic descriptor indicating stability relative to elemental references. Inputs are the crystalline structures and associated compositions First Author et al.: Preprint submitted to Elsevier
mapped to material ids. Labels are computed formation energies from high throughput DFT pipelines using consistent reference states and applicable corrections throughout calculation processes. These values can combine with compositional energies to derive energy above hull while the label defined for this task remains strictly the per atom formation energy. Negative values represent thermodynamic stabilization relative to pure elements and near zero or positive values reflect metastability or instability of corresponding materials.
6.4.4. Metallicity Metallicity [9] is a binary classification task that predicts whether a material holds metallic or nonmetallic properties including semiconductor and insulator features based on electronic structure information. In this research, it is treated similar to a regression problem. Inputs adopt crystal structures linked to the Materials Project and relevant derived descriptors. The label adopts categorical binary values determined from calculated band gap data. Materials are recognized as metallic when the DFT calculated gap stays within a small threshold range while other substances fall into the nonmetallic category. The dataset may contain imbalanced sample distributions and official preset data splits maintain stable evaluation standards throughout testing processes. The judgment standard reflects inherent limitations existing in conventional GGA and PBE band gap calculation methods. Materials with narrow gap values near the classification boundary show sensitivity to numerical details during computation which carries importance for modeling research focusing on uncertainty analysis. 6.4.5. Refractive index Refractive index [7, 8, 12] task predicts the static refractive index of inorganic materials with no assigned measurement units. For simplification, no directional refraction index is considered. Inputs follow standard structure based representations connected to Materials Project references. The target label corresponds to refractive index values acquired from first principles linear response or density functional perturbation theory workflows relying on dielectric tensor data collected at zero frequency. Results are condensed into a single representative scalar determined from orientational averages of electronic contributions. The property carries no standard measurements and inherent anisotropy gets simplified into one unified value to suit benchmarking needs while explicit uncertainty information is not commonly provided within recorded data. 6.4.6. Shear modulus Shear modulus [7, 8, 10] task focuses on predicting the shear modulus in standard GPa units which characterizes material resistance toward shape change while maintaining constant volume. Inputs adopt crystal structure data referenced to the Materials Project and supporting features cover symmetry density and statistical information derived from chemical composition. The label presents continuous shear modulus values calculated from DFT elastic tensors Page 9 of 16
AutoMatBench: An Automatic Optimization Toolkit
following Voigt averaging standard [40] or else. The dataset clearly defines the adopted averaging approach and maintains unified unit settings across all entries.
6.5. Evaluated ML models This subsection introduces five ML models evaluated with AutoMatBench in our experiments.
6.5.1. ALIGNN Atomistic Line Graph Neural Network (ALIGNN) [13] is a graph neural network tailored for crystalline materials that emphasizes explicit encoding of many body interactions through a dual graph message passing scheme built with an atomistic graph and an auxiliary line graph for representing three body bond angle relationships. Atoms serve as nodes with learnable embeddings initialized from elemental attributes. Edges record interatomic distances and bond angle contexts processed with radial basis function expansions. Its core architecture applies alternating edge gated graph convolution operations on the two graphs to capture key coordination patterns closely connected to material properties. A global pooling module aggregates node feature states to support scalar property prediction tasks. The model delivers strong performance for formation energy, band gap, elastic modulus, and stability classification applications where structure related local atomic environments dominate material behaviors and maintains competitive results on mainstream regression benchmarks and standard evaluation leaderboards for material research datasets. 6.5.2. SchNet Schütt’s Network (SchNet) [14, 15], a variant of Deep Tensor Neural Networks (DTNN) [41], is a continuous filter convolutional neural network designed for molecular and crystalline systems that operates directly on nuclear charge information and three dimensional atomic positions. Interatomic interactions are modeled through continuous filter convolution modules defined with radial basis expansions of pairwise atomic distances within a preset cutoff range to build smooth and differentiable potential descriptions. The core structure alternates continuous filter interaction blocks and independent atom feature updates while sum pooling operations maintain consistent representation for extensive material properties. SchNet achieves reliable performance in quantum chemical property prediction covering energy force and dipole related outputs and delivers strong generalization for crystalline property regression tasks.
The model operates without explicit structural input features and learns underlying connections between chemical composition and material properties while generating corresponding uncertainty estimation supported by residual connection design and dropout regularization strategies. It performs effectively on prediction tasks dominated by compositional influencing factors covering approximate formation energy evaluation, elastic constant, band gap, and preliminary screening for hardness and critical temperature values when only chemical formula content can be accessed. In this research, CrabNet is also used in structure-known tasks for consistent comparison.
6.5.4. RF-SCM RF-SCM [1] integrates the Random Forest regressor [17] and the Sine Coulomb Matrix [18] while adopting feature sets derived from Magpie descriptors [19]. The applied features cover stoichiometric statistical values average and variance information of elemental properties alongside basic heuristic rules that reflect periodic trends across chemical elements. The model builds on ensemble decision tree structures supported by bootstrap aggregation strategies achieving stable performance with limited hyperparameter adjustment and clear interpretable feedback on feature contribution levels. RF-SCM shows reliable adaption for datasets with limited sample size or obvious noise and performs well on tasks dominated by compositional influences. It supports preliminary evaluation for formation energy analysis, band gap, metallicity, and mechanical tendency recognition. Similar to CrabNet [16], it also works when crystal structure data cannot be obtained. 6.5.5. MEGNet Materials Graph Network [20] is a unified graph neural network that combines atom bond and global state updates through edge node state message passing mechanisms. Nodes represent atoms with elemental embeddings while edges capture pairwise interactions defined by distance and radial basis features. A global state vector records core conditions such as temperature pressure and domain related labels. Each block conducts sequential feature updates with skip connection structures followed by readout pooling modules for final property prediction. Trained on large scale DFT datasets the model reaches high accuracy in formation energy prediction and can adapt to band gap, elastic modulus, dielectric constant, and other scalar or tensor property tasks.
6.5.3. CrabNet Compositionally restricted attention based network (CrabNet) [16] is a composition only neural architecture built for material systems without available crystal structure data. It applies self attention mechanisms across element tokens through learned representation built from compositional information and adopts multi head Transformer modules to capture interaction patterns among constituent elements.
First Author et al.: Preprint submitted to Elsevier
Page 10 of 16
AutoMatBench: An Automatic Optimization Toolkit
Figure 3: This figure presents the framework of AutoMatBench that consists if several modules. The pipeline is represented with procedures with arrows.
First Author et al.: Preprint submitted to Elsevier
Page 11 of 16
AutoMatBench: An Automatic Optimization Toolkit
7. Experiment In this section, we present four comprehensive evaluations of AutoMatBench. These experiments are designed to validate the effectiveness of this framework in identifying optimal models and configurations while reducing computational overhead with more than a half. The outcomes demonstrate that AutoMatBench can recover basic conclusions from exhaustive MatBench evaluations and OOD studies within just 12 optimization steps, meanwhile achieving a high cost reduction. Based on our results, the majority of suggested configuration settings is 𝑛 ≤ 24 and 𝑠 = 7 or 8 (except band gap [7, 8, 10] task, the best configuration is 𝑠 = 1). The ratio 𝑟 is not significant.
7.1. Experimental Setup All the experiments are conducted on a server equipped with an Intel Xeon CPU E5-2620 v3 of x86 architecture, with 63GB RAM and Ubuntu 20.04.6 LTS. The software environment was built using Anaconda with Python 3.7∼3.12, with deep learning models implemented in PyTorch and TensorFlow. Data sources include the MP dataset [7] with mp_api.client.MPRester and the JARVIS dataset [25] with jarvis.db.figshare.data.
7.2. Experiment set 1: Searching for the ML model that has the best prediction performance This experiment aims to identify the top-performing AI model within the AutoMatBench search space. This problem is the basic benchmarking aim for AI model researchers. It has a variant form, that is, to search for the configuration (𝑟, 𝑛, 𝑠) in the entire space where a certain model achieves the best performance. For developers of AI models, this problem is significant for submitting outstanding performances on a leaderboard that has less restrictions. Our experiment is done on the variant form. We evaluated five models on the band gap task [7, 8, 9] and the RF-SCM model [1] on all six tasks. The first observation is that, in the configuration of hundreds of configurations, the performance of an AI model varies considerably. The maximum discrepancy is 10.045 times (max 8.899, min 0.886), for the RF-SCM model [1] on the refractive index prediction task [7, 8, 12]. Figure 4 shows the wide range of multiple models’ MAEs on multiple tasks. The color and the scale of spheres correspond to the MAE, and the position in the three-dimensional space corresponds to the configuration (𝑟, 𝑛, 𝑠). Note that we excluded some outliers in the bulk modulus task [7, 8, 10]. The reason is that some materials have extreme large bulk modulus (> 10, 000), and this may cause divergence in the training process. The second observation is, for all tasks and models, the Bayesian optimizer finds an acceptable configuration within mostly 12 steps. In most conditions, the optimal 𝑠 value is less than or equal to 7. This indicates no more than 7 test samples are required. The 𝑛 values varies, but for most time, taking less than 24 is feasible. This eliminates First Author et al.: Preprint submitted to Elsevier
at least half of benchmarking cost compared to Omee et al.’s study [2] without reliability loss. Figure 8 and 9 show the process and results of optimization in this experiment. Red points stand for newly evaluated MAE. Blue points stand for the minimum. The yellow circle indicates the optimal configuration and its outcome.
7.3. Experiment set 2: Searching for the benchmark configuration that has the maximum degree of discrimination This experiment aims to identify the configuration (𝑟, 𝑛, 𝑠) in the entire space that discriminates the performances of models. An adequate configuration may reveal the fundamental advantages and disadvantages of certain models. This problem is significant for benchmark leaderboard designers and researchers. We evaluated RF-SCM [1] and CrabNet [16] models on all six tasks. Our observation is, in the six tasks, more than a half of tasks’ optimal discrimination configuration takes low 𝑠 quantity (𝑠 ≤ 2). The 𝑠 value stands for the test samples. As a result, taking a few samples affects the stability of performance results, for models may occasionally get mistaken on small datasets. Therefore, a bigger 𝑠 is necessary. Besides 𝑠 value, the 𝑟 ratio has little influence on results. As for the number of folds 𝑛, the range from 4 to 8 is enough. Figure 10 shows the process and results of optimization in this experiment. This eliminates about 75% benchmarking cost compared to Omee et al.’s study [2] without reliability loss. Red points stand for newly evaluated MAE. Blue points stand for the minimum. The yellow circle indicates the optimal configuration and its outcome. However, for stability of evaluation, the proper 𝑠 value should be set as 7 or 8 for tasks (a), (d) and (g). The experiment result for 𝑠 = 7 or 𝑠 = 8 also indicates an enough sufficiency with very little discrimination ability loss. As for task (f), the evaluation fails. The possible explanation is that the prediction for shear modulus [7, 8, 10] fails to converge, as models encounter extreme large values.
7.4. Experiment set 3: Searching for the benchmark configuration closest to the ground-truth of model prediction performance This experiment aims to identify the configuration (𝑟, 𝑛, 𝑠) in the entire space that yields the performances closest to the ground-truth. The ground-truth MAEs of models are defined as the suggested configuration (𝑟, 𝑛, 𝑠) = (0, 48, 10) by the existing study [2]. This assumption ensure that main conclusions with AutoMatBench is capable of reproducing existing findings. A configuration that yields results seriously inconsistent with the ground-truth is considered as unreliable. We evaluated RF-SCM [1] and CrabNet [16] models on all six tasks. Figure 11 and 12 show the process and results of optimization in this experiment. For the ratio parameter 𝑟, no Page 12 of 16
AutoMatBench: An Automatic Optimization Toolkit
Figure 4: This figure reports the wide discrepancies among evaluated MAEs while taking different configurations across different models and tasks. Sphere color/scale: MAE. Position: config.
Figure 5: This figure shows two examples [(a) and (b)] of optimization in Experiment 1. The optimal configurations are found within 12 steps. The full results are presented in the appendix.
significant influence is found. For the number of folds 𝑛, every experiment outcome reports that 𝑠 ≤ 24 is enough. This eliminates at least 50% benchmarking cost compared to Omee et al.’s study [2] without reliability loss. As for parameter 𝑠, most of tasks takes 𝑠 = 7 or 𝑠 = 8. The only exceptions are (a), (g), and (i) with low 𝑠 values. However, for task (i), setting 𝑠 = 7 instead of 1 is feasible. Therefore, for tasks except band gap [7, 8, 9], a bigger 𝑠 is necessary. Red points stand for newly evaluated MAE. Blue points stand for the minimum. The yellow circle indicates the optimal configuration and its outcome.
7.5. Experiment set 4: Case study on difference between approximate data and real dataset Finally, in case of inappropriate results using the synthesized data, we conducted a case study comparing models trained on the approximate data computed with JARVIS dataset [25] versus real data from MP dataset [7]. The tasks
First Author et al.: Preprint submitted to Elsevier
taken are band gap [7, 8, 9] and refractive index[7, 8, 12] prediction. All five models are evaluated. The first data approximation method is taking the band gap value (≤ 0.001 eV) instead of metallicity. The principle of this method is based on the difference in band gap characteristics between metals and nonmetals: metals have no obvious band gap or an extremely small band gap, which enables electrons to move freely and show metallic properties, while nonmetals have a significant band gap. This approximate method is applicable to the preliminary screening of materials, where it can effectively distinguish materials with metallic properties from insulators by means of the band gap criterion. This approach may fail on metalloids. Another data approximation method for calculating the refractive index is based on the dielectric constants (mepsx, √ mepsy, and mepsz), using the formula 𝑛 = 𝜀. The principle of this method lies in the relationship between the refractive index and the relative permittivity: for most non-magnetic Page 13 of 16
AutoMatBench: An Automatic Optimization Toolkit
Figure 6: This figure shows two examples [(a) and (b)] of optimization in Experiment 2. The optimal configurations are found within 12 steps. The full results are presented in the appendix.
Figure 7: This figure shows two examples [(a) and (b)] of optimization in Experiment 3. The optimal configurations are found within 12 steps. The full results are presented in the appendix.
materials, the relative permeability is approximately 1, so the refractive index can be approximated as the square root of the relative permittivity, which describes the material’s response to an electric field. This approach may fail when materials are non-magnetic [42]. It may also fail when the frequency dependence of the dielectric constant and the anisotropy of the material must be considered. The evaluation results are shown in Table 1. For any AI model and each task, the discrepancy between MAE on the approximate dataset and MAE on the real dataset is within 5.4% times. This outcome is an acceptable error. Therefore, we conclude that no significant influence on using the approximate datasets exist. This experiment justifies the soundness of this research.
Table 1 The results of the case study. This table reports that approximation adopted in our study is reliable. Task Band gap Band gap Band gap Band gap Band gap Refractive index Refractive index Refractive index Refractive index Refractive index
Model ALIGNN SchNet CrabNet RF-SCM MEGNet ALIGNN SchNet CrabNet RF-SCM MEGNet
Approx 0.4811 0.7149 0.5400 0.4016 0.6773 0.1993 0.3035 0.2298 0.2219 0.2281
Real 0.4778 0.7584 0.5401 0.3895 0.6233 0.1938 0.2760 0.2459 0.2263 0.2304
Diff 0.33%↑ 4.35%↓ 0.01%↓ 1.21%↑ 5.40%↑ 0.54%↑ 2.76%↑ 3.38%↓ 0.44%↓ 0.24%↓
8. Limitation Despite the promising efficacy and practical insights delivered by AutoMatBench in both ID and OOD benchmarking, some limitations remain to be addressed. The current evaluation scope is constrained to six representative MatBench tasks and five mainstream AI models. It First Author et al.: Preprint submitted to Elsevier
Page 14 of 16
AutoMatBench: An Automatic Optimization Toolkit
excludes a broader spectrum of advanced prediction tasks in MatBench, such as emerging large-scale material screening models. Additionally, the OOD construction paradigm adopted in this work inherits clustering-based data partitioning from prior studies, which only covers structural and compositional distribution shifts while ignoring other realistic OOD possibility. The current tool supports neither multi-modal data fusion nor cross-domain migration evaluation across inorganic crystals, polymers, and amorphous materials. This is still distant from the original intention of AI for materials – accelerating the discovery of novel and excellent materials. The future work is threefold. First, a larger scale of benchmarking on more models and more tasks. Second, attributing concrete causes of evaluated results to inner structures of AI prediction models. Finally, extension of AutoMatBench for the support of multi-modal data fusion and cross-domain evaluation across diverse material systems for better material discovery.
9. Conclusion This study proposes AutoMatBench, an automatic toolkit integrated with Bayesian optimization. Experimental results show that, in all tasks, within twelve optimization steps, AutoMatBench can identify target-compliant configurations for all involved tasks and AI models, which aligns with existing research with no big gaps and reduces over half of the experimental cost. Based on our experimental results and comprehensive statistical analysis across all evaluated tasks, the majority of the optimal suggested configuration settings converge to a regime with the system size parameter satisfying 𝑛 ≤ 24 and the hyperparameter 𝑠 fixed at either 7 or 8. Notably, this general trend holds consistently for most property prediction tasks investigated in this work. The only exception is identified for the band gap prediction task, the best-performing and most robust configuration uniquely adopts 𝑠 = 1. Furthermore, the ratio hyperparameter 𝑟 exhibits negligible influence on the final model performance, showing no significant difference between using MP or JARVIS dataset. Consequently, the tuning of 𝑟 can be largely deprioritized in practical configuration optimization for the proposed framework. Increasing the optimization steps beyond twelve may yield better configurations, yet the performance improvement is limited. AutoMatBench provides an efficient tool with innovative insights in enriching relevant theoretical and empirical research in the AI for materials.
References [1] Dunn, A., Wang, Q., Ganose, A., Dopp, D., Jain, A., 2020. Benchmarking materials property prediction methods: the matbench test set and automatminer reference algorithm. npj Computational Materials 6, 138. [2] Omee, S.S., Fu, N., Dong, R., Hu, M., Hu, J., 2024. Structure-based out-of-distribution (ood) materials property prediction: a benchmark study. npj Computational Materials 10, 144.
First Author et al.: Preprint submitted to Elsevier
[3] Fung, V., Zhang, J., Juarez, E., Sumpter, B.G., 2021. Benchmarking graph neural networks for materials chemistry. npj Computational Materials 7, 84. [4] Segal, N., Netanyahu, A., Greenman, K.P., Agrawal, P., GómezBombarelli, R., 2025. Known unknowns: Out-of-distribution property prediction in materials and molecules. npj Computational Materials 11, 345. [5] Tan, L., Chen, P., Liu, M., Wang, X., Cen, J., Zou, Q., 2025. Benchmarking gnns for ood materials property prediction with uncertainty quantification. arXiv preprint arXiv:2511.11697 . [6] Li, Q., Miklaucic, N., Hu, J., 2025. Out-of-distribution material property prediction using adversarial learning. The Journal of Physical Chemistry C 129, 6372–6385. [7] Jain, A., Ong, S.P., Hautier, G., Chen, W., Richards, W.D., Dacek, S., Cholia, S., Gunter, D., Skinner, D., Ceder, G., et al., 2013. Commentary: The materials project: A materials genome approach to accelerating materials innovation. APL materials 1. [8] Ong, S.P., Cholia, S., Jain, A., Brafman, M., Gunter, D., Ceder, G., Persson, K.A., 2015. The materials application programming interface (api): A simple, flexible and efficient api for materials data based on representational state transfer (rest) principles. Computational Materials Science 97, 209–215. [9] Zhuo, Y., Mansouri Tehrani, A., Brgoch, J., 2018. Predicting the band gaps of inorganic solids by machine learning. The journal of physical chemistry letters 9, 1668–1673. [10] De Jong, M., Chen, W., Angsten, T., Jain, A., Notestine, R., Gamst, A., Sluiter, M., Krishna Ande, C., Van Der Zwaag, S., Plata, J.J., et al., 2015. Charting the complete elastic properties of inorganic crystalline compounds. Scientific data 2, 150009. [11] Castelli, I.E., Landis, D.D., Thygesen, K.S., Dahl, S., Chorkendorff, I., Jaramillo, T.F., Jacobsen, K.W., 2012. New cubic perovskites for one-and two-photon water splitting using the computational materials repository. Energy & Environmental Science 5, 9034–9043. [12] Petousis, I., Mrdjenovich, D., Ballouz, E., Liu, M., Winston, D., Chen, W., Graf, T., Schladt, T.D., Persson, K.A., Prinz, F.B., 2017. Highthroughput screening of inorganic compounds for the discovery of novel dielectric and optical materials. Scientific data 4, 160134. [13] Choudhary, K., DeCost, B., 2021. Atomistic line graph neural network for improved materials property predictions. npj Computational Materials 7, 185. [14] Schütt, K., Kindermans, P.J., Sauceda Felix, H.E., Chmiela, S., Tkatchenko, A., Müller, K.R., 2017. Schnet: A continuous-filter convolutional neural network for modeling quantum interactions. Advances in neural information processing systems 30. [15] Schütt, K.T., Sauceda, H.E., Kindermans, P.J., Tkatchenko, A., Müller, K.R., 2018. Schnet–a deep learning architecture for molecules and materials. The Journal of chemical physics 148. [16] Wang, A.Y.T., Kauwe, S.K., Murdock, R.J., Sparks, T.D., 2021. Compositionally restricted attention-based network for materials property predictions. Npj Computational Materials 7, 77. [17] Kovacs, E., 2021. The lsst-desc 3x2pt tomography optimization challenge. The Open Journal of Astrophysics . [18] Faber, F., Lindmaa, A., Von Lilienfeld, O.A., Armiento, R., 2015. Crystal structure representations for machine learning models of formation energies. International Journal of Quantum Chemistry 115, 1094–1101. [19] Ward, L., Agrawal, A., Choudhary, A., Wolverton, C., 2016. A general-purpose machine learning framework for predicting properties of inorganic materials. npj Computational Materials 2, 16028. [20] Chen, C., Ye, W., Zuo, Y., Zheng, C., Ong, S.P., 2019. Graph networks as a universal machine learning framework for molecules and crystals. Chemistry of Materials 31, 3564–3572. [21] Frazier, P.I., 2018. A tutorial on bayesian optimization. arXiv preprint arXiv:1807.02811 . [22] Perdew, J.P., Burke, K., Ernzerhof, M., 1996. Generalized gradient approximation made simple. Physical review letters 77, 3865. [23] Dion, M., Rydberg, H., Schröder, E., Langreth, D.C., Lundqvist, B.I., 2004. Van der waals density functional for general geometries.
Page 15 of 16
AutoMatBench: An Automatic Optimization Toolkit Physical review letters 92, 246401. [24] Saal, J.E., Kirklin, S., Aykol, M., Meredig, B., Wolverton, C., 2013. Materials design and discovery with high-throughput density functional theory: the open quantum materials database (oqmd). Jom 65, 1501–1509. [25] Choudhary, K., Garrity, K.F., Reid, A.C., DeCost, B., Biacchi, A.J., Hight Walker, A.R., Trautt, Z., Hattrick-Simpers, J., Kusne, A.G., Centrone, A., et al., 2020. The joint automated repository for various integrated simulations (jarvis) for data-driven materials design. npj computational materials 6, 173. [26] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L., 2009. Imagenet: A large-scale hierarchical image database, in: 2009 IEEE conference on computer vision and pattern recognition, Ieee. pp. 248– 255. [27] Zhan, J., Wang, L., Gao, W., Wang, C., Li, H., Fan, F., Kang, G., 2025. Evaluatology: The Science of Uncovering the Effects. Version 1.0 (first edition, 2025) ed., BenchCouncil Press, Hong Kong. [28] Amari, S.i., 1993. Backpropagation and stochastic gradient descent method. Neurocomputing 5, 185–196. [29] Kingma, D.P., Ba, J., 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 . [30] Li, Y., Zhan, J., 2022. Saibench: Benchmarking ai for science. BenchCouncil Transactions on Benchmarks, Standards and Evaluations 2, 100063. [31] Li, Y., Zhan, J., 2023. Saibench: A structural interpretation of ai for science through benchmarks. arXiv preprint arXiv:2311.17869 . [32] Xie, T., Grossman, J.C., 2018. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Physical review letters 120, 145301. [33] Omee, S.S., Louis, S.Y., Fu, N., Wei, L., Dey, S., Dong, R., Li, Q., Hu, J., 2022. Scalable deeper graph neural networks for high-performance materials property prediction. Patterns 3. [34] De Breuck, P.P., Hautier, G., Rignanese, G.M., 2021. Materials property prediction for limited datasets enabled by feature selection and joint learning with modnet. npj computational materials 7, 83. [35] Ahmed, M., Seraj, R., Islam, S.M.S., 2020. The k-means algorithm: A comprehensive survey and performance evaluation. Electronics 9, 1295. [36] Wainer, J., Cawley, G., 2021. Nested cross-validation when selecting classifiers is overzealous for most practical applications. Expert Systems with Applications 182, 115222. [37] Van der Maaten, L., Hinton, G., 2008. Visualizing data using t-sne. Journal of machine learning research 9. [38] Parzen, E., 1962. On estimation of a probability density function and mode. The annals of mathematical statistics 33, 1065–1076. [39] Madsen, G.K., 2007. Functional form of the generalized gradient approximation for exchange: The pbe 𝛼 functional. Physical Review B—Condensed Matter and Materials Physics 75, 195108. [40] Voigt, W., 1908. Lehrbuch der kristallphysik (teubner, leipzig, 1928). There is no corresponding record for this reference . [41] Schütt, K.T., Arbabzadah, F., Chmiela, S., Müller, K.R., Tkatchenko, A., 2017. Quantum-chemical insights from deep tensor neural networks. Nature communications 8, 13890. [42] Wangberg, R., Elser, J., Narimanov, E.E., Podolskiy, V.A., 2006. Nonmagnetic nanocomposites for optical and infrared negative-refractiveindex media. Journal of the Optical Society of America B 23, 498– 505.
First Author et al.: Preprint submitted to Elsevier
A. Full results of Experiment set 1, 2, and 3 This section presents the complete experimental results for Experiment sets 1, 2, and 3. As illustrated in Figure 8 and Figure 9, the detailed optimization processes and corresponding outcomes of Experiment 1 are clearly displayed across parts (a) to (j), covering the entire pipeline from initialization to final convergence. The same full result data from Experiment sets 2 and 3 are also provided. Collectively, these results validate the effectiveness and stability of the proposed method across multiple experimental settings.
Page 16 of 16
AutoMatBench: An Automatic Optimization Toolkit
Figure 8: This figure shows the entire processesand results of optimization in Experiment 1. (Part(a)∼(f))
First Author et al.: Preprint submitted to Elsevier
Page 17 of 16
AutoMatBench: An Automatic Optimization Toolkit
Figure 9: This figure shows the entire processesand results of optimization in Experiment 1. (Part(g)∼(j))
First Author et al.: Preprint submitted to Elsevier
Page 18 of 16
AutoMatBench: An Automatic Optimization Toolkit
Figure 10: This figure shows the entire processesand results of optimization in Experiment 2. Theoptimization aim is the maximize the performancegap between models. [Part (a)∼(f)]
First Author et al.: Preprint submitted to Elsevier
Page 19 of 16
AutoMatBench: An Automatic Optimization Toolkit
Figure 11: This figure shows the entire processesand results of optimization in Experiment 3. Theoptimization aim is the minimize the performancedistance with the ground-truth results. [Part (a)∼(f)]
First Author et al.: Preprint submitted to Elsevier
Page 20 of 16
AutoMatBench: An Automatic Optimization Toolkit
Figure 12: This figure shows the entire processesand results of optimization in Experiment 3. Theoptimization aim is the minimize the performancedistance with the ground-truth results. [Part (g)∼(l)]
First Author et al.: Preprint submitted to Elsevier
Page 21 of 16