Robust model benchmarking and bias-imbalance in data-driven materials science: a case study on MODNet

Abstract

As the number of novel data-driven approaches to material science continues to grow, it is crucial to perform consistent quality, reliability and applicability assessments of model performance. In this paper, we benchmark the Materials Optimal Descriptor Network (MODNet) method and architecture against the recently released MatBench v0.1, a curated test suite of materials datasets. MODNet is shown to outperform current leaders on 4 of the 13 tasks, whilst closely matching the current leaders on a further 3 tasks; MODNet performs particularly well when the number of samples is below 10,000. Attention is paid to two topics of concern when benchmarking models. First, we encourage the reporting of a more diverse set of metrics as it leads to a more comprehensive and holistic comparison of model performance. Second, an equally important task is the uncertainty assessment of a model towards a target domain. By applying a distance metric in feature space, we found that significant variations in validation errors can be observed, depending on the imbalance and bias in the training set (i.e., similarity between training and application space). Both issues are often overlooked, yet important for successful real-world applications of machine learning in materials science and condensed matter.

Formula Graph Self‐Attention Network for Representation‐Domain Independent Materials Discovery

“…It is known that non‐graph models such as Automatminer [ 54 ] and MODNet [ 62 , 63 ] that perform extensive hyperparameter tuning and feature selection steps at the inner loop of NCV typically outperform GNNs on small datasets. Once the internal optimization is complete, such models are fit on the entire fold so that no validation data is left out.…”

Accurate experimental band gap predictions with multifidelity correction learning

“…The GA keeps randomness, while giving more importance to local optima. Therefore, a satisfactory set of hyperparameters is found more quickly and at a reduced computational cost compared to the standard gridor random-search previously used [31] . As is shown below, this approach results in a relative improvement of up to 12% on the Matbench tasks, compared to the previously used grid-search.…”

“…This is especially the case for the smaller experimental datasets. The new approach brings a significant increase in performance compared to the standard grid-or random-search that was adopted previously [31] . This shows the importance of hyperparameters for accurate generalization.…”

Predicting Solid State Material Platforms for Quantum Technologies

“…Additionally, they provide the tools to construct possibly thousands of features from calculations based on a material's composition, structure and electronic properties from DFT calculations, and have frameworks for visualization and automatic machine learning. To apply Matminer's featurization tools, we extend an existing implementation by Breuck et al [94], which was used to generate a supervised machine learning framework called the MODnet. The implementation by Breuck et al provides featurization for a material's composition, structure and atomic sites.…”