Random projections and kernelised leave one cluster out cross validation: universal baselines and evaluation tools for supervised machine learning of material properties

Durdy, Samantha; Gaultois, Michael W.; Gusev, Vladimir V.; Bollegala, Danushka; Rosseinsky, Matthew J.

doi:10.1039/d2dd00039c

Cited by 7 publications

(8 citation statements)

References 47 publications

(135 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We used four-fold cross-validation methods inside the training set to tune the hyperparameters. One of them was conventional k-fold CV, while the other three were modified with respect to the previous methods 44,[49][50][51][52] to enhance extrapolative performance, as shown in Supplementary Fig. 11.…”

Section: Validation Methodsmentioning

confidence: 99%

“…While these AD designs aim to differentiate reliable interpolation data from unreliable extrapolation data, they do not significantly enhance extrapolative performance. Meanwhile, various validation methods have been proposed to improve extrapolative performance by considering structure bias 44,49,50 and property bias 51,52 . DL techniques such as generative models 37 , and Neural Network Potential (NNP) models 53 have also shown potential for improving extrapolation tasks.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Extrapolative prediction of small-data molecular property using quantum mechanics-assisted machine learning

Shimakawa,

Kumada,

Sato

2024

npj Comput Mater

View full text Add to dashboard Cite

Data-driven materials science has realized a new paradigm by integrating materials domain knowledge and machine-learning (ML) techniques. However, ML-based research has often overlooked the inherent limitation in predicting unknown data: extrapolative performance, especially when dealing with small-scale experimental datasets. Here, we present a comprehensive benchmark for assessing extrapolative performance across 12 organic molecular properties. Our large-scale benchmark reveals that conventional ML models exhibit remarkable performance degradation beyond the training distribution of property range and molecular structures, particularly for small-data properties. To address this challenge, we introduce a quantum-mechanical (QM) descriptor dataset, called QMex, and an interactive linear regression (ILR), which incorporates interaction terms between QM descriptors and categorical information pertaining to molecular structures. The QMex-based ILR achieved state-of-the-art extrapolative performance while preserving its interpretability. Our benchmark results, QMex dataset, and proposed model serve as valuable assets for improving extrapolative predictions with small experimental datasets and for the discovery of novel materials/molecules that surpass existing candidates.

show abstract

Section: Validation Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Extrapolative prediction of small-data molecular property using quantum mechanics-assisted machine learning

Shimakawa,

Kumada,

Sato

2024

npj Comput Mater

View full text Add to dashboard Cite

show abstract

“…If we are interested in assessing model performance on new molecules, we can train a model with many reaction templates but use substructure splitting to create training, validation, and testing sets. Bemis-Murcko scaffolds [70] are commonly used to partition the data for this purpose, though clustering based on other input features or chemical similarity to measure extrapolation has also been explored [23,[71][72][73][74][75][76][77][78][79][80][81][82][83][84][85][86][87][88] as has quantifying domains of model applicability [89][90][91][92][93]. Scaffold splitting is not perfect, but by ensuring that molecules in the testing set are structurally different than those in the training set, it offers a much better assessment of generalizability than splitting randomly [17,24,67,[94][95][96][97][98][99][100][101][102][103][104][105][106][107][108][109]…”

Section: Interpolation Vs Extrapolationmentioning

confidence: 99%

Comment on ‘Physics-based representations for machine learning properties of chemical reactions’

Spiekermann,

Stuyver,

Pattanaik

et al. 2023

Mach. Learn.: Sci. Technol.

View full text Add to dashboard Cite

In a recent article in this journal, van Gerwen et al (2022 Mach. Learn.: Sci. Technol. 3 045005) presented a kernel ridge regression model to predict reaction barrier heights. Here, we comment on the utility of that model and present references and results that contradict several statements made in that article. Our primary interest is to offer a broader perspective by presenting three aspects that are essential for researchers to consider when creating models for chemical kinetics: (1) are the model’s prediction targets and associated errors sufficient for practical applications? (2) Does the model prioritize user-friendly inputs so it is practical for others to integrate into prediction workflows? (3) Does the analysis report performance on both interpolative and more challenging extrapolative data splits so users have a realistic idea of the likely errors in the model’s predictions?

show abstract

“…While open questions undoubtedly exist surrounding interpolation vs. extrapolation as well as designing challenging out-of-sample test sets such as those containing new chemistry [26,28,[42][43][44][45][46][47][48][49], larger [9,50,51] and/or outlier [52] molecules/materials, or new sets appearing over time [53][54][55], this topic remains an ongoing community-wide discussion with no clear best practices, and will not be discussed further here.…”

Section: Data Splitsmentioning

confidence: 99%

Reply to Comment on ‘Physics-based representations for machine learning properties of chemical reactions’

van Gerwen,

Wodrich,

Laplaza

et al. 2023

Mach. Learn.: Sci. Technol.

View full text Add to dashboard Cite

Recently, we published an article in this journal that explored physics-based representations in combination with kernel models for predicting reaction properties (i.e. TS barrier heights). In an anonymous comment on our contribution, the authors argue, amongst other points, that deep learning models relying on atom-mapped reaction SMILES are more appropriate for the same task. This raises the question: are deep learning models sounding the death knell for kernel based models? By studying several datasets that vary in the type of chemical (i.e. high-quality atom-mapping) and structural information (i.e. Cartesian coordinates of reactants and products) contained within, we illustrate that physics-based representations combined with kernel models are competitive with deep learning models. Indeed, in some cases, such as when reaction barriers are sensitive to the geometry, physics-based models represent the only viable candidate. Furthermore, we illustrate that the good performance of deep learning models relies on high-quality atom-mapping, which comes with significant human time-cost and, in some cases, is impossible. As such, both physics-based and graph models offer their own relative benefits to predict reaction barriers of differing datasets.

show abstract

Random projections and kernelised leave one cluster out cross validation: universal baselines and evaluation tools for supervised machine learning of material properties

Abstract: With machine learning being a popular topic in current computational materials science literature, creating representations for compounds has become common place. These representations are rarely compared, as evaluating their performance...

Cited by 7 publications

References 47 publications

Extrapolative prediction of small-data molecular property using quantum mechanics-assisted machine learning

Extrapolative prediction of small-data molecular property using quantum mechanics-assisted machine learning

Comment on ‘Physics-based representations for machine learning properties of chemical reactions’

Reply to Comment on ‘Physics-based representations for machine learning properties of chemical reactions’

Contact Info

Product

Resources

About