Calibrated uncertainty for molecular property prediction using ensembles of message passing neural networks

Abstract

Data-driven methods based on machine learning have the potential to accelerate computational analysis of atomic structures. In this context, reliable uncertainty estimates are important for assessing confidence in predictions and enabling decision making. However, machine learning models can produce badly calibrated uncertainty estimates and it is therefore crucial to detect and handle uncertainty carefully. In this work we extend a message passing neural network designed specifically for predicting properties of molecules and materials with a calibrated probabilistic predictive distribution. The method presented in this paper differs from previous work by considering both aleatoric and epistemic uncertainty in a unified framework, and by recalibrating the predictive distribution on unseen data. Through computer experiments, we show that our approach results in accurate models for predicting molecular formation energies with well calibrated uncertainty in and out of the training data distribution on two public molecular benchmark datasets, QM9 and PC9. The proposed method provides a general framework for training and evaluating neural network ensemble models that are able to produce accurate predictions of properties of molecules with well calibrated uncertainty estimates.

Modeling the Solid Electrolyte Interphase: Machine Learning as a Game Changer?

“…The advent of machine and deep learning techniques in the areas of batteries and electrochemical interfaces have added a new dimension to the quest for predictive SEI models, as outlined above. It is, however, important to stress that such data‐driven approaches must be both physics‐informed and uncertainty‐aware [ 279 ] in order to ensure predictive accuracy for spatio‐temporal SEI models. Advanced multi‐scaling approaches in latent space, for example, using hierarchical latent space models, also provide new possibilities for parameter‐free multi scaling and to identify deep descriptors for SEI evolution, as does explainable AI.…”

“…ML potentials can be used for this purpose as discussed in the previous subsection. However, such surrogates are expected to have varying levels of uncertainty in the predicted energies [278,279] requiring reaction network analysis with uncertain data [280] ML models can also help in this aspect both for molecular [281] and solid-state reactions. [282]…”

Implications of the BATTERY 2030+ AI‐Assisted Toolkit on Future Low‐TRL Battery Discoveries and Chemistries

“…The latter is critical to get sufficient data to obtain reliable statistics to derive hyperparameters and descriptors of the materials in their more complicated electrochemical environments. Developing physics‐ and uncertainty‐aware data‐driven methods [ 217 ] capable of training on such multi‐sourced experimental and simulational data will strongly enhance the quality of the deep interface descriptors and features that play a critical role in shortening the path to realizing emerging battery technologies and concepts.…”

Prediction uncertainty validation for computational chemists

“…Uncertainty quantification (UQ) is becoming a major issue for chemical machine learning (ML), 1 notably for the prediction of molecular and material properties. [2][3][4][5][6][7][8] This is also the case for quantum chemistry, when a level of confidence on predictions is sought out. 1,[9][10][11][12][13][14][15][16][17][18][19] In these contexts, the validation of UQ outputs is essential to enable their re-use in applications such as active learning or actionable predictions for the industry.…”

“…The calibration-sharpness (CS) framework 20 provides a principled approach to ML-UQ validation. 4,5,8 Scalia et al 4 distinguish two validation settings: (i) confidence-or interval-based calibration, 21 comparing the empirical coverage of prediction intervals to their intended confidence level; and…”