Prediction uncertainty validation for computational chemists

Abstract

Validation of prediction uncertainty (PU) is becoming an essential tool for modern computational chemistry. Designed to quantify the reliability of predictions in meteorology, the calibration-sharpness (CS) framework is now widely used to optimize and validate uncertainty-aware machine learning (ML) methods. However, its application is not limited to ML and it can serve as a principled framework for any PU validation. The present article is intended as a step-by-step introduction to the concepts and techniques of PU validation in the CS framework, adapted to the specifics of computational chemistry. The presented methods range from elementary graphical checks to more sophisticated ones based on local calibration statistics. The concept of tightness, is introduced. The methods are illustrated on synthetic datasets and applied to uncertainty quantification data extracted from the computational chemistry literature.

Uncertain of uncertainties? A comparison of uncertainty quantification metrics for chemical data sets

“…Another important aspect of the miscalibration area is the assumption of normal errors. As highlighted by Pernot, such assumptions adds a fragility to the metric, since a non-zero miscalibration area can be interpreted as both a sign that the uncertainties are not calibrated or that the assumption of normal errors was wrong [5]. As an alternative, Pernot suggests the Var(Z) ?…”

“…Plotting RMSE vs. RMV should then produce a linear plot with slope 1 and intercept 0. As suggested by Pernot we add 95% confidence intervals to the binned RMSE values calculated by the BC a bootstrap method [5].…”

“…= 1 changes to the ⟨Z 2 ⟩ ? = 1 test for average calibration [5]. The change from Var(Z) to ⟨Z 2 ⟩ amounts to the addition of ⟨Z⟩ 2 (zero for unbiased errors).…”

“…The miscalibration area is calculated based on an assumption of Gaussian distributed errors, while the Var(Z) ? = 1 test does not assume that [5]. Thus, the miscalibration area test evaluates the agreement of the Z-distribution being Gaussian with variance = 1, while the Var(Z) tests if the Z-distribution has variance = 1, but make no assumption on the distribution.…”

“…With the vast and diverse nature of chemical space, a model with the same low error across chemical space is currently not realistic. Therefore, attention within the chemical machine learning community has lately turned towards quantifying the uncertainty on property predictions made by machine learning methods [1,2,3,4,5,6,7].…”