Beware of R²: Simple, Unambiguous Assessment of the Prediction Accuracy of QSAR and QSPR Models

Abstract

The statistical metrics used to characterize the external predictivity of a model, i.e., how well it predicts the properties of an independent test set, have proliferated over the past decade. This paper clarifies some apparent confusion over the use of the coefficient of determination, R2, as a measure of model fit and predictive power in QSAR and QSPR modelling. R2 (or R2) has been used in various contexts in the literature in conjunction with training and test data, for both ordinary linear regression and regression through the origin as well as with linear and nonlinear regression models. We analyze the widely adopted model fit criteria suggested by Golbraikh and Tropsha1 in a strict statistical manner. Shortcomings in these criteria are identified and a clearer and simpler alternative method to characterize model predictivity is provided. The intent is not to repeat the well-documented arguments for model validation using test data, but to guide the application of R2 as a model fit statistic. Examples are used to illustrate both correct and incorrect use of R2. Reporting the root mean squared error or equivalent measures of dispersion, typically of more practical importance than R2, is also encouraged and important challenges in addressing the needs of different categories of users such as computational chemists, experimental scientists, and regulatory decision support specialists are outlined.

Development of predictive QSAR models for Vibrio fischeri toxicity of ionic liquids and their true external and experimental validation tests

“…It is sometimes more relevant for a good model to appropriately categorize the test chemicals as toxic or non-toxic, and to maintain a correct rank order prediction, rather than to deliver quantitatively precise predictions. 70 In our present study, the developed models could successfully predict the designed chemicals as "harmless" or "practically harmless". Summing up, this work allowed the development of predictive models with good predictability performance considering the ionic liquids' chemical structure and their associated toxicity.…”

Variable Selection in Time Series Forecasting Using Random Forests

“…A discussion on the appropriateness of the metrics to measure the performance of the forecasting methods can be found in [57] (Chapter 2.5) and [63], while [64,65] extensively discuss metrics connected to the regression coefficient. Here we preferred to use the metrics of Table 4, due to their simplicity.…”

Complexity Analysis of Neonatal EEG Using Multiscale Entropy: Applications in Brain Maturation and Sleep Stage Classification

“…where y test denotes the observed response variable, y test is its mean, andŷ test is the corresponding predicted value [25]. Finally, the PMA estimation is evaluated on the test set using the square root of the mean squared error ( √ MSE).…”