Assessing Model Fit by Cross-Validation

Abstract

When QSAR models are fitted, it is important to validate any fitted model-to check that it is plausible that its predictions will carry over to fresh data not used in the model fitting exercise. There are two standard ways of doing this-using a separate hold-out test sample and the computationally much more burdensome leave-one-out cross-validation in which the entire pool of available compounds is used both to fit the model and to assess its validity. We show by theoretical argument and empiric study of a large QSAR data set that when the available sample size is small-in the dozens or scores rather than the hundreds, holding a portion of it back for testing is wasteful, and that it is much better to use cross-validation, but ensure that this is done properly.

A new computer program for QSAR‐analysis: ARTE‐QSAR

“…The use of a large portion of the data for checking the model fit seems a waste of valuable and often costly information. In those cases where external validation is not possible, alternatives for predictive validation are of interest, for example, methods such as crossvalidation 18 and Y-randomization. Crossvalidation is a common technique where a number of modified data sets are created by deleting, in each case, one or a small group of compounds from the data.…”

Use of Graph Invariants in Quantitative Structure-Activity Relationship Studies

“…[27] One also needs to check the ability of QSAR models in providing competent predictions on 'similar' datasets via validation on out-of-sample test sets. [28][29][30][31][32] For a relatively small sample, i.e., a small collection of compounds, this is done by following a leave-one-out (LOO) cross-validation method. For data sets with a large number of compounds, a more computationally economical way is to do a k-fold cross-validation: split the data set randomly into k (previously decided) equal subsets, take each subset in turn as test set and use the remaining compounds as training sets and use the model to obtain predictions.…”

“…Essentially this method ends up using information from the holdout compound/ split subset to predict activity of those very samples. This naïve cross-validation procedure causes synthetic inflation of the cross-validated q 2 , hence compromises the predictive ability of the model [29][30][31][32] (Figure 3). A two-step approach (referred in Figure 3 as 'Two-deep CV') helps avoid this tricky situation.…”

“…In the calculation of q 2 in the rank deficient case, one must follow the two-deep cross-validation procedure; otherwise the calculated q 2 will reflect overfitting. [28][29][30][31] In HiQSAR modeling, we found that of the four types of calculated molecular descriptors, viz., TS, TC, 3-D, and QC indices, in most cases a TS + TC combination gave good quality models; the addition of 3-D or QC descriptors after the utilization of TS and TC combination did not improve the model quality significantly. This is a good news in view of the fact that we already reached the age of big data [57] and easily calculated indices like TS and TC descriptors, if they give good models in many areas, could find wide applications in the in silico evaluation of chemicals.…”

Identification of inhibitors of the nicotine metabolising CYP2A6 enzyme—an in silico approach

“…The stability of the models was tested by cross-validation with two and five groups ( Table 1). As described previously, 26 the crossvalidation procedure provides a reliable picture of the predictivity of QSAR models. All the statistical values obtained from our current CoMFA and CoMSIA models Table 1).…”