The C1C2: A framework for simultaneous model selection and assessment

Abstract

Background: There has been recent concern regarding the inability of predictive modeling approaches to generalize to new data. Some of the problems can be attributed to improper methods for model selection and assessment. Here, we have addressed this issue by introducing a novel and general framework, the C 1 C 2 , for simultaneous model selection and assessment. The framework relies on a partitioning of the data in order to separate model choice from model assessment in terms of used data. Since the number of conceivable models in general is vast, it was also of interest to investigate the employment of two automatic search methods, a genetic algorithm and a brute-force method, for model choice. As a demonstration, the C 1 C 2 was applied to simulated and real-world datasets. A penalized linear model was assumed to reasonably approximate the true relation between the dependent and independent variables, thus reducing the model choice problem to a matter of variable selection and choice of penalizing parameter. We also studied the impact of assuming prior knowledge about the number of relevant variables on model choice and generalization error estimates. The results obtained with the C 1 C 2 were compared to those obtained by employing repeated K-fold cross-validation for choosing and assessing a model.

Benchmarking Variable Selection in QSAR

“…One way to address this difficulty is to use a double loop cross‐validation technique, where all of the variable selection is carried out in the “inner” loop and the derived model then tested against the left‐out test set in the “outer” loop of the procedure. This has been shown by Stone ,13 Eklund et al 14. and Freyhult et al 15.…”

SimSel: a new simulation method for variable selection

“…These results align very well with those presented in [17], where different variable selection and regression coefficient shrinkage methods were applied to the prostate data. The results presented in Figure 4(b) show clearly what is previously known (see [18,19]) about the importance of the six variables in the truncated Selwood data set.…”

“…The dependent variable is the anti-filarial activity of the molecules measured in vitro. Both data sets have been extensively studied by others (see, for instance, respectively, [17] and [18,19], and references therein) and are well characterized in terms of which independent variables are relevant for modelling the response variable. Since SimSel was derived under the assumption that (H T H) −1 exists and the Selwood data contain more variables than observations, we used only six independent variables from the Selwood data: LOGP, SUM_F, and MOFI_Y known to be important and ATCH10, PEAX_Y, and S8_1DX known not to be important (see [16] for details about the variables and [18,19] for discussions about important variables in the Selwood data).…”

Choosing Feature Selection and Learning Algorithms in QSAR

“…Equation 14 is not needed for regression, and eq 13 is changed so that there is only a single k.) The parameters were estimated (15) where N k (x) is the neighborhood defined by the k closest observations in the training set. We here use the Euclidean distance to determine "closeness".…”