Model validation by permutation tests: Applications to variable selection

Abstract

Regression rnodel validation by permutation tests was explored. Especially in cases where the rnodel significance is doubtful, a permutation test adds crucial information which can often can be decisive for the existence of the rnodel. The background and applicability of the test procedure are described. As an example, the use of permutation tests was extended to validation and investigation of four predictor variable selection techniques, namely MUSEUM, GOLPE, VIP and IVS-PLS. "he selection methods are briefly reviewed and cornpared. The permutation tests were applied before, during and after variable selection. Some similarities and differences in the behaviour of the variable selection techniques were found and are cornmented upon. O

Pharmacometabolomic Approach to Predict QT Prolongation in Guinea Pigs

“…Internal validation was also performed with this PLS model to check its ability to predict new observations with no risk of overfitting the data and capturing non-systematic correlations between the metabolites and QTc values. The goodness of fit (R 2 ) and predictability (Q 2 ) measures from 20 and 100 random permutation experiments [64] are shown in Figure 5B and 5C. All R 2 and Q 2 values from the permutation experiments (left) were smaller than those of the PLS model (far right).…”

Effect Symmetry of Benefit Criteria in Postpurchase Evaluations

“…To test structural invariance of the path relationships, a series of permutation tests (Chin & Dibbern, ; Lindgren, Hansen, Karcher, Sjöström, & Eriksson, ) were performed. Specifically, the data were split into two subsamples of equal size.…”

A kernel‐based approach for fault diagnosis in batch processes

“…In order to validate the final model, a permutation test was performed. Supporting information SI.6 shows the validation plots obtained for the BWU kernel matrix.…”