Model validation by permutation tests: Applications to variable selection

Abstract

Regression rnodel validation by permutation tests was explored. Especially in cases where the rnodel significance is doubtful, a permutation test adds crucial information which can often can be decisive for the existence of the rnodel. The background and applicability of the test procedure are described. As an example, the use of permutation tests was extended to validation and investigation of four predictor variable selection techniques, namely MUSEUM, GOLPE, VIP and IVS-PLS. "he selection methods are briefly reviewed and cornpared. The permutation tests were applied before, during and after variable selection. Some similarities and differences in the behaviour of the variable selection techniques were found and are cornmented upon. O

SENYAWA INHIBITOR Α-Glukosidase DAN ANTIOKSIDAN DARI KUMIS KUCING DENGAN PENDEKATAN METABOLOMIK BERBASIS FTIR

“…Nilai p yang menjadi patokan pada metode validasi CV ANOVA yang diperoleh pada penelitian ini sebesar 0,03, lebih kecil dari nilai maksimum yang diterima, yaitu P<0,05 sehingga model dianggap memiliki reabilitas yang baik (Eriksson et al, 2008). Selain itu, test permutasi juga dilakukan untuk mengetahui seberapa besar terjadinya korelasi yang tidak disengaja akibat jumlah sampel yang kecil dengan variabel yang berjumlah besar (Lindgren et al, 1996). Model yang valid seharusnya memiliki nilai intersep R 2 dan Q 2 pada sumbu Y berturut-turut tidak lebih dari 0,3-0,4 (Eriksson et al, 2006).…”

Cordyceps sinensis protects against liver and heart injuries in a rat model of chronic kidney disease: a metabolomic analysis

“…A cluster of 500 permutated models from the first component was visualized using validation plots (Figures 3B, 3D). In the permutation test plots, all of the permutated Q2 values to the left were www.nature.com/aps Liu X et al Acta Pharmacologica Sinica npg lower than the original point to the right [39,40] , which indicated that the original models were valid. Moreover, according to coefficient numbers (|r|), VIP and P values, the major discriminatory metabolites for each group were determined in the pair-wise comparison.…”

Guidance Document on the Validation of (Quantitative) Structure-Activity Relationship [(Q)SAR] Models

“…The test is performed by calculating the quality of the model (usually R 2 or, better, Q 2 ) randomly modifying the sequence of the response vector y, i.e. by assigning to each compound a response randomly selected from the true set of responses ( Figure 5.4) (Lindgren et al, 1996). If the original model has no chance correlation, there is a significant difference in the quality of the original model and that associated with a model obtained with random responses (Figure 5.5).…”