Principles of Proper Validation: use and abuse of re‐sampling for validation

Abstract

Validation in chemometrics is presented using the exemplar context of multivariate calibration/prediction. A phenomenological analysis of common validation practices in data analysis and chemometrics leads to formulation of a set of generic Principles of Proper Validation (PPV), which is based on a set of characterizing distinctions: (i) Validation cannot be understood by focusing on the methods of validation only; validation must be based on full knowledge of the underlying definitions, objectives, methods, effects and consequences-which are all outlined and discussed here. (ii) Analysis of proper validation objectives implies that there is one valid paradigm only: test set validation. (iii) Contrary to much contemporary chemometric practices (and validation myths), cross-validation is shown to be unjustified in the form of monolithic application of a one-for-all procedure (segmented cross-validation) on all data sets. Within its own design and scope, cross-validation is in reality a sub-optimal simulation of test set validation, crippled by a critical sampling variance omission, as it manifestly is based on one data set only (training data set). Other re-sampling validation methods are shown to suffer from the same deficiencies. The PPV are universal and can be applied to all situations in which the assessment of performance is desired: prediction-, classification-, time series forecasting-, modeling validation. The key element of PPV is the Theory of Sampling (TOS), which allow insight into all variance generating factors, especially the so-called incorrect sampling errors, which, if not properly eliminated, are responsible for a fatal inconstant sampling bias, for which no statistical correction is possible. In the light of TOS it is shown how a second data set (test set, validation set) is critically necessary for the inclusion of the sampling errors incurred in all 'future' situations in which the validated model must perform. Logically, therefore, all one data set re-sampling approaches for validation, especially cross-validation and leverage-corrected validation, should be terminated, or at the very least used only with full scientific understanding and disclosure of their detrimental variance omissions and consequences. Regarding PLS-regression, an emphatic call is made for stringent commitment to test set validation based on graphical inspection of pertinent t-u plots for optimal understanding of the X-Y interrelationships and for validation guidance. QSAR/QSAP forms a partial exemption from the present test set imperative with no generalization potential.

The next evolution of MDE: a seamless integration of machine learning into domain modeling

“…When it comes to timeseries, the seasonal effect can introduce a bias when splitting the dataset in equivalent sets, required by the k-fold crossvalidation method [12]. Moreover, in our evaluation we want to demonstrate the accuracy of modelling with micro learning units rather than evaluating the efficiency of the ML algorithm itself.…”

Online acoustic chemometric monitoring of fish feed pellet velocity in a pneumatic conveying system

“…In general, the driving force behind application of multivariate calibration methods is to minimize the time-consuming effort (and cost) of performing actual y-measurements on a process (manual inspection of high speed video recordings in this case). The calibration and validation data were obtained from independent sets of experiments in accordance with the requirements stipulated by Esbensen and Geladi [34] regarding the necessary realism and validity of independent test set validation. Generally, visualization plots and statistical results are used for describing the prediction performance of PLS regression models.…”

Acoustic chemometrics for on-line monitoring and end-point determination of fluidised bed drying

“…There are several validation techniques available [20][21][22]. However, test-set validation has been the recommended validation method because it provides realistic prediction errors and optimal number of PLS-R components [11]. In this regard, over-fitting or under-fitting of the prediction model is avoided.…”

“…The present work is an attempt to adapt acoustic chemometrics for on-line fluidised bed drying progress monitoring and end-point determination using dedicated test material and PLS-R regression models validated with independent test data [11].…”