Sum of ranking differences (SRD) to ensemble multivariate calibration model merits for tuning parameter selection and comparing calibration methods

Kalivas, John H.; Héberger, Károly; Andries, Erik

doi:10.1016/j.aca.2014.12.056

Cited by 42 publications

(51 citation statements)

References 59 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We select as our solution the minimum value of the mean normalized SRD and designate the number of LVs as k ‡ to differentiate from k † used to denote the solution obtained with a single realization of λ used with M 2. A paired Wilcoxon signed rank test between the k = k ‡ solution and all others can be performed to find an alternate number of LVs for which the normalized SRD scores are not statistically significantly different . However, we find that this approach can undermine the constraint on the growth of RMSECV imposed by λ , so it is not used in this work.…”

Section: Methodsmentioning

confidence: 99%

“…For ambient samples in which we lack true reference values, we additionally compare an aggregate estimate of organic carbon (OC) estimated by the sum of FTIR functional groups with the measurements of OC obtained by a different but widely used analytical technique. A set of models selected from an ensemble of model performance curves generated by our proposed metric and combined by sum of ranking differences (SRD) are validated against an independent randomization test , and we further evaluate their suitability for application to laboratory and ambient samples.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Model selection for partial least squares calibration and implications for analysis of atmospheric organic aerosol samples with mid‐infrared spectroscopy

Takahama

Dillner

2015

Journal of Chemometrics

View full text Add to dashboard Cite

In developing partial least squares calibration models, selecting the number of latent variables used for their construction to minimize both model bias and model variance remains a challenge. Several metrics exist for incorporating these trade-offs, but the cost of model parsimony and the potential for underfitting on achievable prediction errors are difficult to anticipate. We propose a metric that penalizes growing model variance against decreasing bias as additional latent variables are added. The magnitude of the penalty is scaled by a user-defined parameter that is formulated to provide a constraint on the fractional increase in root mean square error of cross-validation (RMSECV) when selecting a parsimonious model over the conventional minimum RMSECV solution. We evaluate this approach for quantification of four organic functional groups using 238 laboratory standards and 750 complex atmospheric organic aerosol mixtures with mid-infrared spectroscopy. Parametric variation of this penalty demonstrates that increase in prediction errors due to underfitting is bounded by the magnitude of the penalty for samples similar to laboratory standards used for model training and validation. Imposing an ensemble of penalties corresponding to a 0-30% allowable increase in RMSECV through sum of ranking differences leads to the selection of a model that increases the actual RMSECV up to 20% for laboratory standards but achieves an 85% reduction in the mean error in predicted concentrations for environmental mixtures. Partial least squares models developed with laboratory mixtures can provide useful predictions in complex environmental samples, but may benefit from protection against overfitting.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Model selection for partial least squares calibration and implications for analysis of atmospheric organic aerosol samples with mid‐infrared spectroscopy

Takahama

Dillner

2015

Journal of Chemometrics

View full text Add to dashboard Cite

show abstract

“…The processes known as sum of ranking differences (SRD) is used in this study with numerous model quality measures to automatically select calibration tuning parameter values for λ , η , and the number of eigenvectors. The SRD has been shown to be effective in selecting up to 2 calibration tuning parameters and has been well described. Briefly, a matrix of model quality measures is formed with rows designating respective model quality measures and a column for each tuning parameter (tuning parameter triplet in this study).…”

Section: Sample‐wise Calibrationmentioning

confidence: 99%

Sample‐wise spectral multivariate calibration desensitized to new artifacts relative to the calibration data using a residual penalty

Kalivas

Brownfield

Karki

2017

Journal of Chemometrics

Self Cite

View full text Add to dashboard Cite

Calibration maintenance is an important aspect of multivariate calibration. With spectral measurements, the goal of calibration maintenance involves sustaining the predictability of a primary calibration model in new secondary conditions. Among the many methodologies, penalty‐based Tikhonov regularization variants have been successful by sample augmenting primary calibration data with a matrix of just a few secondary samples as well as operating with an additional sparse penalty to include wavelength selection. Studied in this paper is a new sample‐wise (local) Tikhonov regularization–based penalty calibration approach. Penalized is a diagonal matrix with the residual vector (relative to the primary calibration space) of the new secondary sample. Thus, the same full calibration set is used for each new sample. Changing for each secondary sample is the corresponding sample‐wise residual vector on the penalized diagonal matrix. The intent of the presented approach is to form sample‐wise regression vectors desensitized to characteristics of the new sample not present in the primary calibration set. The more distinct the secondary conditions are relative to the primary conditions, the more unsuccessful this local model updating becomes. Proposed is a sample‐wise outlier mechanism to discern when the residual penalty can or cannot be used to form a useful updated model. The residual penalty modeling and outlier detection processes require tuning parameter optimizations. A fusion approach is used to automatically select tuning parameter values. Simulated and near‐infrared data are evaluated, demonstrating the applicability of the method.

show abstract

Section: Introductionmentioning

confidence: 99%

“…However, harmonious models are not necessarily parsimonious [1]. The scope of the methodology has recently been extended with the idea of sum of ranking differences (SRD) for partial least squares and ridge regression models [2].Principal-component analysis (PCA) has been applied by Geladi [3,4] and Todeschini et al [5] to find the best and worst regression and classification models, respectively. PCAs were completed on a matrix of regression vectors and dominant patterns (grouping, outliers) could be detected among the models.…”

mentioning

confidence: 99%

Consistency of QSAR models: Correct split of training and test sets, ranking of models and performance parameters

Rácz

Bajusz

Héberger

2015

SAR and QSAR in Environmental Research

View full text Add to dashboard Cite

Recent implementations of QSAR modeling software provide the user with numerous models and a wealth of information. In this work, we provide some guidance on how one should interpret the results of QSAR modeling, compare and assess the resulting models and select the best and most consistent ones. Two QSAR datasets are applied as case studies for the comparison of model performance parameters and model selection methods. We demonstrate the capabilities of sum of ranking differences (SRD) in model selection and ranking and identify the best performance indicators and models. While the exchange of the original training and (external) test sets does not affect the ranking of performance parameters, it provides improved models in certain cases (despite the lower number of molecules in the training set). Performance parameters for external validation are substantially separated from the other merits in SRD analyses, highlighting their value in data fusion. IntroductionModel comparison and selection of the best one is an evergreen among scientific investigations. The process is contradictory: bias-variance trade-off, local minima, searching for robust models, the principle of parsimony, etc.; all ideas consider various models inherently. One model is better from one point of view, the other should be better from another point of view. Even if one fixes the aim (and algorithm) according to various criteria: R 2 , Q 2 , Mallows Cp, Akaike Information criterion, Bayesian information criterion, etc., their application on the training, validation and test sets will necessarily provide different models for description of existing data and for prediction of future samples. The case is even more complicated with the fact that we deal with random effects: i.e. it is relatively easy to find conditions where one of the models is clearly superior compared to other models. Many authors select instinctively or deliberately such datasets, splits, etc. for which their own descriptor selection or model building algorithm performs better than the rival approaches.Kalivas et al. suggested selecting harmonious models taking into account the bias-variance trade-off: it is difficult and not unambiguous to find the 'best' model. A biased model provides less variance and vice versa. However, harmonious models are not necessarily parsimonious [1]. The scope of the methodology has recently been extended with the idea of sum of ranking differences (SRD) for partial least squares and ridge regression models [2].Principal-component analysis (PCA) has been applied by Geladi [3,4] and Todeschini et al. [5] to find the best and worst regression and classification models, respectively. PCAs were completed on a matrix of regression vectors and dominant patterns (grouping, outliers) could be detected among the models. The interpretation of PCA results is easy: principal component 1 marks the direction of the best and worst regression models. Principal component 2 reflects various behaviors of the regression models on various datasets. The models lyin...

show abstract

Sum of ranking differences (SRD) to ensemble multivariate calibration model merits for tuning parameter selection and comparing calibration methods

Cited by 42 publications

References 59 publications

Model selection for partial least squares calibration and implications for analysis of atmospheric organic aerosol samples with mid‐infrared spectroscopy

Model selection for partial least squares calibration and implications for analysis of atmospheric organic aerosol samples with mid‐infrared spectroscopy

Sample‐wise spectral multivariate calibration desensitized to new artifacts relative to the calibration data using a residual penalty

Consistency of QSAR models: Correct split of training and test sets, ranking of models and performance parameters

Contact Info

Product

Resources

About