Ridge regression optimization using a harmonious approach

Abstract

A critical component of ridge regression (RR) is determining the optimal ridge parameter value, , where ! 0. Improper selection of not only generates an under-or overfitted model but also leads to incorrect conclusions in inter-model comparison studies such as between RR, PLS, PCR and other modeling methods. Several methods for determining the optimal RR model are evaluated in this paper. For example, the commonly used ridge trace is identified as subjective and impractical. A direct calculation method from the literature yields over-or underfitted RR models with either too small or to large respectively. Methods for determining based on a harmonious approach are discussed. The harmonious approach optimizes by inspecting the bias/variance tradeoff. Of the methods investigated, plotting a variance indicator against a bias measure to yield an L-curve appears not only to simplify selection of but also to reduce the chance of obtaining an under-or overfitted RR model. It is shown with four data sets that the L-curve harmonious approach consistently provides good models. The effective rank of models is also discussed in conjunction with the harmony/parsimony tradeoff.

Sum of ranking differences (SRD) to ensemble multivariate calibration model merits for tuning parameter selection and comparing calibration methods

“…The minimization expression for the TR variant RR [24,[42][43][44] The L-curve for selecting tuning parameters [3,20,21,[24][25][26][27]29] can be formed by plotting mean RMSEC or RMSECV against a model variance or complexity measure. Models in the corner region of the L-curve represent acceptable compromises for the bias/variance tradeoff,…”

“…Two specific merits to be evaluated with SRD in this study are generalized CV (GCV) [46], AIC [47], BIC [48], trace (X T X) + [21], and others [12,18,19]. These merits were not used in this paper, but their usages with SRD are also feasible.…”

“…In this case, bias is the degree of prediction accuracy obtained from a model and variance is related to the extent of uncertainty in the prediction [20][21][22][23]. Methods such as RR and PLS are biased methods and hence a tradeoff in the degree of 4 under-and over-fitting is mandatory to form a model with an "acceptable" bias/variance balance [3,[21][22][23]. Models with acceptable bias/variance tradeoffs were recently shown to also balance the intrinsic model selectivity and sensitivity [23].…”

“…Sensitivity refers to the degree of change in signal relative to a change in the quantity of analyte, e.g., in analytical chemistry, a system is sensitive if a small change in analyte concentration generates a large change in signal [13,17]. It follows then that at least two model merits, each trending in opposite directions, should be simultaneously evaluated in order to characterize the balance between under-and over-fitting [20,21,[24][25][26][27][28].…”

Bayesian linear regression and variable selection for spectroscopic calibration

“…Furthermore the introduction of the regularization term is helpful to control the model complexity and to alleviate the "over-fitting" problem [26]. A number of techniques have been proposed to determine the value of λ, including cross-validation [5] and more recently harmonious approach [31].…”