Efficient approximate leave-one-out cross-validation for kernel logistic regression

Cawley, Gavin C.; Talbot, Nicola L. C.

doi:10.1007/s10994-008-5055-9

Cited by 100 publications

(69 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Cawley and Talbot (2008) propose a method that yields all leave-one-out prediction errors as a by-product of estimating (2) only once, that is, on the full sample. We derive a similar result, extended to allow for the additional linear terms in (3), in Appendix A.3.…”

Section: Selection Of Tuning Parametersmentioning

confidence: 99%

Nonlinear Forecasting with Many Predictors Using Kernel Ridge Regression

et al. 2011

View full text Add to dashboard Cite

This paper puts forward kernel ridge regression as an approach for forecasting with many predictors that are related nonlinearly to the target variable. In kernel ridge regression, the observed predictor variables are mapped nonlinearly into a high-dimensional space, where estimation of the predictive regression model is based on a shrinkage estimator to avoid overfitting. We extend the kernel ridge regression methodology to enable its use for economic time-series forecasting, by including lags of the dependent variable or other individual variables as predictors, as typically desired in macroeconomic and financial applications. Monte Carlo simulations as well as an empirical application to various key measures of real economic activity confirm that kernel ridge regression can produce more accurate forecasts than traditional linear methods for dealing with many predictors based on principal component regression.

show abstract

Section: Selection Of Tuning Parametersmentioning

confidence: 99%

Nonlinear Forecasting with Many Predictors Using Kernel Ridge Regression

et al. 2011

View full text Add to dashboard Cite

show abstract

“…In this report, grid pattern searching technique [40] was used to choose the parameters of RBF. The residual sum of squares was calculated using leave-one-out crossvalidation (LOOCV) [41] during the model building.…”

Section: Methodsmentioning

confidence: 99%

Rapid Identification of Pork Adulterated in the Beef and Mutton by Infrared Spectroscopy

Yang

Liu

et al. 2018

Journal of Spectroscopy

View full text Add to dashboard Cite

Consumers concern about food adulteration. Pork meat is the principal adulterated species of beef and mutton. The conventional detection methods have their own limitations; therefore, we sought to develop an efficient and economical identification method using an infrared spectroscopy technique for meat. The Mahalanobis distance method was used to remove outliers in spectrum data. Interferences were eliminated using multiple scatter correction, standard normal variate, Savitzky-Golay smoothing, and normalization. The partial least square discriminant analysis (PLS-DA) and support vector machine (SVM) were used to establish identification models. In the Mahalanobis distance method, the coefficient of test sets was increased from 0.93 to 0.99; the RMSEC and RMSECV were decreased from 0.17 to 0.09 and 0.21 to 0.11 accordingly. The coefficient of determination in-between the calibration and testing sets in PLS-DA reached 0.99 and 0.99, RMSEC was 0.06, and both the RMSECV and RMSEP were 0.08. In contrast, in SVM, methods were 0.97 and 0.96. The RMSEC, RMSECV, and RMSEP were 0.15, 0.17, and 0.24, respectively. In summary, using a combination of infrared spectroscopy technology with PLS-DA was a better identification method than the SVM method that can be used as an effective method to identify pork, beef, and mutton meat samples.

show abstract

“…Mika et al, 1999;Rifkin, 2002;Cawley and Talbot, 2003;Van Gestel et al, 2004). Kernel Logistic Regression, with a cross-entropy loss that is more obviously suited to statistical pattern recognition, does not out-perform kernel Fisher discriminant analysis (Cawley and Talbot, 2008). Furthermore, we have used these methods in winning entries in a number of open machine learning challenges and highly competitive baseline methods in others (Cawley, 2006;Guyon et al, 2008;Cawley, 2009Cawley, , 2011.…”

Section: Kernel Learning At the First Level Of Inferencementioning

confidence: 99%

Kernel learning at the first level of inference

Cawley

Talbot

2014

Neural Networks

Self Cite

View full text Add to dashboard Cite

Kernel learning methods, whether Bayesian or frequentist, typically involve multiple levels of inference, with the coefficients of the kernel expansion being determined at the first level and the kernel and regularisation parameters carefully tuned at the second level, a process known as model selection. Model selection for kernel machines is commonly performed via optimisation of a suitable model selection criterion, often based on cross-validation or theoretical performance bounds. However, if there are a large number of kernel parameters, as for instance in the case of automatic relevance determination (ARD), there is a substantial risk of over-fitting the model selection criterion, resulting in poor generalisation performance. In this paper we investigate the possibility of learning the kernel, for the Least-Squares Support Vector Machine (LS-SVM) classifier, at the first level of inference, i.e. parameter optimisation. The kernel parameters and the coefficients of the kernel expansion are jointly optimised at the first level of inference, minimising a training criterion with an additional regularisation term acting on the kernel parameters. The key advantage of this approach is that the values of only two regularisation parameters need be determined in model selection, substantially alleviating the problem of over-fitting the model selection criterion. The benefits of this approach are demonstrated using a suite of synthetic and real-world binary classification benchmark problems, where kernel learning at the first level of inference is shown to be statistically superior to the conventional approach, improves on our previous work (Cawley and Talbot, 2007) and is competitive with Multiple Kernel Learning approaches, but with reduced computational expense.

show abstract

Efficient approximate leave-one-out cross-validation for kernel logistic regression

Cited by 100 publications

References 40 publications

Nonlinear Forecasting with Many Predictors Using Kernel Ridge Regression

Nonlinear Forecasting with Many Predictors Using Kernel Ridge Regression

Rapid Identification of Pork Adulterated in the Beef and Mutton by Infrared Spectroscopy

Kernel learning at the first level of inference

Contact Info

Product

Resources

About