Optimized fixed-size kernel models for large data sets

Brabanter, Kris De; Brabanter, Jos De; Suykens, Johan A. K.; Moor, Bart De

doi:10.1016/j.csda.2010.01.024

Cited by 98 publications

(60 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since each output is considered independently, these covariance matrices have dimensions N xN , N x1 and 1x1 respectively. Following the technique presented in [19] to optimize the regularization parameter for FS-LS-SVMs, the covariance matrix A on the training and validation set combined can be calculated as follows:…”

Section: Covariance Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Optimized Parameter Search for Large Datasets of the Regularization Parameter and Feature Selection for Ridge Regression

Buteneers

Caluwaerts

Dambre

et al. 2013

Neural Process Lett

View full text Add to dashboard Cite

In this paper we propose mathematical optimizations to select the optimal regularization parameter for ridge regression using cross-validation. The resulting algorithm is suited for large datasets and the computational cost does not depend on the size of the training set. We extend this algorithm to forward or backward feature selection in which the optimal regularization parameter is selected for each possible feature set. These feature selection algorithms yield solutions with a sparse weight matrix using a quadratic cost on the norm of the weights.A naive approach to optimizing the ridge regression parameter has a computational complexity of the order O(RKN 2 M ) with R the number of applied regularization parameters, K the number of folds in the validation set, N the number of input features and M the number of data samples in the training set. Our implementation has a computational complexity of the order O(KN 3 ). This computational cost is smaller than that of regression without regularization O(N 2 M ) for large datasets and is independent of the number of applied regularization parameters and the size of the training set. Combined with a feature selection algorithm the algorithm is of complexity O(RKN N To show the performance and reduction in computational cost, we apply this technique to train recurrent neural networks using the reservoir computing approach, windowed ridge regression, least-squares support vector machines (LSSVMs) in primal space using the fixed-size LS-SVM approximation and extreme learning machines.

show abstract

Section: Covariance Methodsmentioning

confidence: 99%

“…This formula can be used on the entire training set U to generate the input data X. For more details on this process we refer to [12] and [19]. We apply the FS-LS-SVM approach to the UCI Adult dataset as described in [15] and the heat exchanger task as described in [16].…”

Section: Fixed-size Least-squares Support Vector Machines Applied To mentioning

confidence: 99%

Optimized Parameter Search for Large Datasets of the Regularization Parameter and Feature Selection for Ridge Regression

Buteneers

Caluwaerts

Dambre

et al. 2013

Neural Process Lett

View full text Add to dashboard Cite

show abstract

“…Furthermore, the algorithm can be tailored to a given application by using the most appropriate kernel function. Beyond that, by using sparse formulations and a fixed-size (Suykens et al 2002, De Brabanter et al 2010) approach, it is possible to readily handle big data. Finally, by means of adequate adaptations of the core algorithm, hierarchical clustering and a soft clustering approach have been proposed.…”

Section: Introductionmentioning

confidence: 99%

Kernel Spectral Clustering and Applications

Langone

Mall

Alzate

et al. 2016

Unsupervised Learning Algorithms

View full text Add to dashboard Cite

“…As a consequence, it needs a high computational load in training and has bad robustness. To overcome these drawbacks, many efforts have been made by Suykens [12], De Kruif and De Vries [13], Hoegaerts [14], Zeng and Chen [15] and Jiao [16] et al For large data sets, Brabanter et al [17] and Karsmakers et al [6] recently developed the fixed-size kernel (SVR) modeling method. More recently, a novel and much sparser LSSVR method named improved recursive reduced LSSVR (IRR-LSSVR) is proposed by Zhao and Sun et al [18] after combining a reduced technique [19] with the iterative strategy [16].…”

Section: Introductionmentioning

confidence: 99%

A novel online adaptive kernel method with kernel centers determined by a support vector regression approach

et al. 2014

View full text Add to dashboard Cite

The optimality of the kernel number and kernel centers plays a significant role in determining the approximation power of nearly all kernel methods. However, the process of choosing optimal kernels is always formulated as a global optimization task, which is hard to accomplish. Recently, an algorithm, namely improved recursive reduced least squares support vector regression (IRR-LSSVR), was proposed for establishing a global nonparametric offline model, which demonstrates significant advantage in choosing representing and fewer support vectors compared with others. Inspired by the IRR-LSSVR, a new adaptive parametric kernel method called WV-LSSVR is proposed in this paper using the same type of kernels and the same centers as those used in the IRR-LSSVR. Furthermore, inspired by the multikernel semiparametric support vector regression, the effect of the kernel extension is investigated in a recursive regression framework, and a recursive kernel method called GPK-LSSVR is proposed using a compound type of kernels which are recommended for Gaussian process regression. Numerical experiments on benchmark data sets confirm the validity and effectiveness of the presented algorithms. The WV-LSSVR algorithm shows higher approximation accuracy than the recursive parametric kernel method using the centers calculated by the k-means clustering approach. The extended recursive kernel method (i.e. GPK-LSSVR) has not shown advantage in terms of global approximation accuracy when validating the test data set without real-time updation, but it can increase modeling accuracy if the real-time identification is involved.

show abstract

Optimized fixed-size kernel models for large data sets

Cited by 98 publications

References 48 publications

Optimized Parameter Search for Large Datasets of the Regularization Parameter and Feature Selection for Ridge Regression

Optimized Parameter Search for Large Datasets of the Regularization Parameter and Feature Selection for Ridge Regression

Kernel Spectral Clustering and Applications

A novel online adaptive kernel method with kernel centers determined by a support vector regression approach

Contact Info

Product

Resources

About