An Improved Conjugate Gradient Scheme to the Solution of Least Squares SVM

Abstract

The least square support vector machines (LS-SVM) formulation corresponds to the solution of a linear system of equations. Several approaches to its numerical solutions have been proposed in the literature. In this letter, we propose an improved method to the numerical solution of LS-SVM and show that the problem can be solved using one reduced system of linear equations. Compared with the existing algorithm for LS-SVM, the approach used in this letter is about twice as efficient. Numerical results using the proposed method are provided for comparisons with other existing algorithms.Index Terms-Conjugate gradient (CG), least square support vector machines (LS-SVM), sequential minimal optimization (SMO).

SMO-Based Pruning Methods for Sparse Least Squares Support Vector Machines

“…LS-SVM greatly simplifies the problem by characterizing the solution as a linear system. For large scale problems, linear system (5) can be efficiently solved using CG method [5], [6]. A drawback of LS-SVM is that the sparseness is lost in its solution: from the KKT conditions (4), every data point contributes to the model and the relative importance of a data point is reflected by its support value.…”

“…The radial basis function is used as the kernel function. For each dataset, we use the parameter suggested in [6] and [7]. We will compare the proposed pruning methods to CG methods in terms of both computational cost and classification accuracy.…”

“…Instead of solving a quadratic programming problem as in SVMs, LS-SVMs find the solutions of a set of linear equations [4]. Several numerical algorithms have been proposed for training LS-SVM: Suykens et al propose an iterative algorithm based on conjugate gradient (CG) algorithms [5]; Chu et al improve the efficiency of the CG algorithm using one reduced system of linear equations [6]; and Keerthi and Shevade extend the sequential minimal optimization (SMO) algorithms to solve the linear equations in LS-SVMs [7]. These numerical algorithms are computationally attractive.…”

Permeability Measurement of Ferromagnetic Materials in Microwave Frequency Range Using Support Vector Machine Regression

“…Recently, Vapnic's SVM theory has been applied successfully for classification and regression problems [8][9][10]. SVM solves a constrained quadratic optimization problem, and it is based on statistical learning theory that gives the possibility to control the model's complexity and hence, to control its generalization ability.…”

Accelerating Levenberg-Marquardt Algorithm using Variable Damping Parameter

“…오차 함수의 2차 미분을 이용한 방법으로는 Conjugate Gradient 알고리즘 [7,8], Newton 방법과 유사 Newton 방법인 LM(Levenberg-Marquardt) 방법 [9,10], Quasi-Newton 방법 [10,11]등이 있다. 이런 방법 모두 2차 도함수인 Hessian 행렬을 직접 구하는 대신 근사적인 방법으로 2차 도함수를 구 하여 학습에 적용함으로써 빠른 학습 속도를 보인다.…”