Generalization of Parameter Selection of SVM and LS-SVM for Regression

Abstract

A Support Vector Machine (SVM) for regression is a popular machine learning model that aims to solve nonlinear function approximation problems wherein explicit model equations are difficult to formulate. The performance of an SVM depends largely on the selection of its parameters. Choosing between an SVM that solves an optimization problem with inequality constrains and one that solves the least square of errors (LS-SVM) adds to the complexity. Various methods have been proposed for tuning parameters, but no article puts the SVM and LS-SVM side by side to discuss the issue using a large dataset from the real world, which could be problematic for existing parameter tuning methods. We investigated both the SVM and LS-SVM with an artificial dataset and a dataset of more than 200,000 points used for the reconstruction of the global surface ocean CO2 concentration. The results reveal that: (1) the two models are most sensitive to the parameter of the kernel function, which lies in a narrow range for scaled input data; (2) the optimal values of other parameters do not change much for different datasets; and (3) the LS-SVM performs better than the SVM in general. The LS-SVM is recommended, as it has less parameters to be tuned and yields a smaller bias. Nevertheless, the SVM has advantages of consuming less computer resources and taking less time to train. The results suggest initial parameter guesses for using the models.

Modelling of Mechanical Properties of Fresh and Stored Fruit of Large Cranberry Using Multiple Linear Regression and Machine Learning

“…The correct value of γ parameter in RBF kernel can avoid under-fitting and over-fitting phenomena in prediction [39]. The ε influences the bias significantly, and its optimal value depends on the type of noise present in the dataset [40,41]. The C parameter affects the number of support vectors, and the proper value of C can minimize the over-fitting problem [42].…”

Detection of saffron adulteration with Crocus sativus style using NIR-hyperspectral imaging and chemometrics

“…A high value of C results in cases outside the tolerance margin being heavily penalized, thereby reducing training bias but increasing prediction variance and potentially leading to overfitting. In contrast, low values of C may increase training bias (Zeng et al, 2019).…”

“…To prevent overfitting, SVM adjusts the classification decision function based on the principle of structural risk minimization, rather than simply minimizing the misclassification error on the training set (Chen et al, 2007). The optimal parameters for the SVM regression model were estimated as per the method outlined by Zeng et al (2019).…”

Atrial Fibrillation Detection Through ML Approach: A Comparitive Study

“…During the training phase, SVM aims to minimize the classification error while maximizing the margin. It does this by formulating a cost function that penalizes misclassification and accounts for the margin width (Zeng et al, 2019). The optimization process involves finding the Lagrange multipliers associated with the support vectors to determine the hyperplane coefficients.…”