relf: robust regression extended with ensemble loss function

Abstract

Ensemble techniques are powerful approaches that combine several weak learners to build a stronger one. As a meta-learning framework, ensemble techniques can easily be applied to many machine learning methods. Inspired by ensemble techniques, in this paper we propose an ensemble loss functions applied to a simple regressor. We then propose a half-quadratic learning algorithm in order to find the parameter of the regressor and the optimal weights associated with each loss function. Moreover, we show that our proposed loss function is robust in noisy environments. For a particular class of loss functions, we show that our proposed ensemble loss function is Bayes consistent and robust. Experimental evaluations on several data sets demonstrate that the our proposed ensemble loss function significantly improves the performance of a simple regressor in comparison with state-of-the-art methods.Loss functions are fundamental components of machine learning systems and are used to train the parameters of the learner model. Since standard training methods aim to determine the parameters that minimize the average value of the loss given an annotated training set, loss functions are crucial for successful trainings [49,55]. Bayesian estimators are obtained by minimizing the expected loss function. Different loss functions lead to different Optimum Bayes with possibly different characteristics. Thus, in each environment the choice of the underlying loss function is important, as it will impact the performance [44,48].Lettingθ denote the estimated parameter of a correct parameter θ, the loss function L(θ, θ) is a positive function which assigns a loss value to each estimation, indicating how inaccurate the estimation is [42]. Loss functions assign a value to each sample, indicating how much that sample contributes to solving the optimization problem. Each loss function comes with its own advantages and disadvantages. In order to put our results in context, we start by reviewing three popular loss functions (0-1, Ramp and Sigmoid) and we will give an overview of their advantages and disadvantages.Loss functions assign a value to each sample representing how much that sample contributes to solving the optimization problem. If an outlier is given a very large value by the loss function, it might dramatically affect the decision function [18]. The 0-1 loss function is known as a robust loss because it assigns value 1 to all misclassified samples -including outliers -and thus an outlier does not influence the decision function, leading to a robust learner. On the other hand, the 0-1 loss penalizes all misclassified samples equally with value 1, and since it does not enhance the margin, it cannot be an appropriate choice for applications with margin importance [49].The Ramp loss function, as another type of loss functions, is defined similarly to the 0-1 loss function with the only difference that ramp loss functions also penalize some correct samples, those with small margins. This minor difference makes the Ramp loss function appro...

Automatic Facial Aesthetic Prediction Based on Deep Learning with Loss Ensembles

“…A crucial component of a learning system is a loss function that quantifies the accuracy of the predicted value relative to the ground-truth value. In machine learning, there are two loss functions: those based on the margin employed in the classification process and those based on distance in regression problems [36]. The loss function is denoted as follows: loss function = L(𝑥 ⃗, yt; yp)…”

“…Furthermore, the pretrained networks are regarded as important in machine-learning-model evaluation. To show the effectiveness of the proposed loss function in estimating the beauty score of the facial image, three pretrained CNNs (AlexNet [35], VGG16-Net [36], and FIAC-Net [37]) were fine-tuned to obtain benefit from the gained knowledge of each network, utilizing the transfer-learning aspects.…”