A Comprehensive Survey of Loss Functions in Machine Learning

Wang, Qi; Ma, Yue; Zhao, Kun; Tian, Yingjie

doi:10.1007/s40745-020-00253-5

Cited by 291 publications

(176 citation statements)

References 47 publications

Supporting

Mentioning

131

Contrasting

Unclassified

Order By: Relevance

“…In the future, we plan to compare R 2 with other regression rates such as Huber metric H δ ( Huber, 1992 ), LogCosh loss ( Wang et al, 2020 ) and Quantile Q γ ( Yue & Rue, 2011 ). We will also study some variants of the coefficient of determination, such as the adjusted R -squared ( Miles, 2014 ) and the coefficient of partial determination ( Zhang, 2017 ).…”

Section: Discussionmentioning

confidence: 99%

The coefficient of determination R-squared is more informative than SMAPE, MAE, MAPE, MSE and RMSE in regression analysis evaluation

Chicco

Warrens

Jurman

2021

PeerJ Computer Science

1,759

574

View full text Add to dashboard Cite

Regression analysis makes up a large part of supervised machine learning, and consists of the prediction of a continuous independent target from a set of other predictor variables. The difference between binary classification and regression is in the target range: in binary classification, the target can have only two values (usually encoded as 0 and 1), while in regression the target can have multiple values. Even if regression analysis has been employed in a huge number of machine learning studies, no consensus has been reached on a single, unified, standard metric to assess the results of the regression itself. Many studies employ the mean square error (MSE) and its rooted variant (RMSE), or the mean absolute error (MAE) and its percentage variant (MAPE). Although useful, these rates share a common drawback: since their values can range between zero and +infinity, a single value of them does not say much about the performance of the regression with respect to the distribution of the ground truth elements. In this study, we focus on two rates that actually generate a high score only if the majority of the elements of a ground truth group has been correctly predicted: the coefficient of determination (also known as R-squared or R2) and the symmetric mean absolute percentage error (SMAPE). After showing their mathematical properties, we report a comparison between R2 and SMAPE in several use cases and in two real medical scenarios. Our results demonstrate that the coefficient of determination (R-squared) is more informative and truthful than SMAPE, and does not have the interpretability limitations of MSE, RMSE, MAE and MAPE. We therefore suggest the usage of R-squared as standard metric to evaluate regression analyses in any scientific domain.

show abstract

Section: Discussionmentioning

confidence: 99%

The coefficient of determination R-squared is more informative than SMAPE, MAE, MAPE, MSE and RMSE in regression analysis evaluation

Chicco

Warrens

Jurman

2021

PeerJ Computer Science

1,759

574

View full text Add to dashboard Cite

show abstract

“…Loss functions. A survey of loss functions used in machine learning is provided by [10]. In total, 31 loss functions are analyzed with respect to their purpose task and application scenario.…”

Section: Related Workmentioning

confidence: 99%

Generalization of Cross-Entropy Loss Function for Image Classification

Andreieva

Shvai

2021

MMJ

View full text Add to dashboard Cite

Classification task is one of the most common tasks in machine learning. This supervised learning problem consists in assigning each input to one of a finite number of discrete categories. Classification task appears naturally in numerous applications, such as medical image processing, speech recognition, maintenance systems, accident detection, autonomous driving etc.In the last decade methods of deep learning have proven to be extremely efficient in multiple machine learning problems, including classification. Whereas the neural network architecture might depend a lot on data type and restrictions posed by the nature of the problem (for example, real-time applications), the process of its training (i.e. finding model’s parameters) is almost always presented as loss function optimization problem.Cross-entropy is a loss function often used for multiclass classification problems, as it allows to achieve high accuracy results.Here we propose to use a generalized version of this loss based on Renyi divergence and entropy. We remark that in case of binary labels proposed generalization is reduced to cross-entropy, thus we work in the context of soft labels. Specifically, we consider a problem of image classification being solved by application of convolution neural networks with mixup regularizer. The latter expands the training set by taking convex combination of pairs of data samples and corresponding labels. Consequently, labels are no longer binary (corresponding to single class), but have a form of vector of probabilities. In such settings cross-entropy and proposed generalization with Renyi divergence and entropy are distinct, and their comparison makes sense.To measure effectiveness of the proposed loss function we consider image classification problem on benchmark CIFAR-10 dataset. This dataset consists of 60000 images belonging to 10 classes, where images are color and have the size of 32×32. Training set consists of 50000 images, and the test set contains 10000 images.For the convolution neural network, we follow [1] where the same classification task was studied with respect to different loss functions and consider the same neural network architecture in order to obtain comparable results.Experiments demonstrate superiority of the proposed method over cross-entropy for loss function parameter value α < 1. For parameter value α > 1 proposed method shows worse results than cross-entropy loss function. Finally, parameter value α = 1 corresponds to cross-entropy.

show abstract

“…DNNs are trained to minimize a loss function such as mean squared error (MSE) on the training data [67]. The loss minimization is typically performed using one of the popular gradient descent algorithms such as stochastic gradient descent (SGD) [68] or adaptive moment estimation (Adam) [69].…”

Section: Gradient Descent Optimizationmentioning

confidence: 99%

Automatic Detection of Impervious Surfaces from Remotely Sensed Data Using Deep Learning

Parekh¹,

Poortinga²,

Bhandari

et al. 2021

Remote Sensing

View full text Add to dashboard Cite

The large scale quantification of impervious surfaces provides valuable information for urban planning and socioeconomic development. Remote sensing and GIS techniques provide spatial and temporal information of land surfaces and are widely used for modeling impervious surfaces. Traditionally, these surfaces are predicted by computing statistical indices derived from different bands available in remotely sensed data, such as the Landsat and Sentinel series. More recently, researchers have explored classification and regression techniques to model impervious surfaces. However, these modeling efforts are limited due to lack of labeled data for training and evaluation. This in turn requires significant effort for manual labeling of data and visual interpretation of results. In this paper, we train deep learning neural networks using TensorFlow to predict impervious surfaces from Landsat 8 images. We used OpenStreetMap (OSM), a crowd-sourced map of the world with manually interpreted impervious surfaces such as roads and buildings, to programmatically generate large amounts of training and evaluation data, thus overcoming the need for manual labeling. We conducted extensive experimentation to compare the performance of different deep learning neural network architectures, optimization methods, and the set of features used to train the networks. The four model configurations labeled U-Net_SGD_Bands, U-Net_Adam_Bands, U-Net_Adam_Bands+SI, and VGG-19_Adam_Bands+SI resulted in a root mean squared error (RMSE) of 0.1582, 0.1358, 0.1375, and 0.1582 and an accuracy of 90.87%, 92.28%, 92.46%, and 90.11%, respectively, on the test set. The U-Net_Adam_Bands+SI Model, similar to the others mentioned above, is a deep learning neural network that combines Landsat 8 bands with statistical indices. This model performs the best among all four on statistical accuracy and produces qualitatively sharper and brighter predictions of impervious surfaces as compared to the other models.

show abstract

A Comprehensive Survey of Loss Functions in Machine Learning

Cited by 291 publications

References 47 publications

The coefficient of determination R-squared is more informative than SMAPE, MAE, MAPE, MSE and RMSE in regression analysis evaluation

The coefficient of determination R-squared is more informative than SMAPE, MAE, MAPE, MSE and RMSE in regression analysis evaluation

Generalization of Cross-Entropy Loss Function for Image Classification

Automatic Detection of Impervious Surfaces from Remotely Sensed Data Using Deep Learning

Contact Info

Product

Resources

About