From Convex to Nonconvex: A Loss Function Analysis for Binary Classification

Abstract

Problems of data classification can be studied in the framework of regularization theory as ill-posed problems. In this framework, loss functions play an important role in the application of regularization theory to classification. In this paper, we review some important convex loss functions, including hinge loss, square loss, modified square loss, exponential loss, logistic regression loss, as well as some non-convex loss functions, such as sigmoid loss, ϕ-loss, ramp loss, normalized sigmoid loss, and the loss function of 2 layer neural network. Based on the analysis of these loss functions, we propose a new differentiable nonconvex loss function, called smoothed 0-1 loss function, which is a natural approximation of the 0-1 loss function. To compare the performance of different loss functions, we propose two binary classification algorithms for binary classification, one for convex loss functions, the other for non-convex loss functions. A set of experiments are launched on several binary data sets from the UCI repository. The results show that the proposed smoothed 0-1 loss function is robust, especially for those noisy data sets with many outliers.

A Robust Loss Function for Multiclass Classification

“…It is possible to first design a nice Fisher consistent loss function and then construct the corresponding classifier [14]. A Fisher consistent smoothed 0-1 loss function for binary classification, constructed the corresponding classification algorithms and the experimental results show some good properties of the new algorithm [13]. Following the same line, we will extend this work to multiclass classification problems.…”

“…It has been demonstrated that Quasi Secant Method (QSM) [18] outperforms some traditional local optimization methods for binary classification problem [13]. In this algorithm, QSMis adopted as the optimization solver.…”

“…∀ = ( 1 , 2 , 3 ), we can have a set of hyper planes. However, the hyper planes that minimize Equation (13) have to pass the three red points. Therefore, the solution of this problem is also unique, and Equation (14) is satisfied.…”

“…In order to deal with data sets that contaminated by outliers, many robust loss functions including Hinge Loss Function [1], Exponential Loss Function [2], [3], Log Loss Function [3], [4], Mada boost Loss Function [5], Sigmoid Loss Function [6], Ф Loss Function [7], Ramp Loss Function [8], [9](also known as Robust Truncated Hinge Loss Function [10]), Normalized Sigmoid Lost Function [11], Logistic Difference Loss Function [12], have been proposed and evaluated. Based on the analysis of these loss functions, Smoothed 0-1 Loss Function has been proposed, applied and evaluated in binary data classification [13]. However, in many real applications we are more interested in solving multiclass problems where there are more than two classes.…”

“…

Based on the analysis of these loss functions, Smoothed 0-1 Loss Function has been proposed, applied and evaluated in binary data classification [13]. However, in many real applications we are more interested in solving multiclass problems where there are more than two classes.

…”