A Robust Loss Function for Multiclass Classification

Abstract

Based on the analysis of these loss functions, Smoothed 0-1 Loss Function has been proposed, applied and evaluated in binary data classification [13]. However, in many real applications we are more interested in solving multiclass problems where there are more than two classes. This paper examines the application of Smoothed 0-1 Loss Function to multiclass classification problems. Fisher consistency of the smoothed 0-1 loss function is investigated. Based on the smoothed 0-1 loss function, a multiclass classification algorithm is proposed. We hypothesize that this algorithm will perform well on noisy data sets, in particular for those noisy data sets with many outliers. To prove this, we compare this algorithm with hinge loss based multiclass classification algorithm through a systematic evaluation on several well-known multiclass data sets corrupted with two kinds of noise: class noise and attribute noise. Our experiments indicate that the smoothed 0-1 loss function based multiclass classification algorithm is robust for label noise and attribute noise.The structure of this paper is as follows. We first review some important loss functions for multiclass classification problem in Section II. Then Fisher consistent loss functions for multiclass classification is investigated in Section III. In Manuscript received September 17, 2013; revised November 1, 2013. Lei Zhao is with the Office of Research, University of the Sunshine Coast, Maroochydore QLD 4558 Australia (e-mail: lzhao@usc.edu.au).Section IV, a single machine approach for multiclass data classification is proposed. Numerical experiments are conducted in Section V, followed by conclusions in Section VI. II. MULTICLASS CLASSIFICATIONGiven a multiclass data set: Given a single data point that belongs to class i, for each Lei Zhao A Robust Loss Function for Multiclass ClassificationInternational Journal of Machine Learning and Computing, Vol. 3, No. 6, December 2013 462 DOI: 10.7763/IJMLC.2013.V3.361 pair ≠ we pay a penalty which is measured by loss function (•). This penalty plays an important role in the optimizationmodule, which will be further discussed in Section IV. III. FISHER CONSISTENT LOSS FUNCTIONS FOR MULTICLASS CLASSIFICATIONFisher consistency plays a fundamental role in the construction of successful binary margin-based classifiers. It is possible to first design a nice Fisher consistent loss function and then construct the corresponding classifier [14]. A Fisher consistent smoothed 0-1 loss function for binary classification, constructed the corresponding classification algorithms and the experimental results show some good properties of the new algorithm [13]. Following the same line, we will extend this work to multiclass classification problems.There has been a considerable amount of work in the literature to extend the binary classifiers to the multiclass case. In particular, a wide class of smooth convex loss functions that are Fisher consistency for multiclass classification has been investigated by Zou et al. [15].A loss function ...