Do unbalanced data have a negative effect on LDA?

Abstract

For two-class discrimination, Ref.~\cite{Xie:2007} claimed that, when covariance matrices of the two classes were unequal, a (class) unbalanced dataset had a negative effect on the performance of linear discriminant analysis (LDA). Through re-balancing $10$ real-world datasets, Ref.~\cite{Xie:2007} provided empirical evidence to support the claim using AUC (Area Under the receiver operating characteristic Curve) as the performance metric. We suggest that such a claim is vague if not misleading, there is no solid theoretical analysis presented in~\cite{Xie:2007}, and AUC can lead to a quite different conclusion from that led to by misclassification error rate (ER) on the discrimination performance of LDA for unbalanced datasets. Our empirical and simulation studies suggest that, for LDA, the increase of the median of AUC (and thus the improvement of performance of LDA) from re-balancing is relatively small, while, in contrast, the increase of the median of ER (and thus the decline in performance of LDA) from re-balancing is relatively large. Therefore, from our study, there is no reliable empirical evidence to support the claim that a (class) unbalanced data set has a negative effect on the performance of LDA. In addition, re-balancing affects the performance of LDA for datasets with either equal or unequal covariance matrices, indicating that having unequal covariance matrices is not a key reason for the difference in performance between original and re-balanced data.

What Works Better? A Study of Classifying Requirements

“…However, a study by Xue and Titterington [21] revealed that there is no reliable empirical evidence to support the claim that an unbalanced data set negatively impacts the performance of the LDA/BTM approaches. Further, a recent study by López et al [22] shows that the unbalanced ratio by itself does not have the most significant effect on the classifiers performance, but there are other issues such as (a) the presence of small disjuncts, (b) the lack of density, (c) the class overlapping, (d) the noisy data, (e) the management of borderline examples, and (f) the dataset shift that must be taken into account.…”

Why Does Rebalancing Class-Unbalanced Data Improve AUC for Linear Discriminant Analysis?

“…1 shows a motivating example, using a scatter plot and a panel of nine boxplots of AUC to illustrate visually the fact that rebalancing the training data can often improve the performance of LDA in terms of AUC [5], [6]. This example is extracted from an experiment on simulated data arising from two four-dimensional, Gaussian-distributed classes C 0 and C 1 .…”

“…This example is extracted from an experiment on simulated data arising from two four-dimensional, Gaussian-distributed classes C 0 and C 1 . With a slightly different setting, the experiment explores more rebalancing scenarios than in [6]. It includes the following four steps.…”

“…Using the rebalanced training data can often increase (i.e. improve) the area (AUC) under the receiver operating characteristic (ROC) curve for the original, unbalanced test data [4], [5], [6], [7]. The AUC is a widely-used quantitative measure of classification performance, but the empirical property that rebalancing increases AUC lacks theoretical justification.…”

Determination of an optimal training strategy for a BCI classification task with LDA

“…Even though the LDA has been extensively studied [7][8][9], the effect of unbalanced training datasets using electroencephalographic (EEG) data and the number of patterns necessary to reach a performance plateau have not been tested. That is, the point at which no significant performance gain will exist when adding more training patterns has not been determined.…”