Large Margin Distribution Learning with Cost Interval and Unlabeled Data

Abstract

In many real-world applications, different types of misclassification usually suffer from different costs, but the accurate cost is often hard to be determined and usually one can only get an interval-estimation like that one type of mistake is about five to ten times more serious than the other type. On the other hand, there are usually abundant unlabeled data available, leading to great research effort about semi-supervised learning. It is noticeable that cost interval and unlabeled data usually appear simultaneously in practice tasks; however, there is rare study tackling them together. In this paper, we propose the cisLDM approach which is able to handle cost interval and exploit unlabeled data in a principled way. Rather than maximizing the minimum margin like traditional large margin classifiers, cisLDM tries to optimize the margin distribution on both labeled and unlabeled data when minimizing the worst-case totalcost and the mean total-cost simultaneously according to the cost interval. Experiments on a broad range of datasets and cost settings exhibit the impressive performance of cisLDM. In particular, cisLDM is able to reduce 47% more total-cost than standard SVM and 27% more total-cost than cost-sensitive semi-supervised SVM which assumes the true cost value is known in advance.

Multi-label optimal margin distribution machine

“…Thereafter, varieties of methods based on margin distribution have been proposed. Zhou and Zhou (2016) and Zhou (2017, 2018) generalized ODM to class imbalance learning, multi-class learning and unsupervised learning respectively. In weakly supervised learning (Zhou 2018), Zhang and Zhou (2018a) proposed the semi-supervised ODM(ssODM), which achieved significant improvement in performance compared to SVM-based methods.…”

“…In addition to classic supervised learning tasks, there is also a series of work in various tasks verifying the better generalization performance of optimizing margin distribution. For example, Zhou and Zhou (2016) extended the idea to exploit unlabeled data and handle unequal misclassification cost; Zhang and Zhou (2018) proposed the margin distribution machine for clustering. Tan et al (2019) accelerated the kernel methods and applied the idea to large-scale datasets.…”

Entropy‐based hybrid sampling ensemble learning for imbalanced data

“…Instead of changing the distribution of the original data, this approach improves the algorithm to fit the imbalanced data. There are many works based on the algorithm level, such as the threshold method, 9 one‐class learning, 10,11 and cost‐sensitive learning 12,13 . The third is hybrid approach, which combines the sampling methods with the algorithm‐level approaches to form an ensemble system or a neural network 4 .…”

Optimal Margin Distribution Learning in Dynamic Environments

“…Specifically, Zhang and Zhou (2014; proposed the optimal margin distribution machines (ODMs) for binary and multi-class classification problems respectively, and achieved superior generalization performance to the traditional large margin based methods. After that, the optimal margin distribution learning has turned into a promising research direction and attracted a lot of attentions, just to name a few, Zhou and Zhou (2016) exploit it to handle unequal misclassification cost; Cheng, Zhang, and Wen (2016) use it for imbalanced data; Ou et al (2017) applies it to feature elimination; Zhang and Zhou (2018a;2018b) extends it to clustering and semi-supervised learning settings.…”