Data scarcity, robustness and extreme multi-label classification

Abstract

The goal in extreme multi-label classification is to learn a classifier which can assign a small subset of relevant labels to an instance from an extremely large set of target labels. Datasets in extreme classification exhibit a long tail of labels which have small number of positive training instances. In this work, we pose the learning task in extreme classification with large number of tail-labels as learning in the presence of adversarial perturbations. This view motivates a robust optimization framework and equivalence to a corresponding regularized objective.Under the proposed robustness framework, we demonstrate efficacy of Hamming loss function for taillabel detection in extreme classification. The equivalent regularized objective, in combination with proximal gradient based optimization, performs better than state-ofthe-art methods on propensity scored versions of preci-sion@k and nDCG@k(upto 20% relative improvement over PFastreXML -a leading tree-based approach and 60% relative improvement over SLEEC -a leading labelembedding approach). Furthermore, we also highlight the sub-optimality of a sparse solver in a widely used package for large-scale linear classification, which is interesting in its own right. We also investigate the spectral properties of label graphs for providing novel insights towards understanding the conditions governing the performance of Hamming loss based one-vs-rest scheme vis-à-vis label embedding methods.

Does learning require memorization? a short tale about a long tail

“…Additional evidence that classes can be viewed as long-tailed mixtures of subpopulations comes from extreme multiclass problems. Specifically, these problems often have more than 10, 000 fine-grained labels and the number of examples per class is longtailed [3,4,21,30,49,51]. Observe that fine-grained labels in such problems correspond to subcategories of coarser classes (for example, different species of birds all correspond to the "bird" label in a coarse classification problem).…”

Accelerating Extreme Classification via Adaptive Feature Agglomeration

“…This algorithm can be derived as a special case (for k = 2) of the refinement step in the constrained clustering routine proposed in [Banerjee and Ghosh, 2006]. It is notable that [Prabhu et al, 2018] derive essentially the same algorithm for splitting labels into balanced cluster, but they derive their approach starting from a different graph flow-based approach to constrained clustering.…”

Bonsai: diverse and shallow trees for extreme multi-label classification

“…One-vs-rest Sometimes also referred to as binary relavance (Zhang et al 2018), these methods learn a classifier per label which distinguishes it from rest of the labels. In terms of prediction accuracy and label diversity, these methods have been shown to be among the best performing ones for XMC (Babbar and Schölkopf 2017;Yen et al 2017;Babbar and Schölkopf 2019). However, due to their reliance on a distributed training framework, it remains challenging to employ them in resource constrained environments.…”