Bonsai: diverse and shallow trees for extreme multi-label classification

Abstract

Extreme multi-label classification (XMC) refers to supervised multi-label learning involving hundreds of thousands or even millions of labels. In this paper, we develop a suite of algorithms, called Bonsai, which generalizes the notion of label representation in XMC, and partitions the labels in the representation space to learn shallow trees. We show three concrete realizations of this label representation space including: (i) the input space which is spanned by the input features, (ii) the output space spanned by label vectors based on their co-occurrence with other labels, and (iii) the joint space by combining the input and output representations. Furthermore, the constraint-free multi-way partitions learnt iteratively in these spaces lead to shallow trees. By combining the effect of shallow trees and generalized label representation, Bonsai achieves the best of both worlds-fast training which is comparable to state-of-the-art tree-based methods in XMC, and much better prediction accuracy, particularly on tail-labels. On a benchmark Amazon-3M dataset with 3 million labels, Bonsai outperforms a state-of-the-art one-vs-rest method in terms of prediction accuracy, while being approximately 200 times faster to train. The code for Bonsai is available at https ://githu b.com/xmc-aalto /bonsa i.

An Empirical Study on Large-Scale Multi-Label Text Classification Including Few and Zero-Shot Labels

“…When a leaf node is visited, the multi-label classifier of that node decides which labels of the node will be assigned to the document. PARABEL, BONSAI: We experiment with PARA-BEL (Prabhu et al, 2018) and BONSAI (Khandagale et al, 2019), two state-of-the-art PLT-based methods. PARABEL employs binary PLTs (k = 2), while BONSAI uses non-binary PLTs (k > 2), which are shallower and wider.…”

“…• We show that hierarchical LMTC approaches based on Probabilistic Label Trees (PLTs) (Prabhu et al, 2018;Khandagale et al, 2019;You et al, 2019) outperform flat neural state-of-the-art methods, i.e., LWAN (Mullenbach et al, 2018) in two out of three datasets (EURLEX57K, AMAZON13K).…”

Generalized Zero-Shot Extreme Multi-label Learning

“…(1) State of the art extreme classifiers such as AttentionXML [66], Astec [11], DiSMEC [2], Parabel [45] and Bonsai [26] (2) Extreme classifiers which improve performance on few-shot labels such as DECAF [40], XReg [46] and PFastreXML [20] (3) Dense retrieval methods based on the state of the art natural language modelling architectures such as Sentence BERT bi-encoder [48], Fasttext [24] and WarpLDA (topic model) [10], these algorithms provide strong scalable baseline to compare ZestXML's performance over zero-shot and few-shot labels (4) Leading zero-shot multi-label learners such as 0-BIGRU-WLAN, 0-CNN-LWAN [50] and CoNSE [43], these baselines don't scale on extreme datasets, hence, ZestXML's comparison against these baselines is reported only for EURLex-4.3K in Table ??. The implementation of all the aforementioned algorithms were provided by their authors.…”

Large scale multi-label learning using Gaussian processes

“…Tree-based methods (Prabhu and Varma 2014;Jain et al 2016;Jasinska et al 2016;Niculescu-Mizil and Abbasnejad 2017;Si et al 2017;Siblini et al 2018;Prabhu et al 2018;Wydmuch et al 2018;Khandagale et al 2020) can be seen as transformation methods that aim to divide the initial large-scale problem into a multiple small-scale sub-problems by recursively partitioning the feature or label space. Those subsets are connected with the nodes of the trees.…”