Generalized Zero-Shot Extreme Multi-label Learning

Abstract

Extreme Multi-label Learning (XML) involves assigning the subset of most relevant labels to a data point from millions of label choices. A hitherto unaddressed challenge in XML is that of predicting unseen labels with no training points. These form a significant fraction of total labels and contain fresh and personalized information desired by end users. Most existing extreme classifiers are not equipped for zero-shot label prediction and hence fail to leverage unseen labels. As a remedy, this paper proposes a novel approach called ZestXML for the task of Generalized Zero-shot XML (GZXML) where relevant labels have to be chosen from all available seen and unseen labels. ZestXML learns to project a data point's features close to the features of its relevant labels through a highly sparsified linear transform. This 0-constrained linear map between the two highdimensional feature vectors is tractably recovered through a novel optimizer based on Hard Thresholding. By effectively leveraging the sparsities in features, labels and the learnt model, ZestXML achieves higher accuracy and smaller model size than existing XML approaches while also promoting efficient training & prediction, real-time label update as well as explainable prediction.Experiments on large-scale GZXML datasets demonstrated that ZestXML can be up to 14% and 10% more accurate than state-ofthe-art extreme classifiers and leading BERT-based dense retrievers respectively, while having 10x smaller model size. ZestXML trains on largest dataset with 31M labels in just 30 hours on a single core of a commodity desktop. When added to an large ensemble of existing models in Bing Sponsored Search Advertising, ZestXML significantly improved click yield of IR based system by 17% and unseen query coverage by 3.4% respectively. ZestXML's source code and benchmark datasets for GZXML will be publically released for research purposes here.

Metadata-Induced Contrastive Learning for Zero-Shot Multi-Label Text Classification

“…Zero-shot multi-label text classification [50] aims to tag each document with labels that are unseen during training time but available for prediction. Most previous studies [4,16,34,38] assume that there is a set of seen classes, each of which has some annotated documents. Trained on these documents, their proposed text classifiers are expected to transfer the knowledge from seen classes to the prediction of unseen ones.…”

“…Therefore, new scoring functions are designed to promote infrequent label prediction by giving the model a higher "reward" when it predicts a tail label correctly. Propensity-scored P@𝑘 (PSP@𝑘) and propensity-scored NDCG@𝑘 (PSNDCG@𝑘, abbreviated to PSN@𝑘 in this paper) are thus proposed in [18] and widely used in LMTC evaluation [16,32,40,55,62]. PSP@𝑘 and PSN@𝑘 are defined as follows.…”

“…Following previously established parameter values [18,55,62], we set 𝐴 = 0.55, 𝐵 = 1.5, and 𝐶 = (log |D| − 1)(𝐵 + 1) 𝐴 . Therefore, the less frequent 6 When reporting PSP@𝑘 and PSN@𝑘, previous studies [16,32,40,55,62] normalize the original PSP@𝑘 and PSN@𝑘 scores by their maximum possible values (just like how DCG@𝑘 is normalized to NDCG@𝑘). Following these studies, we perform the same normalization in our calculation.…”

“…Yin et al [61] divide existing studies on zero-shot text classification into two settings: the restrictive setting [50] assumes training documents are given for some seen classes and the trained classifier should be able to predict unseen classes; the wild setting does not assume any seen classes and the classifier needs to make prediction without any annotated training data. Most previous studies on zero-shot LMTC [4,16,33,34,38] focus on the restrictive setting, while this paper studies the more challenging wild setting. Under the wild setting, a pioneering approach is dataless classification [5,44] which maps documents and labels into the same space of Wikipedia concepts.…”

“…Being aware of the annotation cost and frequent emergence of new labels, some studies [4,16,34,38] focus on zero-shot LMTC. In their settings, annotated training documents are given for a set of seen classes, and they are tasked to build a classifier to predict unseen classes.…”