LightXML: Transformer with Dynamic Negative Sampling for High-Performance Extreme Multi-label Text Classification

Abstract

Extreme Multi-label text Classification (XMC) is a task of finding the most relevant labels from a large label set. Nowadays deep learning-based methods have shown significant success in XMC. However, the existing methods (e.g., Atten-tionXML and X-Transformer etc) still suffer from 1) combining several models to train and predict for one dataset, and 2) sampling negative labels statically during the process of training label ranking model, which reduces both the efficiency and accuracy of the model. To address the above problems, we proposed LightXML, which adopts endto-end training and dynamic negative labels sampling. In LightXML, we use generative cooperative networks to recall and rank labels, in which label recalling part generates negative and positive labels, and label ranking part distinguishes positive labels from these labels. Through these networks, negative labels are sampled dynamically during label ranking part training by feeding with the same text representation. Extensive experiments show that LightXML outperforms state-of-the-art methods in five extreme multi-label datasets with much smaller model size and lower computational complexity. In particular, on the Amazon dataset with 670K labels, LightXML can reduce the model size up to 72% compared to AttentionXML. Our code is available at http://github.com/kongds/LightXML.

Long-tailed Extreme Multi-label Text Classification with Generated Pseudo Label Descriptions

“…Although previous works leverage negative sampling to alleviate the problem (Jiang et al, 2021;Chang et al, 2020), we argue that it is important to initialize the label embedding with the label side information.…”

“…The XML-CNN (Liu et al, 2017) and SLICE (Jain et al, 2019) employ the convolutional neural network on word embeddings for document representation. More recently, X-Transformer (Chang et al, 2020), LightXML (Jiang et al, 2021) and APLC-XLNet tames large pre-trained Transformer models to encode the input document into a fixed vector. AttentionXML (You et al, 2018) applies a label-word attention mechanism to calculate label-aware document embeddings, but it requires more computational cost proportional to the document length.…”

“…The DEPL+c model looks like an ensemble of the two systems at the first sight, but there are two major differences: 1) As the BERT encoder is shared between the classification and retrieval modules, it doesn't significantly increase the number of parameters as in (Chang et al, 2020;Jiang et al, 2021); and 2) when the two modules are optimized together, the system can take advantages of both units according to the situation of head or tail label predictions.…”

“…For the tail label evaluation, our method is compared with the SOTA deep learning models including X-Transformer (Chang et al, 2020), XLNet-APLC , LightXML (Jiang et al, 2021), and AttentionXML (You et al, 2018). X-Transformer, LightXML, and XLNet-APLC employ pre-trained Transformers for document representation.…”

Sparse and Dense Approaches for the Full-rank Retrieval of Responses for Dialogues

“…This problem is also present for other machine learning techniques that do self-supervision through contrastive learning in different domains such as computer vision, natural language and graphs [21,52,70]. For example the well-known word2vec [38] word embedding technique randomly samples words that are not relevant for the context (other words in the sentence) to distinguish from the actual word that is part of the context.…”