Statistical topic models for multi-label document classification

Abstract

Machine learning approaches to multi-label document classification have to date largely relied on discriminative modeling techniques such as support vector machines. A drawback of these approaches is that performance rapidly drops off as the total number of labels and the number of labels per document increase. This problem is amplified when the label frequencies exhibit the type of highly skewed distributions that are often observed in real-world datasets. In this paper we investigate a class of generative statistical topic models for multi-label documents that associate individual word tokens with different labels. We investigate the advantages of this approach relative to discriminative models, particularly with respect to classification problems involving large numbers of relatively rare labels. We compare the performance of generative and discriminative approaches on document labeling tasks ranging from datasets with several thousand labels to datasets with tens of labels. The experimental results indicate that probabilistic generative models can achieve competitive multi-label classification performance compared to discriminative methods, and have advantages for datasets with many labels and skewed label frequencies.

A Bayesian nonparametric model for multi-label learning

“…One category of using topic model idea for multi-label learning is to directly replace topics in Latent Dirichlet Allocation (LDA) (Blei et al 2003) by labels, such as Labeled LDA (Ramage et al 2009) and Flat-LDA (Rubin et al 2012). Prior-LDA (Rubin et al 2012) is further proposed to account for the label frequency differences within a corpus through introducing a label sampling step by multinomial distribution.…”

“…Prior-LDA (Rubin et al 2012) is further proposed to account for the label frequency differences within a corpus through introducing a label sampling step by multinomial distribution. However, the dependency between the labels is not considered, which is resolved by the Dependency-LDA (Rubin et al 2012) later. Parametric Mixture Models (Ueda and Saito 2002) are also proposed to capture the pairwise label correlation.…”

“…The generative models learn a joint distribution of data and the latent variables, while discriminative models only learn a conditional distribution of latent variables given data. Comparing with discriminative models, generative ones are characterized by the capability of handling the following situations: 1) the number of labels is large (Rubin et al 2012); 2) the number of training data is small. Current Generative models for multi-label learning are mainly based on topic models (Rai et al 2015;Rubin et al 2012), which learn a low-dimensional label embedding (Rai et al 2015).…”

“…Comparing with discriminative models, generative ones are characterized by the capability of handling the following situations: 1) the number of labels is large (Rubin et al 2012); 2) the number of training data is small. Current Generative models for multi-label learning are mainly based on topic models (Rai et al 2015;Rubin et al 2012), which learn a low-dimensional label embedding (Rai et al 2015). It means that the labels and instances could be represented by a relatively low-dimensional vector and each dimension of vectors is seen as a topic.…”

Predicting protein function via multi-label supervised topic model on gene ontology

“…Nonetheless, a topic model is not only a clustering algorithm [20]. Specially, the characteristics of LDA have been discussed for multi-label problem in [21]: (1) it transforms the word-level statistics of each document to its label-level distribution; (2) it models all labels at the same time rather than treating each label independently. In multi-label classification problem, LDA can also incorporate a label set into its learning procedure and become a supervised model.…”