Prediction-Constrained Training for Semi-Supervised Mixture and Topic Models

Abstract

Supervisory signals have the potential to make low-dimensional data representations, like those learned by mixture and topic models, more interpretable and useful. We propose a framework for training latent variable models that explicitly balances two goals: recovery of faithful generative explanations of high-dimensional data, and accurate prediction of associated semantic labels. Existing approaches fail to achieve these goals due to an incomplete treatment of a fundamental asymmetry: the intended application is always predicting labels from data, not data from labels. Our prediction-constrained objective for training generative models coherently integrates loss-based supervisory signals while enabling effective semisupervised learning from partially labeled data. We derive learning algorithms for semi-supervised mixture and topic models using stochastic gradient descent with automatic differentiation. We demonstrate improved prediction quality compared to several previous supervised topic models, achieving predictions competitive with high-dimensional logistic regression on text sentiment analysis and electronic health records tasks while simultaneously learning interpretable topics.

On Learning Prediction-Focused Mixtures

“…While this bound does explicitly demonstrate a generative-discriminative trade-off analogous to Hughes et al (2017), the bound can neither be optimized directly in a computationally efficient manner, nor does it imply that the joint likelihood in Eq. 1 captures this trade-off-indeed, it would be unfortunate if our results were because of our choice of approximate inference rather than the due to the model itself.…”

“…Baselines. We compare the generative and predictive performance of pf-GMM to (1) an approach that first learns a generative model using only the inputs and then trains a logistic regression classifier on the posterior beliefs p(Z|X) (2-Step-GMM), (2) a supervised Gaussian mixture model without the switch parameters (sup-GMM), (3) a logistic regression model, which is a discriminative approach that maximizes log p(Y |X) directly (LogReg), and (4) a prediction-constrained GMM (Hughes et al 2017) (pc-GMM). For the pf-HMM, we similarly compare to sup-HMM, 2-Step-HMM and LogReg.…”

“…Most of these train for the discriminative loss only. Closer to our goal of managing the generative-discriminative trade-off, Hughes et al (2017) proposed prediction-constrained learning in the context of mixture and topic models, noting the asymmetry of the task in predicting labels from documents (and not the other way around). Their framework balances the maximum likelihood objective with predictive performance by introducing the latter as a constraint and then solving the resulting constrained optimization problem.…”

“…The simplest two-step approach of maximizing the likelihood of the observed data and using that model for prediction largely fails because the learned model may not have modeled aspects of the inputs most useful for the prediction task. Alternatives such as prediction-constrained models (Hughes et al 2017) explicitly model the generative-discriminative trade-off in a joint optimization, but are difficult to optimize in practice.…”

“…In our work, we manage the trade-off between explaining the data and the predictive objective by explicitly using 'switch' variables that allow our models to select only those dimensions that are relevant to the downstream task, and enable accurate predictions even in settings where number of components is misspecified. Unlike Hughes et al 2017, the entire problem can be written as a graphical model with the discriminative-generative trade-off being managed by a single hyperparameter corresponding to the probability that a dimension is relevant. This setting was partially explored in Ren et al 2020, who considered a specific context of relevant words for topic models; we add the following contributions:…”