Interpretable Probabilistic Embeddings: Bridging the Gap Between Topic Models and Neural Networks

Abstract

We consider probabilistic topic models and more recent word embedding techniques from a perspective of learning hidden semantic representations. Inspired by a striking similarity of the two approaches, we merge them and learn probabilistic embeddings with online EM-algorithm on word co-occurrence data. The resulting embeddings perform on par with Skip-Gram Negative Sampling (SGNS) on word similarity tasks and benefit in the interpretability of the components. Next, we learn probabilistic document embeddings that outperform para-graph2vec on a document similarity task and require less memory and time for training. Finally, we employ multimodal Additive Regularization of Topic Models (ARTM) to obtain a high sparsity and learn embeddings for other modalities, such as timestamps and categories. We observe further improvement of word similarity performance and meaningful inter-modality similarities.

Deep Neural Model Inspection and Comparison via Functional Neuron Pathways

“…We observe that neural interpretation approaches fall within several broad categories: visualizations and heatmaps (Karpathy et al, 2015;Strobelt et al, 2016), gradient-based analyses (Potapenko et al, 2017;Samek et al, 2017b;Bach et al, 2015;Arras et al, 2017), learning disentangled representations during training (Whitney, 2016;Siddharth et al, 2017;Esmaeili et al, 2018), and model probes (Shi et al, 2016a;Adi et al, 2016;Conneau et al, 2018;Zhu et al, 2018;Kuncoro et al, 2018;Khandelwal et al, 2018). Our work uses linear probes as a method to identify the function of groups of neurons that are correlated with linguistic and tasklevel features, rather than for interpretation of individual neurons.…”

Unsupervised dialogue intent detection via hierarchical topic model

“…The topic model simultaneously computes words and document embeddings and perform clusterization. It should be noted that in some cases topic model-based embeddings outperform traditional word embeddings, (Potapenko et al, 2017). The probability of the word w in the document d is represented by formula below:…”

Mixing syntagmatic and paradigmatic information for concept detection

“…Li et al 2016;Hu and Tsujii 2016;Wang et al 2017;Le and Lauw 2017;Li et al 2018;Peng et al 2018;Zhang, Feng, and Liang 2019) aim at making topic models that work well with short documents like tweets, where too few words are employed (sparsity problem). Others target the problem of homonymy/polysemy (Liu et al 2015;Law et al 2017), seek more interpretable topics (Potapenko, Popov, and Vorontsov 2017;Zhao et al 2018), or aim at exploiting complementary representations (S. Moody 2016;Bunk and Krestel 2018). Often word embeddings are simply seen as a means to make a more realistic model (Das, Zaheer, and Dyer 2015;Batmanghelich et al 2016;Hu and Tsujii 2016;X.…”