Scaling Hidden Markov Language Models

Abstract

The hidden Markov model (HMM) is a fundamental tool for sequence modeling that cleanly separates the hidden state from the emission structure. However, this separation makes it difficult to fit HMMs to large datasets in modern NLP, and they have fallen out of use due to very poor performance compared to fully observed models. This work revisits the challenge of scaling HMMs to language modeling datasets, taking ideas from recent approaches to neural modeling. We propose methods for scaling HMMs to massive state spaces while maintaining efficient exact inference, a compact parameterization, and effective regularization. Experiments show that this approach leads to models that are more accurate than previous HMM and n-gram-based methods, making progress towards the performance of state-of-the-art neural models.

PCFGs Can Do Better: Inducing Probabilistic Context-Free Grammars with Many Symbols

“…When evaluating our model with a large number of symbols, we find that only a small fraction of the symbols are predicted in the parse trees (for example, when our model uses 250 nonterminals, only tens of them are found in the predicted parse trees of the test corpus). We expect that our models can benefit from regularization techniques such as state dropout (Chiu and Rush, 2020).…”

“…For example, the best model from Petrov et al (2006) contains over 1000 nonterminal and preterminal symbols. We are also motivated by the recent work of Buhai et al (2019) who show that when learning latent variable models, increasing the number of hidden states is often helpful; and by Chiu and Rush (2020) who show that a neural hidden Markov model with up to 2 16 hidden states can achieve surprisingly good performance in language modeling. A major challenge in employing a large number of nonterminal and preterminal symbols is that representing and parsing with a PCFG requires a computational complexity that is cubic in its symbol number.…”

BERTifying the Hidden Markov Model for Multi-Source Weakly Supervised Named Entity Recognition

“…For example, Dai et al (2017) and incorporate recurrent units into the hidden semi-Markov model (HSMM) to segment and label highdimensional time series; learn discrete template structures for conditional text generation using neuralized HSMM. Wessels and Omlin (2000) and Chiu and Rush (2020) factorize HMM with neural networks to scale it and improve its sequence modeling capacity. The work most related to ours leverages neural HMM for sequence labeling (Tran et al, 2016 mizes the marginal likelihood of the observations.…”

Revisiting Simple Neural Probabilistic Language Models

“…The constrained local attention in Transformer-C is adopted at all layers of models such as Longformer (Beltagy et al, 2020) and Big Bird (Zaheer et al, 2020) due to its sparsity. Our work conceptually resembles that of Chiu and Rush (2020), who modernize HMM language models, as well as simple RNN-based language models (Merity et al, 2018). Our linguistic analysis is inspired by experiments from Khandelwal et al (2018).…”