Dynamic Programming in Rank Space: Scaling Structured Inference with Low-Rank HMMs and PCFGs

Abstract

Hidden Markov Models (HMMs) and Probabilistic Context-Free Grammars (PCFGs) are widely used structured models, both of which can be represented as factor graph grammars (FGGs), a powerful formalism capable of describing a wide range of models. Recent research found it beneficial to use large state spaces for HMMs and PCFGs. However, inference with large state spaces is computationally demanding, especially for PCFGs. To tackle this challenge, we leverage tensor rank decomposition (aka. CPD) to decrease inference computational complexities for a subset of FGGs subsuming HMMs and PCFGs. We apply CPD on the factors of an FGG and then construct a new FGG defined in the rank space. Inference with the new FGG produces the same result but has a lower time complexity when the rank size is smaller than the state size. We conduct experiments on HMM language modeling and unsupervised PCFG parsing, showing better performance than previous work. Our code is publicly available at https://github.com/ VPeterV/RankSpace-Models.

Simple Hardware-Efficient PCFGs with Independent Left and Right Productions

“…Baselines. Our HMM baselines include neural HMM (NHMM) (Chiu et al, 2021) , LHMM (Chiu et al, 2021), and Rank HMM (Yang et al, 2022).…”

“…Figure 1: Bayesian network-like representations of PCFG binary rules: (a) original grammar, (b) after tensor decomposition (Yang et al, 2021b), and (c) rank space grammar (Yang et al, 2022). Our simple PCFG is almost the same as (c) but uses a flexible parameterization.…”

“…Comparing simple vs. low-rank PCFGs. The previous approach to scaling HMMs and PCFGs to thousands of nontermals is parameterizing the rule probability tensor T ∈ R |N |×|N |×|N | to be lowrank (Chiu et al, 2021;Yang et al, 2021bYang et al, , 2022. Low-rank PCFGs can be viewed as introducing a new latent variable, namely a "rank variable" R, to decompose π A→BC into R π A→R π B↶R π R↷C , as shown in Fig.…”

“…Despite the improvements in unsupervised parsing obtained through scaling neural probabilistic context-free grammars (PCFGs), their language model performance scales less favorably compared to, for example, hidden Markov models (HMMs) and neural language models. On the Penn Treebank, a neural PCFG with 30 nonterminals and 60 preterminals obtains ≈ 250 perplexity (Kim et al, 2019), and while scaling neural PCFGs to thousands of states via a low-rank parameterization can improve perplexity to ≈ 170 (Yang et al, 2022), this still lags behind a similarly-sized HMM, which obtains ≈ 130 perplexity (Chiu et al, 2021), despite the fact that HMMs are a subclass of PCFGs This work proposes SimplePCFG, a simple PCFG formalism with independent left and right productions. We find that this simple PCFG scales more effectively (in terms of both language modeling and unsupervised parsing) than previous approaches which scale PCFGs by factorizing the rule probability tensor into low-rank components (Yang et al, 2021b(Yang et al, , 2022.…”

“…On the Penn Treebank, a neural PCFG with 30 nonterminals and 60 preterminals obtains ≈ 250 perplexity (Kim et al, 2019), and while scaling neural PCFGs to thousands of states via a low-rank parameterization can improve perplexity to ≈ 170 (Yang et al, 2022), this still lags behind a similarly-sized HMM, which obtains ≈ 130 perplexity (Chiu et al, 2021), despite the fact that HMMs are a subclass of PCFGs This work proposes SimplePCFG, a simple PCFG formalism with independent left and right productions. We find that this simple PCFG scales more effectively (in terms of both language modeling and unsupervised parsing) than previous approaches which scale PCFGs by factorizing the rule probability tensor into low-rank components (Yang et al, 2021b(Yang et al, , 2022. In particular, we find that simple PCFGs can obtain significantly lower perplexity in language modeling while achieving higher unsupervised parsing performance compared to lowrank PCFGs with a similar number of nonterminals, achieving a near state-of-the-art unsupervised parsing performance on the Penn Treebank with an F1 of 65.1.…”