Compound Probabilistic Context-Free Grammars for Grammar Induction

Abstract

We study a formalization of the grammar induction problem that models sentences as being generated by a compound probabilistic context-free grammar.In contrast to traditional formulations which learn a single stochastic grammar, our context-free rule probabilities are modulated by a per-sentence continuous latent variable, which induces marginal dependencies beyond the traditional context-free assumptions. Inference in this grammar is performed by collapsed variational inference, in which an amortized variational posterior is placed on the continuous variable, and the latent trees are marginalized with dynamic programming. Experiments on English and Chinese show the effectiveness of our approach compared to recent state-of-theart methods for grammar induction. probabilities on previous history, i.e. πz,T →w t ∝ exp(u w f2([wT ; z; ht])) where ht is the hidden state from an LSTM over x<t. However the unsupervised parsing performance was far worse (≈ 25 F1 on the PTB). 15 The train/test F1 was similar for all models. 16 https://github.com/harvardnlp/urnng

Depth-Bounded Statistical PCFG Induction as a Model of Human Grammar Acquisition

“…Other linguistic constraints and heuristics such as constraints of root nodes (Noji, Miyao, and Johnson 2016), attachment rules (Naseem et al 2010), acoustic cues (Pate and Goldwater 2013), and punctuation as phrasal boundaries (Seginer 2007a;Ponvert, Baldridge, and Erk 2011) have also been used in induction. More recently, neural PCFG induction systems (Jin et al 2019;Kim et al 2019;Kim, Dyer, and Rush 2019) and unsupervised parsing models (Shen et al 2018(Shen et al , 2019Drozdov et al 2019) have been shown to predict accurate syntactic structures. These more complex neural network models may not contain explicit biases, but may contain implicit confounding factors implemented during development on English or other natural languages, which may function like linguistic universals in constraining the search over possible grammars.…”

Scaling Hidden Markov Language Models

“…A ∝ exp(H in H out ) (4) The MLP architecture follows Kim et al (2019), with details in the appendix. This factorized parameterization, shown in Figure 1, reduces the total parameters to O(h 2 + h|Z| + h|X |).…”

Calibrating generative models: The probabilistic Chomsky–Schützenberger hierarchy

“…Probabilistic context-free grammars (PCFGs) are undoubtedly the most thoroughly studied of probabilistic grammars. Due to their ability to capture hierarchical structure in language, and the existence of good learning models, PCFGs have been ubiquitous in computational approaches to language and grammar (see, e.g., Klein and Manning 2003;Levy 2008;Kim et al 2019). But they have also seen many applications in other areas of psychology, for example, as encoding a probabilistic hypothesis space for concepts (Tenenbaum et al, 2011).…”

“…The driving motivation for early theoretical work on formal grammars came from the psychology of language, where the focus was on finding adequate frameworks for describing and explaining human grammatical competence (Chomsky, 1959(Chomsky, , 1965Chomsky and Schützenberger, 1963). With increased emphasis on detailed processing, parsing, and acquisition accounts, in addition to large-scale grammar induction from corpora in computational linguistics, a considerable amount of applied work has explored probabilistic grammars (see, e.g., Klein and Manning 2003;Levy 2008;Kim et al 2019 for representative examples), the straightforward result of adding probabilistic transitions to classical grammar formalisms. Here again, much of the interest has been to find formalisms that are powerful enough-but not too powerful-to capture relevant linguistic structure.…”

“…We consider two grammar classes that are not part of the original hierarchy, situated in between context-free and context-sensitive, namely indexed (Aho, 1968) and linear indexed (Duske and Parchmann, 1984) grammars. Both have played a notable role in computational linguistics (e.g., Gazdar 1988;Joshi et al 1991;Kim et al 2019). Probabilistic indexed grammars are capable of defining distributions with transcendental generating functions (Proposition 23).…”