Unsupervised Recurrent Neural Network Grammars

Abstract

Recurrent neural network grammars (RNNG) are generative models of language which jointly model syntax and surface structure by incrementally generating a syntax tree and sentence in a top-down, left-to-right order. Supervised RNNGs achieve strong language modeling and parsing performance, but require an annotated corpus of parse trees. In this work, we experiment with unsupervised learning of RNNGs. Since directly marginalizing over the space of latent trees is intractable, we instead apply amortized variational inference. To maximize the evidence lower bound, we develop an inference network parameterized as a neural CRF constituency parser. On language modeling, unsupervised RNNGs perform as well their supervised counterparts on benchmarks in English and Chinese. On constituency grammar induction, they are competitive with recent neural language models that induce tree structures from words through attention mechanisms.

Compound Probabilistic Context-Free Grammars for Grammar Induction

“…Also note that the F1 scores for the URNNGs in Table 3 are optimistic since we selected the best-performing runs of the original models based on validation F1 to parse the training set. Finally, as noted by Kim et al (2019), a URNNG trained from scratch fails to outperform a right-branching baseline on this version of PTB where punctuation is removed. he retired as senior vice president finance and administration and chief financial officer of the company oct. N kenneth j. unk who was named president of this thrift holding company in august resigned citing personal reasons the former president and chief executive eric w. unk resigned in june unk 's president and chief executive officer john unk said the loss stems from several factors mr. unk is executive vice president and chief financial officer of unk and will continue in those roles charles j. lawson jr. N who had been acting chief executive since june N will continue as chairman unk corp. received an N million army contract for helicopter engines boeing co. received a N million air force contract for developing cable systems for the unk missile general dynamics corp. received a N million air force contract for unk training sets grumman corp. received an N million navy contract to upgrade aircraft electronics thomson missile products with about half british aerospace 's annual revenue include the unk unk missile family already british aerospace and french unk unk unk on a british missile contract and on an air-traffic control radar system meanwhile during the the s&p trading halt s&p futures sell orders began unk up while stocks in new york kept falling sharply but the unk of s&p futures sell orders weighed on the market and the link with stocks began to fray again on friday some market makers were selling again traders said futures traders say the s&p was unk that the dow could fall as much as N points meanwhile two initial public offerings unk the unk market in their unk day of national over-the-counter trading friday traders said most of their major institutional investors on the other hand sat tight Table 4: For each query sentence (bold), we show the 5 nearest neighbors based on cosine similarity, where we take the representation for each sentence to be the mean of the variational posterior.…”

“…As with NLMs, RNNGs make no independence assumptions, and have been shown to outperform NLMs in terms of perplexity and grammaticality judgment when trained on gold trees (Kuncoro et al, 2018;Wilcox et al, 2019). We take the best run from each model and parse the training set, 15 and use the induced trees to supervise an RNNG for each model using the parameterization from Kim et al (2019). 16 We are also interested in syntactic evaluation of our models, and for this we utilize the framework and dataset from Marvin and Linzen (2018), where a model is presented two minimally different sentences such as: the senators near the assistant are old *the senators near the assistant is old and must assign higher probability to grammati- Kuncoro et al (2018 also observe that models that achieve lower perplexity do not necessarily perform better on syntactic evaluation tasks.…”

“…Shen et al (2018Shen et al ( , 2019 learn tree structures through soft gating layers within neural language models, while Drozdov et al (2019) combine recursive autoencoders with the inside-outside algorithm. Kim et al (2019) train unsupervised recurrent neural network grammars with a structured inference network to induce latent trees, and Shi et al (2019) utilize image captions to identify and ground constituents.…”

Tree Transformer: Integrating Tree Structures into Self-Attention

“…On-LSTM allows hidden neurons to learn long-term or short-term information by the proposed new gating mechanism and new activation function. In URNNG (Kim et al, 2019b), they applied amortized variational inference between a recurrent neural network grammar (RNNG) (Dyer et al, 2016) decoder and a tree structures inference network, which encourages the decoder to generate reasonable tree structures. DIORA (Drozdov et al, 2019) proposed using inside-outside dynamic programming to compose latent representations from all possible binary trees.…”

“…We mainly compare our model to PRPN (Shen et al, 2018a), On-lstm (Shen et al, 2018b) and Compound PCFG(C-PCFG) (Kim et al, 2019a), in which the evaluation settings and the training data are identical to our model. DIORA (Drozdov et al, 2019) and URNNG (Kim et al, 2019b) use a relative larger training data and the evaluation settings are slightly different from our model. Our model performs much better than trivial trees (i.e.…”