Unsupervised Latent Tree Induction with Deep Inside-Outside Recursive Auto-Encoders

Abstract

We introduce deep inside-outside recursive autoencoders (DIORA), a fully-unsupervised method for discovering syntax that simultaneously learns representations for constituents within the induced tree. Our approach predicts each word in an input sentence conditioned on the rest of the sentence and uses inside-outside dynamic programming to consider all possible binary trees over the sentence. At test time the CKY algorithm extracts the highest scoring parse. DIORA achieves a new state-of-the-art F1 in unsupervised binary constituency parsing (unlabeled) in two benchmark datasets, WSJ and MultiNLI.

Compound Probabilistic Context-Free Grammars for Grammar Induction

“…We observe that even on PTB, there is enough variation in setups across prior work on grammar induction to render a meaningful comparison difficult. Some important dimensions along which prior works vary include, (1) lexicalization: earlier work on grammar induction generally assumed gold (or induced) partof-speech tags (Klein and Manning, 2004;Smith and Eisner, 2004;Bod, 2006;Snyder et al, 2009), while more recent works induce grammar directly from words (Spitkovsky et al, 2013;Shen et al, 2018); (2) use of punctuation: even within papers that induce a grammar directly from words, some papers employ heuristics based on punctuation as punctuation is usually a strong signal for start/end of constituents (Seginer, 2007;Ponvert et al, 2011;Spitkovsky et al, 2013), some train with punctuation (Jin et al, 2018;Drozdov et al, 2019;Kim et al, 2019), while others discard punctuation altogether for training (Shen et al, 2018(Shen et al, , 2019; (3) train/test data: some works do not explicitly separate out train/test sets (Reichart and Rappoport, 2010;Golland et al, 2012) while some do (Huang et al, 2012;Parikh et al, 2014;Htut et al, 2018). Maintaining train/test splits is less of an issue for unsupervised structure learning, however in this work we follow the latter and separate train/test data.…”

An Imitation Learning Approach to Unsupervised Parsing

“…Yogatama et al (2017) propose to use reinforcement learning, and Maillard et al (2017) introduce the Tree-LSTM to jointly learn sentence embeddings and syntax trees, later combined with a Straight-Through Gumbel-Softmax estimator by Choi et al (2018). In addition to sentence classification tasks, recent research has focused on unsupervised structure learning for language modeling (Shen et al, 2018(Shen et al, , 2019Drozdov et al, 2019;Kim et al, 2019b). In our work, we explore the possibility for combining the merits of both sentence classification and language modeling.…”

“…As can be seen, our model is able to handle the period correctly in these examples. Although this could be specified by hand-written rules (Drozdov et al, 2019), it is in fact learned by our approach in an unsupervised manner, since punctuation marks are treated as tokens just like other words, and our training signal gives no clue regarding how punctuation marks should be processed. Moreover, our model is able to parse the verb phrases more accurately than the PRPN, including is a powerful and evocative museum and seemed a trifle embarrassed.…”

Tree Transformer: Integrating Tree Structures into Self-Attention

“…We mainly compare our model to PRPN (Shen et al, 2018a), On-lstm (Shen et al, 2018b) and Compound PCFG(C-PCFG) (Kim et al, 2019a), in which the evaluation settings and the training data are identical to our model. DIORA (Drozdov et al, 2019) and URNNG (Kim et al, 2019b) use a relative larger training data and the evaluation settings are slightly different from our model. Our model performs much better than trivial trees (i.e.…”