Unsupervised Parsing with S-DIORA: Single Tree Encoding for Deep Inside-Outside Recursive Autoencoders

Abstract

The deep inside-outside recursive autoencoder (DIORA;Drozdov et al. 2019a) is a selfsupervised neural model that learns to induce syntactic tree structures for input sentences without access to labeled training data. In this paper, we discover that while DIORA exhaustively encodes all possible binary trees of a sentence with a soft dynamic program, its vector averaging approach is locally greedy and cannot recover from errors when computing the highest scoring parse tree in bottom-up chart parsing. To fix this issue, we introduce S-DIORA, an improved variant of DIORA that encodes a single tree rather than a softlyweighted mixture of trees by employing a hard argmax operation and a beam at each cell in the chart. Our experiments show that through fine-tuning a pre-trained DIORA with our new algorithm, we improve the state of the art in unsupervised constituency parsing on the English WSJ Penn Treebank by 2.2 6% F1, depending on the data used for fine-tuning.

R2D2: Recursive Transformer based on Differentiable Tree for Interpretable Hierarchical Language Modeling

“…We compared examples of trees inferred by our model with the corresponding ground truth constituency trees (see Appendix), encountering reasonable structures that are different from the constituent structure posited by the manually defined gold trees. Experimental results of previous work (Drozdov et al, 2020;Kim et al, 2019a) also show significant variance with different random seeds. Thus, we hypothesize that an isomorphy-focused F 1 evaluation with respect to gold constituency trees is insufficient to evaluate how reasonable the induced structures are.…”

“…Analysis. In order to better understand why our model works better when evaluating on word-piece level golden trees, we compute the recall of constituents following Kim et al (2019b) and Drozdov et al (2020). Besides standard constituents, we also compare the recall of word-piece chunks and proper noun chunks.…”

PCFGs Can Do Better: Inducing Probabilistic Context-Free Grammars with Many Symbols

“…Grammar induction using neural networks: There is a recent resurgence of interest in unsupervised constituency parsing, mostly driven by neural network based methods (Shen et al, 2018a(Shen et al, , 2019Drozdov et al, 2019Drozdov et al, , 2020Kim et al, 2019a,b;Jin et al, 2019;Zhu et al, 2020). These methods can be categorized into two major groups: those built on top of a generative grammar and those without a grammar component.…”

Learning Algebraic Recombination for Compositional Generalization

“…Unsupervised parsing (or grammar induction) trains syntax-dependent models to produce syntactic trees of natural language expressions without direct syntactic annotation (Klein and Manning, 2002;Bod, 2006;Ponvert et al, 2011;Pate and Johnson, 2016;Shen et al, 2018;Kim et al, 2019;Drozdov et al, 2020). Comparing to them, our model learns both syntax and semantics jointly.…”