Unsupervised Labeled Parsing with Deep Inside-Outside Recursive Autoencoders

Abstract

Understanding text often requires identifying meaningful constituent spans such as noun phrases and verb phrases. In this work, we show that we can effectively recover these types of labels using the learned phrase vectors from deep inside-outside recursive autoencoders (DIORA). Specifically, we cluster span representations to induce span labels. Additionally, we improve the model's labeling accuracy by integrating latent code learning into the training procedure. We evaluate this approach empirically through unsupervised labeled constituency parsing. Our method outperforms ELMo and BERT on two versions of the Wall Street Journal (WSJ) dataset and is competitive to prior work that requires additional human annotations, improving over a previous state-of-the-art system that depends on ground-truth part-of-speech tags by 5 absolute F1 points (19% relative error reduction).

Unsupervised Parsing with S-DIORA: Single Tree Encoding for Deep Inside-Outside Recursive Autoencoders

“…Typically unsupervised constituency parsing is purely evaluated by its structure, although recent work fromDrozdov et al (2019b) shows that a simple approach to induce labels with DIORA can be done by clustering the inside and outside phrase vectors.…”

Co-training an Unsupervised Constituency Parser with Weak Supervision

“…Table 1 shows the unlabeled F 1 scores for our model compared to existing unsupervised parsers on PTB. The vanilla inside model is in itself competitive and is already in the range of previous best models like DIORA (Drozdov et al, 2019), Compound PCFG (Kim et al, 2019a). 4 See Appendix 3 https://nlp.cs.nyu.edu/evalb 4 We do not include the results of Shi et al (2021) in our analysis because their boost in the performance is contingent on the nature of the supervision data (especially the QA-SRL dataset) rather than on the actual learning process itself.…”

“…Following prior work (Kim et al, 2019a;Shen et al, 2018Shen et al, , 2019Cao et al, 2020), we remove punctuation and collapse unary chains before evaluation, and calculate F 1 ignoring trivial spans, i.e., single-word spans and whole-sentence spans, and we perform the averaging at sentence-level (macro average) rather than span-level (micro average), which means that we compute F 1 for each sentence and later average over all sentences. We also mention the oracle (Shen et al, 2019) 47.7 49.4 63.9 -Tree Transformer † (Wang et al, 2019) 50.5 52.0 66.2 -Neural PCFG † (Kim et al, 2019a) 50.8 52.6 64.6 -DIORA (Drozdov et al, 2019) -58.9 60.5 -Compound PCFG † (Kim et al, 2019a) 55.2 60.1 70.5 -S-DIORA † (Drozdov et al, 2020) 57.6 64.0 71.8 -Constituency Test (Cao et al, 2020) 62.8 65.9 (2019a) and take the baseline numbers of certain models from (Kim et al, 2019a;Cao et al, 2020). † denotes models trained without punctuation and denotes models trained on additional data.…”

“…For preprocessing, we keep all punctuation and remove any trailing punctuation. To maintain the unsupervised nature of our experiments, we avoid the common practice of using gold parses of the validation set for either early stopping (Shen et al, 2018(Shen et al, , 2019Drozdov et al, 2019) or hyperparameter tuning (Kim et al, 2019a). Additionally, we experiment on Chinese with version 5.1 of the Chinese Penn Treebank (CTB; Xue et al 2005) with the same splits as in Chen and Manning (2014), and the Japanese Keyaki Treebank (KTB; Butler et al 2012).…”

“…PLMs that possess rich contextualized textual representations can assist parsing when we have a match syntactic constituents, hence restricting the scope for the actual algorithm to derive meaningful trees. (Kim et al, 2019b) 10 10.2 22.7 22.7 DIORA (Drozdov et al, 2019) 24.9 26.0 42.3 43.3 DIORA-all (Hong et al, 2020) 36.4 40.0 47.1 48.9 Ours (using inside) 33.7 36.3 53.8 55.9 Ours (using inside w/ self-training) 37.6 39.8 55.5 58.2 Ours (using inside and outside w/ co-training) 39.2 41.1 56.7 59.1 Upper Bound 76.5 76.6…”