Do latent tree learning models identify meaningful structure in sentences?

Abstract

Recent work on the problem of latent tree learning has made it possible to train neural networks that learn to both parse a sentence and use the resulting parse to interpret the sentence, all without exposure to groundtruth parse trees at training time. Surprisingly, these models often perform better at sentence understanding tasks than models that use parse trees from conventional parsers. This paper aims to investigate what these latent tree learning models learn. We replicate two such models in a shared codebase and find that (i) only one of these models outperforms conventional tree-structured models on sentence classification, (ii) its parsing strategies are not especially consistent across random restarts, (iii) the parses it produces tend to be shallower than standard Penn Treebank (PTB) parses, and (iv) they do not resemble those of PTB or any other semantic or syntactic formalism that the authors are aware of.

Visually Grounded Neural Syntax Acquisition

“…We evaluate the induced constituency parse trees via the overall F 1 score, as well as the recall of four types of constituents: noun phrases (NP), verb phrases (VP), prepositional phrases (PP), and adjective phrases (ADJP) ( Table 1). We also evaluate the robustness of models trained with fixed data and hyperparameters, but different random initialization, in two ways: via the standard deviation of performance across multiple runs, and via the selfagreement F 1 score (Williams et al, 2018), which is the average F 1 taken over pairs of different runs. Among all of the models which do not require extra labels, VG-NSL with the head-initial inductive bias (VG-NSL+HI) achieves the best F 1 score.…”

“…Recent work has proposed several approaches for inducing latent syntactic structures, including constituency trees (Choi et al, 2018;Yogatama et al, 2017;Maillard and Clark, 2018;Havrylov et al, 2019;Kim et al, 2019;Drozdov et al, 2019) and dependency trees (Shi et al, 2019), from the distant supervision of downstream tasks. However, most of the methods are not able to produce linguistically sound structures, or even consistent ones with fixed data and hyperparameters but different random initializations (Williams et al, 2018).…”

“…Trivial tree structures. Similarly to recent work on latent tree structures (Williams et al, 2018;Phu Mon Htut et al, 2018;Shi et al, 2018b), we include three types of trivial baselines without linguistic information: random binary trees, left-branching binary trees, and right-branching binary trees.…”

“…The learning Table 1: Recall of specific typed phrases, and overall F 1 score, evaluated on the MSCOCO test split, averaged over 5 runs with different random initializations. We also include self-agreement F 1 score (Williams et al, 2018) across the 5 runs. ± denotes standard deviation.…”

Learning Latent Trees with Stochastic Perturbations and Differentiable Dynamic Programming

“…For example, the structured attention method (Kim et al, 2017;Liu and Lapata, 2018) does not sample entire trees but rather computes arc marginals, and hence does not faithfully represent higher-order statistics. Much of other previous work relies either on reinforce-ment learning Nangia and Bowman, 2018;Williams et al, 2018a) or does not treat the latent structure as a random variable (Peng et al, 2018). Niculae et al (2018) marginalizes over latent structures, however, this necessitates strong sparsity assumptions on the posterior distributions which may inject undesirable biases in the model.…”