Representations of Syntax [MASK] Useful: Effects of Constituency and Dependency Structure in Recursive LSTMs

Abstract

Sequence-based neural networks show significant sensitivity to syntactic structure, but they still perform less well on syntactic tasks than tree-based networks. Such tree-based networks can be provided with a constituency parse, a dependency parse, or both. We evaluate which of these two representational schemes more effectively introduces biases for syntactic structure that increase performance on the subject-verb agreement prediction task. We find that a constituency-based network generalizes more robustly than a dependencybased one, and that combining the two types of structure does not yield further improvement. Finally, we show that the syntactic robustness of sequential models can be substantially improved by fine-tuning on a small amount of constructed data, suggesting that data augmentation is a viable alternative to explicit constituency structure for imparting the syntactic biases that sequential models are lacking.

A‐PGRD: Attention‐based automatic business process model generation from RPA process description

“…Although Reference 26 argues that compositional parsing is more important than dependency parsing for the word‐to‐word hierarchy in a sentence, given that the sentence‐to‐sentence relationships in a paragraph cannot be defined in terms of compositions and that our experimental results (Figure 7) suggest that particular sentence‐initial verbs are more effective for parsing the sentence‐to‐sentence hierarchy, we reckon that our unsupervised model is more biased toward dependency parsing. Therefore, a larger dataset of business process descriptions would be very effective for our work.…”

Can Recurrent Neural Networks Validate Usage-Based Theories of Grammar Acquisition?

“…Lepori et al ( 2020 ) experiment with an artificially constructed set of simple transitive sentences (Subject-Verb-Object), containing optional adjectival or prepositional modifiers in a controlled, probabilistic setting. They show that when a BiLSTM is fine-tuned on a distribution which explicitly requires moving beyond lexical co-occurrences and creating more abstract representations, performance dramatically improves: this suggests that a simple sequential mechanism can be enough if the linguistic signal is structured in a way that abstraction is encouraged.…”

Can RNNs trained on harder subject-verb agreement instances still perform well on easier ones?

“…We also conduct an experiment that has been proposed in a previous study (Lepori et al, 2020) to impart hierarchical bias to the models, and found out that it did not help in our scenario ( §A.7).…”

“…Note that, since the projection is in 2 dimensions, to measure the spread evaluate l 2 norm of a vector of standard deviations across individual components. A.7 Fine-Tuning Lepori et al (2020) showed that the syntactic robustness of RNNs could be improved by finetuning the trained models on a small amount of syntactically challenging data. We consider a similar exercise for our trained language models (Selective sampling), where we further fine-tuned the model with the challenging artificially generated sentences.…”