A Systematic Assessment of Syntactic Generalization in Neural Language Models

Abstract

While state-of-the-art neural network models continue to achieve lower perplexity scores on language modeling benchmarks, it remains unknown whether optimizing for broad-coverage predictive performance leads to human-like syntactic knowledge. Furthermore, existing work has not provided a clear picture about the model properties required to produce proper syntactic generalizations. We present a systematic evaluation of the syntactic knowledge of neural language models, testing 20 combinations of model types and data sizes on a set of 34 English-language syntactic test suites. We find substantial differences in syntactic generalization performance by model architecture, with sequential models underperforming other architectures. Factorially manipulating model architecture and training dataset size (1M-40M words), we find that variability in syntactic generalization performance is substantially greater by architecture than by dataset size for the corpora tested in our experiments. Our results also reveal a dissociation between perplexity and syntactic generalization performance.

Structural Supervision Improves Few-Shot Learning and Syntactic Generalization in Neural Language Models

“…Much has been written about the ability of ANNs to learn number agreement (Linzen et al, 2016;Gulordava et al, 2018;Giulianelli et al, 2018), including their ability to maintain the dependency across different types of intervening material (Marvin and Linzen, 2018) and with coordinated noun phrases . Hu et al (2020) find that model architecture, rather than training data size, may contribute most to performance on number agreement and related tasks. Focusing on RNN models, Lakretz et al (2019) find evidence that number agreement is tracked by specific "number" units that work in concert with units that carry more general syntactic information like tree depth.…”

Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)

“…find that while LMs learn agreement phenomena at a similarly early stage, other phenomena require more data to learn. Finally, Hu et al (2020) find that adopting architectures that build in linguistic bias, such as RNNGs (Dyer et al, 2016), has a big-ger effect on the acceptability task than increasing training data from 1M to 40M words.…”

“…For instance, an RNN classifier is capable of representing any function, but prefers ones that focus mostly on local relationships within the input sequence (Dhingra et al, 2018;Ravfogel et al, 2019). Some recent work seeks to design neural architectures that build in desirable inductive biases (Dyer et al, 2016;Battaglia et al, 2018), or compares the immutable biases of different architectures Hu et al, 2020). However, inductive biases can also be learned by biological (Harlow, 1949) and artificial systems alike (Lake et al, 2017).…”

Discourse structure interacts with reference but not syntax in neural language models

“…Recent work has suggested that LMs acquire abstract, often human-like, knowledge of syntax (e.g., Gulordava et al, 2018;Hu et al, 2020). Additionally, knowledge of grammatical and referential aspects linking a pronoun to its antecedent noun (reference) have been demonstrated for both transformer and long short-term memory architectures (Sorodoc et al, 2020).…”