Representation of Constituents in Neural Language Models: Coordination Phrase as a Case Study

Abstract

Neural language models have achieved stateof-the-art performances on many NLP tasks, and recently have been shown to learn a number of hierarchically-sensitive syntactic dependencies between individual words. However, equally important for language processing is the ability to combine words into phrasal constituents, and use constituent-level features to drive downstream expectations. Here we investigate neural models' ability to represent constituent-level features, using coordinated noun phrases as a case study. We assess whether different neural language models trained on English and French represent phrase-level number and gender features, and use those features to drive downstream expectations. Our results suggest that models use a linear combination of NP constituent number to drive CoordNP/verb number agreement. This behavior is highly regular and even sensitive to local syntactic context, however it differs crucially from observed human behavior. Models have less success with gender agreement. Models trained on large corpora perform best, and there is no obvious advantage for models trained using explicit syntactic supervision.

Structural Supervision Improves Few-Shot Learning and Syntactic Generalization in Neural Language Models

“…Bayesian models of word learning have shown successes in acquiring proper syntactic generalizations from minimal exposure (Tenenbaum and Xu, 2000;Wang et al, 2017), however it is not clear how well neural network models would exhibit these rapid generalizations. Comparing between neural network architectures, recent work has shown that models enhanced with explicit structural supervision during training produce more humanlike syntactic generalizations (Kuncoro et al, 2017(Kuncoro et al, , 2018Wilcox et al, 2019), but it remains untested whether such supervision helps learn properties of tokens that occur rarely during training.…”