Can LSTM Learn to Capture Agreement? The Case of Basque

Abstract

Sequential neural networks models are powerful tools in a variety of Natural Language Processing (NLP) tasks. The sequential nature of these models raises the questions: to what extent can these models implicitly learn hierarchical structures typical to human language, and what kind of grammatical phenomena can they acquire? We focus on the task of agreement prediction in Basque, as a case study for a task that requires implicit understanding of sentence structure and the acquisition of a complex but consistent morphological system. Analyzing experimental results from two syntactic prediction tasks-verb number prediction and suffix recovery-we find that sequential models perform worse on agreement prediction in Basque than one might expect on the basis of a previous agreement prediction work in English. Tentative findings based on diagnostic classifiers suggest the network makes use of local heuristics as a proxy for the hierarchical structure of the sentence. We propose the Basque agreement prediction task as challenging benchmark for models that attempt to learn regularities in human language.

Adversarial Removal of Demographic Attributes Revisited

“…Our main observation here is that training, development and test splits and random subsamples of one sample of data. Using random subsamples this way is common in machine learning, including bias detection studies (Elazar and Goldberg, 2018;Zhao et al, 2019) and probing studies (Ravfogel et al, 2018;Lin et al, 2019), but is known to overestimate performance (Globerson and Roweis, 2016), in particular for highdimensional problems.…”

Analyzing and interpreting neural networks for NLP: A report on the first BlackboxNLP workshop

“…This approach often leverages experimental paradigms from psycholinguistics that were originally developed to characterize the representations used by humans. Ravfogel et al (2018) trained RNNs to predict the agreement features of a verb in Basque; perfect accuracy on this task requires identifying the subject of the verb, which in turn requires sophisticated syntactic representations (Linzen et al 2016). In Basque, which differs from English in a large number of properties, accuracy was substantially lower than in earlier studies on English.…”

What Should/Do/Can LSTMs Learn When Parsing Auxiliary Verb Constructions?

“…However, Kuncoro et al (2018) have also shown that although sequential LSTMs can learn syntactic information, a recursive neural network that explicitly models hierarchy (the Recurrent Neural Network Grammar model from Dyer et al [2015]) is better at this: It performs better on the number agreement task from Linzen, Dupoux, and Goldberg (2016). In addition, Ravfogel, Goldberg, and Tyers (2018) and Ravfogel, Goldberg, and Linzen (2019) have cast some doubts on the results by Linzen, Dupoux, and Goldberg (2016) and Gulordava et al (2018) by looking at Basque and synthetic languages with different word orders, respectively, in the two studies.…”