Are we there yet? Encoder-decoder neural networks as cognitive models of English past tense inflection

Abstract

The cognitive mechanisms needed to account for the English past tense have long been a subject of debate in linguistics and cognitive science. Neural network models were proposed early on, but were shown to have clear flaws. Recently, however, Kirov and Cotterell (2018) showed that modern encoder-decoder (ED) models overcome many of these flaws. They also presented evidence that ED models demonstrate humanlike performance in a nonce-word task. Here, we look more closely at the behaviour of their model in this task. We find that (1) the model exhibits instability across multiple simulations in terms of its correlation with human data, and (2) even when results are aggregated across simulations (treating each simulation as an individual human participant), the fit to the human data is not strong-worse than an older rule-based model. These findings hold up through several alternative training regimes and evaluation measures. Although other neural architectures might do better, we conclude that there is still insufficient evidence to claim that neural nets are a good cognitive model for this task.

Against stored abstractions: A radical exemplar model of language acquisition

“…When given the task of predicting the judgement and production data from Albright and Hayes’ (2003) novel verb study, each of these exemplar models – depending on the particular instantiation – equals or betters both a state-of-the-art connectionist model (cf. Chandler, 2010, Table 1; Kirov & Cotterell, 2018, Table 5; though see Corkery, Matusevych, & Goldwater, 2019, for concerns regarding the stability of these simulations) and Albright and Hayes’ (2003) own model which constructs an explicit micro-rule for each and every sub regularity; an approach which shows no regard for psychological plausibility, in contrast to many exemplar models which have their origins in models and findings from the non-linguistic categorization literature.…”

Learning biases in opaque interactions

“…This neural network architecture was originally designed for machine translation, but has been proposed as a baseline for morphophonological learning, and correlates well with human behaviour in a number of such tasks (Kirov 2017). For example, when tested on the experimental results from Albright & Hayes (2003), a Seq2Seq model's predictions correlated with human behaviour better than any previously proposed model (Kirov & Cotterell 2018; although see Corkery et al 2019 for a critique of these results). The Seq2Seq network learns string-to-string mappings (UR to SR mappings in this case) by updating weights for connections between nodes that are organised into multiple layers.…”

Modeling morphological learning, typology, and change: What can the neural sequence-to-sequence framework contribute?

“…Sequenceto-sequence models are also capable of learning stem-affix relationships, both morphological and phonological, as discussed in Section 3. Faruqui et al (2016) illustrates this for the case of Finnish vowel harmony (see also Corkery et al 2019); many earlier models with explicit morpheme segmentation were forced to represent this process as suppletive allomorphy (for example, positing the two phonological [ 71 ] variants of the inessive suffix, -ssa and -ssä, as suppletive allomorphs), which could lead to overassessment of the system's complexity (Stump and Finkel 2015). But the sequence-to-sequence model learns a generalizable harmony rule.…”

“…We address these questions below. acquisition Cotterell et al (2018b) suggests that sequence-to-sequence models can function as cognitive models of infant language learners (though see Corkery et al (2019) for some differences in behavior for nonce words). But to use a sequence-to-sequence model as a credible stand-in for the human infant, we must determine what the input for acquisition of morphology looks like -the right representation and learning algorithm cannot tell us anything if it is supplied with the wrong data.…”