LSTM recurrent networks learn simple context-free and context-sensitive languages

Abstract

Previous work on learning regular languages from exemplary training sequences showed that long short-term memory (LSTM) outperforms traditional recurrent neural networks (RNNs). We demonstrate LSTMs superior performance on context-free language benchmarks for RNNs, and show that it works even better than previous hardwired or highly specialized architectures. To the best of our knowledge, LSTM variants are also the first RNNs to learn a simple context-sensitive language, namely a(n)b(n)c(n).

Deep learning in neural networks: An overview

“…It is possible to evolve good problemspecific topologies (Bayer et al, 2009). Some LSTM variants also use modifiable self-connections of CECs (Gers and Schmidhuber, 2001).…”

Incremental training of first order recurrent neural networks to predict a context-sensitive language

“…Together with recent parallel or subsequent work (e.g. Bodén and Wiles, 2000;Gers and Schmidhuber, 2001;Rodriguez, 2001) on learning of MA and similar tasks our study might therefore contribute to a better understanding of the capabilities of recurrent neural networks to process natural language. While our first experimental results were announced in (Chalup and Blair, 1999) we now provide a detailed exposition of our investigation.…”

“…With the aim of investigating learnability of the one-step look-ahead prediction task for non-regular languages we performed experiments using training sequences formed by strings of one of the following types, where always n ≥ 1: Since our interest is focused on the language of multiple agreements MA = {s n ; n ≥ 1} we will from now on use strings s n = a Since the depth n is not known to the network at the start of a string it cannot predict when the first b will occur and logically it cannot know how many a's 3 Some authors have applied a more rigid interpretation, insisting that the output for the predicted symbol must be above a fixed threshold and that the outputs for all other symbols must be below that threshold (e. g. Rodriguez et al, 1999;Bodén and Wiles, 2000;Gers and Schmidhuber, 2001). …”

“…Tonkes and Wiles (1999) suggested that the limited generalisation ability could model human performance when processing center embedded sentences. In two recent studies Melnik et al (2000) evolved RAAM networks capable of expressing all strings of the a n b n language, while Gers and Schmidhuber (2001) trained Long Short-Term Memory Networks to predict the a n b n c n language with substantial generalisation ability. Both these network types are more complex than the first order networks of the present study and also the task presentation was different.…”

“…This would alleviate the need for data juggling because the order of the strings would then be irrelevant. Gers and Schmidhuber (2001) employed an end-of-string marker but did not reset activations at the end of the strings. Wiles (2000, 2001) did not use the end-of-string marker and updated the weights at the end of each string without resetting at all.…”