Pollack (1991) demonstrated that second-order recurrent neural networks can act as dynamical recognizers for formal languages when trained on positive and negative examples, and observed both phase transitions in learning and interacted function system-like fractal state sets. Followon work focused mainly on the extraction and minimization of a finite state automaton (FSA) from the trained network. However, such networks are capable of inducing languages that are not regular and therefore not equivalent to any FSA. Indeed, it may be simpler for a small network to fit its training data by inducing such a nonregular language. But when is the network's language not regular? In this article, using a low-dimensional network capable of learning all the Tomita data sets, we present an empirical method for testing whether the language induced by the network is regular. We also provide a detailed ε-machine analysis of trained networks for both regular and nonregular languages.
Recurrent neural network processing of regular language is reasonably well understood. Recent work has examined the less familiar question of context-free languages. Previous results regarding the language a n b n suggest that while it is possible for a small recurrent network to process context-free languages, learning them is difficult. This paper considers the reasons underlying this difficulty by considering the relationship between the dynamics of the network and weightspace. We are able to show that the dynamics required for the solution lie in a region of weightspace close to a bifurcation point where small changes in weights may result in radically different network behaviour. Furthermore, we show that the error gradient information in this region is highly irregular. We conclude that any gradient-based learning method will experience difficulty in learning the language due to the nature of the space, and that a more promising approach to improving learning performance may be to make weight changes in a nonindependent manner.
In recent years it has been shown that first order recurrent neural networks trained by gradient-descent can learn not only regular but also simple context-free and context-sensitive languages. However, the success rate was generally low and severe instability issues were encountered. The present study examines the hypothesis that a combination of evolutionary hill climbing with incremental learning and a well-balanced training set enables first order recurrent networks to reliably learn context-free and mildly context-sensitive languages. In particular, we trained the networks to predict symbols in string sequences of the context-sensitive language
Grammatical Evolution is an algorithm for evolving complete programs in an arbitrary language. By utilising a Backus Naur Form grammar the advantages of typing are achieved. A separation of genotype and phenotype allows the implementation of operators that manipulate (for instance by crossover and mutation) the genotype (in Grammatical Evolution -a sequence of bits) irrespective of the genotype to phenotype mapping (in Grammatical Evolution -an arbitrary grammar). This paper introduces a new type of crossover operator for Grammatical Evolution. The crossover operator uses information automatically extracted from the grammar to minimise any destructive impact from the crossover. The information, which is extracted at the same time as the genome is initially decoded, allows the swapping between entities of complete expansions of non-terminals in the grammar without disrupting useful blocks of code on either side of the two point crossover. In the domains tested, results confirm that the crossover is (i) more productive than hill-climbing; (ii) enables populations to continue to evolve over considerable numbers of generations without intron bloat; and (iii) allows populations (in the domains tested) to reach higher fitness levels, quicker.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
hi@scite.ai
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.