Learning and Generalization in Cascade Network Architectures

Abstract

Incrementally constructed cascade architectures are a promising alternative to networks of predefined size. This paper compares the direct cascade architecture (DCA) proposed in Littmann and Ritter (1992) to the cascade-correlation approach of Fahlman and Lebiere (1990) and to related approaches and discusses the properties on the basis of various benchmark results. One important virtue of DCA is that it allows the cascading of entire subnetworks, even if these admit no error-backpropagation. Exploiting this flexibility and using LLM networks as cascaded elements, we show that the performance of the resulting network cascades can be greatly enhanced compared to the performance of a single network. Our results for the Mackey-Glass time series prediction task indicate that such deeply cascaded network architectures achieve good generalization even on small data sets, when shallow, broad architectures of comparable size suffer from overfitting. We conclude that the DCA approach offers a powerful and flexible alternative to existing schemes such as, e.g., the mixtures of experts approach, for the construction of modular systems from a wide range of subnetwork types.

An empirical comparison of cascade and flat variants of incremental node creation algorithm

“…Previous research did not find out the conclusive decision about generalization capabilities of standard CCN and flat CCN. Littman and Ritter suggested that standard CCN generalized better than the flat CCN whereas Sjogaard suggested that the flat CCN was a better choice than the standard CCN [39,10]. Prechelt empirically found that the flat variant was superior to the cascade variant for some problems [17].…”

An Ensemble of Cooperative Extended Kohonen Maps for Complex Robot Motion Tasks

“…Our indirect-mapping EKM resembles the EKM models of (Littmann and Ritter, 1996;Walter and Schulten, 1993), which utilize locally linear mappings. In their models, each neuron stores both the motor control vector and the matrix of motor control parameters as output weights (Eq.…”

“…Furthermore, we have shown that training the control parameters with recursive least squares enables faster convergence and better performance compared to gradient descent. Their EKM models (Littmann and Ritter, 1996;Walter and Schulten, 1993) have only used gradient descent to learn the control parameters.…”

“…Such a high degree of smoothness, flexibility, and precision in motion control is essential for efficiently executing complex tasks and interacting with humans. It is well understood how a SOFM or EKM is used for learning sensorimotor control (Littmann and Ritter, 1996;Walter and Schulten, 1993). However, the non-trivial problem of combining multiple SOFMs or EKMs for sophisticated control (e.g., negotiation of unforeseen complex obstacles and cooperative multi-robot tracking of moving targets (Low et al, 2003)) is not well studied.…”

Investigation of the CasCor Family of Learning Algorithms

“…Ash, 1989, Wang et al, 1994, propose addition of units in the hidden layers of standard MLPs during normal backpropagation training, but this approach severely disturbs the training process because of the interaction of hidden units. Not even constructive unit splitting with reasonable initializations for the new units works well Hanson, 1989, WynneJones, 1991. Littmann and Ritter, 1993 propose direct cascading where local linear maps or di erent neural modules are cascaded and produce the output from the union of the original network inputs and the outputs of previous modules.…”