Optimal ensemble averaging of neural networks

Abstract

Based on an observation about the different effect of ensemble averaging on the bias and variance portions of the prediction error, we discuss training methodologies for ensembles of networks. We demonstrate the effect of variance reduction and present a method of extrapolation to the limit of an infinite ensemble. A significant reduction of variance is obtained by averaging just over initial conditions of the neural networks, without varying architectures or training sets. The minimum of the ensemble prediction error is reached later than that of a single network. In the vicinity of the minimum, the ensemble prediction error appears to be flatter than that of the single network, thus simplifying optimal stopping decision. The results are demonstrated on sunspots data, where the predictions are among the best obtained, and on the 1993 energy prediction competition data set B.

Agnostic Learning with Ensembles of Classifiers

“…The second sum contains the cross terms of the ensemble members and disappears if the models are completely uncorrelated [8]. So the reduction in the variance of the ensemble is related to the degree of independence of the single models [7]. This is the key feature of the ensemble approach.…”

“…The extension to classification problems was straightforward after the formulation of a bias-variance decomposition for zero-one loss functions [4], [5]. The key feature of the ensemble approach is the introduction of model diversity [6], [7], [8] that helps to reduce the variance of the resulting ensemble model. There are several ways to achieve diverse models like the well known bootstrap aggregating or 'bagging' (see Breiman [9]) where the models are trained on different subsets of the training data or heterogeneous ensembles, that consist of several different model classes like Neural Networks, nearest-neighbor models, decision trees, etc [10], [11].…”

Classification of Radar Environment Using Ensemble Neural Network with Variation of Hidden Neuron Number

“…Some of the ensemble models are built and implemented as follows. In 1997, Naftaly, Intrator, and Horn revealed that the ensemble averaging was a powerful procedure which, when used correctly, improved on single network performance [12]. In 2004, the ensemble of various neural networks consisted of MLPN, Elman recurrent neural network, radial basis function network and Hopfield model was proposed for building a r obust weather forecasting [13].…”

Ensemble Approach for Recovering Phase Synchronization From Time Series

“…The approach is not very reliable because the result is so sensitive to the estimated model parameters. The main purpose of the present letter is to introduce the idea of ensemble technique [Hansen & Salamon, 1990;Meir, 1995;Krogh & Sollich, 1997;Naftaly et al, 1997] in order to realize a robust reconstruction of the synchronization diagrams from time series data.…”