Bayesian selective combination of multiple neural networks for improving long-range predictions in nonlinear process modelling

Abstract

A Bayesian selective combination method is proposed for combining multiple neural networks in nonlinear dynamic process modelling. Instead of using fixed combination weights, the probability of a particular network being the true model is used as the combination weight for combining that network. The prior probability is calculated using the sum of squared errors of individual networks on a sliding window covering the most recent sampling times. A nearest neighbour method is used for estimating the network error for a given input data point, which is then used in calculating the combination weights for individual networks. Forward selection and backward elimination are used to select the individual networks to be combined. In forward selection, individual networks are gradually added into the aggregated network until the aggregated network error on the original training and testing data sets cannot be further reduced. In backward elimination, all the individual networks are initially aggregated and some of the individual networks are then gradually eliminated until the aggregated network error on the original training and testing data sets cannot be further reduced. Application results demonstrate that the proposed techniques can significantly improve model generalisation and perform better than aggregating all the individual networks.

Multiple neural networks modeling techniques in process control: a review

“…Nonlinear combination methods include Dempster-Shafer belief based method, rank-based information, majority voting, order statistic and Tumer and Ghosh. [35] Among these nonlinear methods, Dempster-Shafer seems to be the most renowned method. The Demspter-Shafer belief based method is somewhat complex and it has to deal with the uncertainty and ignorance of the classifiers.…”

“…Unlike PCR or MLR, the aggregating weight for a particular network is selected as the posterior probability that the network is a true model with the posterior probability of the entire networks sum to 1. The individual network model and aggregated neural network model are represented respectively as follows [35] :…”

“…Therefore, excluding certain networks could be a wise method to tackle the problem. [35] There are several methods on how to select proper networks for combination. For example, Perrone and Copper suggested a heuristics selection method whereby the population of trained networks were ordered in terms of increasing MSE and only those with lower sum of squared errors were selected for combination.…”

Face Recognition Using DCT and Hierarchical RBF Model

“…Hierarchical RBF network has been proved effective in the reconstruction of smooth surfaces from sparse noisy data points [5]. In order to improve the model generalization performance, a selective combination of multiple neural networks by using Bayesian method was proposed in [6].…”

Stock Index Modeling Using Hierarchical Radial Basis Function Networks

“…HiRBF has been proved effective in the reconstruction of smooth surfaces from sparse noisy data points [18]. In order to improve the model generalization performance, a selective combination of multiple neural networks by using Bayesian method was proposed in [19].…”