Reduced HyperBF Networks: Regularization by Explicit Complexity Reduction and Scaled Rprop-Based Training

Abstract

Hyper basis function (HyperBF) networks are generalized radial basis function neural networks (where the activation function is a radial function of a weighted distance). Such generalization provides HyperBF networks with high capacity to learn complex functions, which in turn make them susceptible to overfitting and poor generalization. Moreover, training a HyperBF network demands the weights, centers, and local scaling factors to be optimized simultaneously. In the case of a relatively large dataset with a large network structure, such optimization becomes computationally challenging. In this paper, a new regularization method that performs soft local dimension reduction in addition to weight decay is proposed. The regularized HyperBF network is shown to provide classification accuracy competitive to a support vector machine while requiring a significantly smaller network structure. Furthermore, a practical training to construct HyperBF networks is presented. Hierarchal clustering is used to initialize neurons followed by a gradient optimization using a scaled version of the Rprop algorithm with a localized partial backtracking step. Experimental results on seven datasets show that the proposed training provides faster and smoother convergence than the regular Rprop algorithm.

A survey on hyper basis function neural networks

“…When classification is made by machine learning, the SVM method is considered as the most successful classification method due to its high accuracy and fast speed. The classification accuracy of HBFNNs based on regularization optimization can be comparable to SVM (Mahdi & Rouchka, 2011).…”

“…where j is a positive-definite weighted matrix. According to different requirements, the matrix j can be divided into different forms (Mahdi & Rouchka, 2011) as follows:…”

“…Through this approach, HBFNNs can decrease the correlation interference between input variables (Wen, Zhang, & Zhu, 2015). Moreover, the number of HBFNN neurons needed is smaller than that of a RBFNN in order to learn a complex function, which makes HBFNNs more competitive to achieve fast learning with limited computation resources (Mahdi & Rouchka, 2011;Vukovi & Miljkovi, 2013;Wu, Kong, & Yang,2018).…”

“…However, there were several problems in original HBFNNs to limit their applications, for example (Mahdi & Rouchka, 2011;Vukovi & Miljkovi, 2013):…”

“…In view of the above shortcomings of HBFNNs, different training methods have been proposed for HBFNNs. Mahdi and Rouchka (2011) proposed a new regularization algorithm to train HBFNNs. When dealing with classification problems, the classification accuracy of HBFNNs was comparable to the support vector machine (SVM) approach.…”