Determining the Orders of Feature and Hidden Unit Prunings of Artificial Neural Networks

Abstract

There is a great deal of research undertaken for pruning away features and hidden units in order to reduce the size of Artificial Neural Networks (ANNs). However, none of these methods mentions about the relationship between the pruned unit and the number of epochs needed for retraining when the unit is pruned away from the network. In this paper, we present two heuristics for determining the pruning orders, which lead to the near smallest number of retraining epochs. The heuristics are based on the employment of the modified information gain calculated from all features in training data. Then, we test our proposed heuristics on an exclusive-or data set. The experimental results show the success of using information gain as a criterion for determining the pruning orders.

Hidden Unit Reduction of Artificial Neural Network on English Capital Letter Recognition

“…The analysis of the experimental results show exceeding low number of hidden units required to the classification process. In addition, the results support our heuristics [14] in terms of the compact network and the nearly minimum pruning time.…”

“…We describe a hidden unit pruning heuristic (Jearanaitanakij and Pinngern, 2005, [14]) used as ordering criterions for the hidden unit pruning in the artificial neural network. Before performing the hidden unit pruning, we must calculate the information gains of all features and then pass these gains to the hidden units in the next layer.…”

“…Hagiwara [13] presented the Consuming Energy and the Weights Power methods for removal of both hidden units and weights, respectively. Jearanaitanakij and Pinngern [14] proposed the informationgain based pruning heuristic that can efficiently remove unnecessary hidden units within the nearly minimum period of time.…”

“…Our objective is to determine the minimum number of hidden units that is required to classify the 26-English letters with the practical recognition rate. Firstly, unnecessary features are filtered out of the data set by the feature pruning heuristic [14]. Then the hidden unit pruning heuristic [14] is utilized in order to find a suitable number of hidden units.…”

“…Firstly, unnecessary features are filtered out of the data set by the feature pruning heuristic [14]. Then the hidden unit pruning heuristic [14] is utilized in order to find a suitable number of hidden units. The analysis of the experimental results show exceeding low number of hidden units required to the classification process.…”