There is a great deal of research undertaken for pruning away features and hidden units in order to reduce the size of Artificial Neural Networks (ANNs). However, none of these methods mentions about the relationship between the pruned unit and the number of epochs needed for retraining when the unit is pruned away from the network. In this paper, we present two heuristics for determining the pruning orders, which lead to the near smallest number of retraining epochs. The heuristics are based on the employment of the modified information gain calculated from all features in training data. Then, we test our proposed heuristics on an exclusive-or data set. The experimental results show the success of using information gain as a criterion for determining the pruning orders.
Having more hidden units than necessary can produce a neural network that has a poor generalization. This paper proposes a new algorithm for pruning unnecessary hidden units away from the single-hidden layer feedforward neural networks, resulting in a Spartan network. Our approach is simple and easy to implement, yet produces a very good result. The idea is to train the network until it begins to lose its generalization. Then the algorithm measures the sensitivity and automatically prunes away the most irrelevant unit. We define this sensitivity as the absolute difference between the desirable output and the output of the pruned network. Unlike other pruning methods, our algorithm is distinct in calculating the sensitivity from the validation set, instead of the training set, without increasing the asymptotic time complexity of the back-propagation algorithm. In addition, for a classification problem, we raise a point that the sensitivities of some well-known pruning algorithms may still underestimate the irrelevance of hidden unit even though the validation set is used in measuring the sensitivity. We resolve this problem by considering the number of misclassified patterns as the main concern. The Spartan simplicity algorithm is applied to three artificial and seven standard benchmarks. In most problems, the algorithm can produce a compact-sized network with high generalization ability in comparison with other pruning algorithms.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.