Neuron importance algorithm for continual learning

Abstract

Most regularization-based continual learning methods are based on synapse importance. In contrast to them, we propose a neuron importance algorithm that uses Taylor criterion to calculate the importance of neurons. Then, for the fully connected layer and the convolutional layer, we propose two different approaches to convert neuron importance to synapse importance. Compared with existing neuron importance based method, the proposed algorithm is simple to implement, requiring only one forward propagation and one backward propagation to calculate the neuron importance. The effectiveness of the algorithm was validated on two datasets, 5-Split-MNIST and 5-Split-CIFAR10.