Class-Incremental Learning with Generative Classifiers

Abstract

Incrementally training deep neural networks to recognize new classes is a challenging problem. Most existing class-incremental learning methods store data or use generative replay, both of which have drawbacks, while 'rehearsal-free' alternatives such as parameter regularization or bias-correction methods do not consistently achieve high performance. Here, we put forward a new strategy for class-incremental learning: generative classification. Rather than directly learning the conditional distribution p(y|x), our proposal is to learn the joint distribution p(x, y), factorized as p(x|y)p(y), and to perform classification using Bayes' rule. As a proof-of-principle, here we implement this strategy by training a variational autoencoder for each class to be learned and by using importance sampling to estimate the likelihoods p(x|y). This simple approach performs very well on a diverse set of continual learning benchmarks, outperforming generative replay and other existing baselines that do not store data.

Dendritic Self-Organizing Maps for Continual Learning

“…However, this algorithm employs SI to introduce the concept of parameter importance and thus tune the shared weights Θ across the entire dataset. Finally, in generative classification [18], a separate variational autoencoder (VAE) is trained in order to learn the data distribution for each class. Next, importance sampling is employed to estimate the likelihood of a test sample x under the VAE model of class y denoted as p(x|y), while p(y) can either be assumed to follow a normal distribution or is calculated by counting sample observations, where the Bayesian rule is used for inference.…”

“…Finally, we compare the performance of our algorithm to the state-of-the-art models [18] in Class-IL scenario, on both Split-MNIST (Figure 8C) and Split-CIFAR-10 (Figure 8D) datasets.…”