Logarithmic Continual Learning

Abstract

We introduce a neural network architecture that logarithmically reduces the number of self-rehearsal steps in the generative rehearsal of continually learned models. In continual learning (CL), training samples come in subsequent tasks, and the trained model can access only a single task at a time. To replay previous samples, contemporary CL methods bootstrap generative models and train them recursively with a combination of current and regenerated past data. This recurrence leads to superfluous computations as the same past samples are regenerated after each task, and the reconstruction quality successively degrades. In this work, we address these limitations and propose a new generative rehearsal architecture that requires at most logarithmic number of retraining for each sample. Our approach leverages allocation of past data in a set of generative models such that most of them do not require retraining after a task. The experimental evaluation of our logarithmic continual learning approach shows the superiority of our method with respect to the state-of-the-art generative rehearsal methods.

A Multi-label Continual Learning Framework to Scale Deep Learning Approaches for Packaging Equipment Monitoring

“…In the classic paradigm, model retraining using the new data leads to a sharp decrease in performance on previously learned tasks, a phenomenon known as Catastrophic Forgetting(CF) [Goodfellow et al, 2013]. The related literature suggests that the Rehearsal (also know as replay-based) approach appears to be a strong solution to CF [Pellegrini et al, 2020, Buzzega et al, 2021, Masarczyk et al, 2022, Kim et al, 2020. To the best of our knowledge, currently, only two main replay-based approaches designed for Multi-Label classification are proposed in the literature:…”