Mnemonics Training: Multi-Class Incremental Learning Without Forgetting

Nondestructive Phenolic Compounds Measurement and Origin Discrimination of Peated Barley Malt Using Near-Infrared Hyperspectral Imagery and Machine Learning

“…Due to the limited data quantity and uneven distribution of the phenol concentrations, this challenge cannot be fully addressed without expanding the data samples. Two possible solutions can be applied to tackle this particular issue, one is to enrich the phenol concentrations of the collected barley samples, and the other is to further refine the developed machine learning models, such as the incremental learning [55,56], for more robust modelling and prediction even with new categories of data.…”

Learning with Selective Forgetting

“…2. Our method uses a multiheaded network architecture that has one head per task, which is a common architecture in lifelong learning [Li and Hoiem, 2017;Chaudhry et al, 2018a]. We first introduce mnemonic code, which is the key to solving the LSF problem, and then we present loss functions for learning our model.…”

“…Regularization: This approach leverages the previous tasks' knowledge implicitly by introducing additional regularization terms. This approach can be grouped into data-drivenbased [Li and Hoiem, 2017;Hou et al, 2018;Dhar et al, 2019] andweight-constrain-based [Kirkpatrick et al, 2017;Zenke et al, 2017;Aljundi et al, 2018;Chaudhry et al, 2018a;Lee et al, 2017;Yu et al, 2020]. The former utilizes the knowledge distillation, while the latter introduces a prior on the model parameters.…”

“…Regularization Term L R : The regularization term is often introduced to prevent catastrophic forgetting. In this work, we consider using three existing regularization terms, namely, Learning without Forgetting (LwF) [Li and Hoiem, 2017], Elastic Weight Consolidation (EWC) [Kirkpatrick et al, 2017], and Memory Aware Synapses (MAS) [Aljundi et al, 2018]. LwF and EWC are originally designed to retain all classes, while we only need to memorize the classes included in the preservation set for our LWS problem.…”

“…To mitigate this harmful effect, lifelong learning (or continual learning) has been explored, in which the network is updated to adapt to a new task (e.g., a new set of classes and a new instance) without forgetting the results of past learning. Major methods can be categorized into memory-replay-based [Rebuffi et al, 2017;Lopez-Paz and Ranzato, 2017], parameterfreezing-based [Mallya and Lazebnik, 2018;Mallya et al, 2018], and regularization-based [Kirkpatrick et al, 2017;Li and Hoiem, 2017;Zenke et al, 2017;Aljundi et al, 2018] approaches. Most existing methods have been designed to learn a highly expressive model for the new task while preserving all of the knowledge for the previous tasks.…”