Overcoming Catastrophic Forgetting With Unlabeled Data in the Wild

Lee, Kibok; Lee, Kimin; Shin, Jinwoo; Lee, Honglak

doi:10.1109/iccv.2019.00040

Cited by 151 publications

(145 citation statements)

References 32 publications

Supporting

Mentioning

136

Contrasting

Order By: Relevance

“…Catastrophic forgetting appears when weights in neural networks trained on previous tasks change with learning new ones. In order to resist forgetting, these methods adopt dynamic architecture (architecture grows with new tasks) [22], memorizing the representative samples of learned tasks for rehearsal [23], [24], preventing significant changes in the representation for previous tasks [25], [26], or Bayesian methods [27], [28]. These methods alleviate catastrophic forgetting in continual learning for neural networks, however improving performance through learning multiple related tasks has not been achieved.…”

Section: Related Workmentioning

confidence: 99%

A Perpetual Learning Algorithm That Incrementally Improves Performance With Deliberation

Qin

Zhang

2020

IEEE Access

View full text Add to dashboard Cite

During recent years, several different proposals for continuous learning (lifelong learning, never-ending learning, perpetual learning) have attracted much attention from researchers in the field of machine learning. In this paper, a perpetual learning algorithm, which is augmented with a deliberation mechanism that is geared toward incrementally improving performance on all tasks learned thus far, is described. The algorithm maintains a prototype library where each prototype in the library is a model parameters vector for a representative task learned, and produces the model for a new task as a linear combination of the prototypes. The deliberation process ensues based on two criteria (difference in loss and cosine similarity between single task model and encoded model) for the newly learned task model. If the prototypes in the library represent a new task well, then the library only needs to be tuned. Otherwise, if the prototypes in the library cannot represent a new task well, then the algorithm further determines if replacing a prototype in the library makes the library better suited for all learned tasks. If so, the prototype library will be reconstructed. Compared with other existing methods, the proposed method has several salient features. Firstly, by selecting the best prototypes to form the library to represent all learned tasks, the algorithm provided in the paper has better classification and regression performance than, say, Efficient Lifelong Learning Algorithm (ELLA) in circumstances in which only the training set for the current task can be buffered. Secondly, without the need to buffer training sets for multiple tasks, the proposed algorithm shows competitive performance when compared with curriculum learning.

show abstract

Section: Related Workmentioning

confidence: 99%

A Perpetual Learning Algorithm That Incrementally Improves Performance With Deliberation

Qin

Zhang

2020

IEEE Access

View full text Add to dashboard Cite

show abstract

“…For experimental validations, we use CIFAR100 [48] as it is the most popular benchmark dataset for the IL algorithms (also called iCIFAR100) [3], [7], [9], [20], TinyIma-geNet [49] used in [50], and a subset of ImageNet (ILSVRC 2012) dataset [51] with 100 classes used in [7], [9], which we call as ImageNet-100.…”

Section: Experiments a Experimental Set-up A: Datasetsmentioning

confidence: 99%

“…For the evaluation metrics, we use average accuracy (A avg ) in all task transitions (excluding the first task's accuracy as it is not incrementally learned [9], [20]), final model's accuracy (A k , where k is the final or last task. It is denoted as 'Average' in [44]), forgetting (F k ), intransigence (I k ) and multi-head intransigence (I mh k ) metrics at task t k proposed by [3].…”

Section: D: Evaluation Metricsmentioning

confidence: 99%

“…Incremental learning (IL) is a sequential learning paradigm with limited memory resources [7], [9], [12]. There are many proposals about memory managing strategies [3], [7], [9], a better regularizing strategy [8], [15]- [19], and using an external memory with unlabeled data [20] and a method robust to the order of given tasks with task identity given [21]. We will review the ones addressing the forgetting and the sample memory management policy in detail.…”

Section: Introductionmentioning

confidence: 99%

“…Average accuracy of incrementally trained classification model at all task transitions is the most widely used evaluation metric [9], [20]. The average may not include the accuracy of the model at the first task as the model on the first task is not yet incrementally trained [9], [20]. The accuracy at the last task has also been used [3].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Confidence Calibration for Incremental Learning

Kang

Nam

et al. 2020

IEEE Access

View full text Add to dashboard Cite

Class incremental learning is an online learning paradigm wherein the classes to be recognized are gradually increased with limited memory, storing only a partial set of examples of past tasks. At a task transition, we observe an unintentional imbalance of confidence or likelihood between the classes of the past and the new task. We argue that the imbalance aggravates a catastrophic forgetting for class incremental learning. We propose a simple yet effective learning objective to balance the confidence of classes of old tasks and new task in the class incremental learning setup. In addition, we compare various sample memory configuring strategies and propose a novel sample memory management policy to alleviate the forgetting further. The proposed method outperforms the state of the arts in many evaluation metrics including accuracy and forgetting F by a large margin (up to 5.71% in A 10 and 17.1% in F 10) in extensive empirical validations on multiple visual recognition datasets such as CIFAR100, TinyImageNet and a subset of the ImageNet.

show abstract

Secure and efficient parameters aggregation protocol for federated incremental learning and its applications

Wang

Liang

Koe

et al. 2021

Int J of Intelligent Sys

View full text Add to dashboard Cite

Federated Learning (FL) enables the deployment of distributed machine learning models over the cloud and Edge Devices (EDs) while preserving the privacy of sensitive local data, such as electronic health records. However, despite FL advantages regarding security and flexibility, current constructions still suffer from some limitations. Namely, heavy computation overhead on limited resources EDs, communication overhead in uploading converged local models' parameters to a centralized server for parameters aggregation, and lack of guaranteeing the acquired knowledge preservation in the face of incremental learning over new local data sets. This paper introduces a secure and resourcefriendly protocol for parameters aggregation in federated incremental learning and its applications. In this study, the central server relies on a new method for parameters aggregation called orthogonal gradient aggregation. Such a method assumes constant changes of each local data set and allows updating parameters in the orthogonal direction of previous parameters spaces.As a result, our new construction is robust against catastrophic forgetting, maintains the federated neural network accuracy, and is efficient in computation and

show abstract

Overcoming Catastrophic Forgetting With Unlabeled Data in the Wild

Cited by 151 publications

References 32 publications

A Perpetual Learning Algorithm That Incrementally Improves Performance With Deliberation

A Perpetual Learning Algorithm That Incrementally Improves Performance With Deliberation

Confidence Calibration for Incremental Learning

Secure and efficient parameters aggregation protocol for federated incremental learning and its applications

Contact Info

Product

Resources

About