Memory-Efficient Semi-Supervised Continual Learning: The World is its Own Replay Buffer

Smith, James Seale; Balloch, Jonathan; Hsu, Yen-Chang; Kira, Zsolt

doi:10.1109/ijcnn52387.2021.9534361

Cited by 14 publications

(10 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Some variants of supervised CL exist that alleviate the need for annotated data. The reduction of annotations can apply for restricted access to the task labels as in task-agnostic CL [48,49], or by reducing the labels' availability as in continual active learning [50,51], or in semi-supervised continual learning, [52]. As described in Sec.…”

Section: Towards a Generalization Of The Default Settingmentioning

confidence: 99%

Tutorial - Continual Learning beyond classification

Gepperth¹,

Lesort²

2022

ESANN 2022 Proceedings

View full text Add to dashboard Cite

Continual Learning (CL, sometimes also termed incremental learning) is a flavor of machine learning where the usual assumption of stationary data distribution is relaxed or omitted. When naively applying, e.g., DNNs in CL problems, changes in the data distribution can cause the so-called catastrophic forgetting (CF) effect: an abrupt loss of previous knowledge. Although many significant contributions to enabling CL have been made in recent years, most works address supervised (classification) problems. This article reviews literature that study CL in other settings, such as learning with reduced supervision, fully unsupervised learning, and reinforcement learning. Besides proposing a simple schema for classifying CL approaches w.r.t. their level of autonomy and supervision, we discuss the specific challenges associated with each setting and the potential contributions to the field of CL in general. Framework and Goals of Continual LearningContinual learning (CL) is a machine learning sub-field that studies learning under time-varying data distributions. This relaxes one of the fundamental assumptions of statistical learning theory ([1]), which states that the data follows 67

show abstract

Section: Towards a Generalization Of The Default Settingmentioning

confidence: 99%

Tutorial - Continual Learning beyond classification

Gepperth¹,

Lesort²

2022

ESANN 2022 Proceedings

View full text Add to dashboard Cite

show abstract

“…CL with missing labels. While most CL research focuses on the supervised setting, recent works consider CL where few or no labels are available [17,18,19]. However, none of them deals with the OGCL setting except the STAM architecture [20], where the authors present an online clustering suited for unsupervised OGCL.…”

Section: Continual Learningmentioning

confidence: 99%

“…Even though other semi-supervised CL methods exist, none respects the OSSGCL setting and thus cannot be used in the comparison [14,18,26].…”

Section: Baselinesmentioning

confidence: 99%

Contrastive Learning for Online Semi-Supervised General Continual Learning

Marilleau

Negrel

Chierchia

et al. 2022

2022 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

We study Online Continual Learning with missing labels and propose SemiCon, a new contrastive loss designed for partly labeled data. We demonstrate its efficiency by devising a memory-based method trained on an unlabeled data stream, where every data added to memory is labeled using an oracle. Our approach outperforms existing semi-supervised methods when few labels are available, and obtain similar results to state-of-the-art supervised methods while using only 2.6% of labels on Split-CIFAR10 and 10% of labels on Split-CIFAR100.

show abstract

“…However, recent works challenge some of those common assumptions such as fixed label [6,2], uncorrelated data stream [5,42], labeled new data [16], or even no metadata about tasks and unavailability of information about the future [11].…”

Section: Questioning Our Assumptionsmentioning

confidence: 99%

“…As well-annotated data eases the training on a task, the definition of tasks and the continual evaluation can be more clearly investigated. Although note that, while the most common approach in continual image classification is based on full supervision, semi-supervised approaches also exist [42].…”

Section: Image Classificationmentioning

confidence: 99%

Continuum: Simple Management of Complex Continual Learning Scenarios

Douillard,

Lesort

2021

Preprint

View full text Add to dashboard Cite

Continual learning is a machine learning sub-field specialized in settings with non-iid data. Hence, the training data distribution is not static and drifts through time. Those drifts might cause interferences in the trained model and knowledge learned on previous states of the data distribution might be forgotten. Continual learning's challenge is to create algorithms able to learn an ever-growing amount of knowledge while dealing with data distribution drifts. One implementation difficulty in these field is to create data loaders that simulate non-iid scenarios. Indeed, data loaders are a key component for continual algorithms. They should be carefully designed and reproducible. Small errors in data loaders have a critical impact on algorithm results, e.g. with bad preprocessing, wrong order of data or bad test set. Continuum is a simple and efficient framework with numerous data loaders that avoid researcher to spend time on designing data loader and eliminate time-consuming errors. Using our proposed framework, it is possible to directly focus on the model design by using the multiple scenarios and evaluation metrics implemented. Furthermore the framework is easily extendable to add novel settings for specific needs.

show abstract

Memory-Efficient Semi-Supervised Continual Learning: The World is its Own Replay Buffer

Cited by 14 publications

References 6 publications

Tutorial - Continual Learning beyond classification

Tutorial - Continual Learning beyond classification

Contrastive Learning for Online Semi-Supervised General Continual Learning

Continuum: Simple Management of Complex Continual Learning Scenarios

Contact Info

Product

Resources

About