Online Continual Learning via the Meta-learning update with Multi-scale Knowledge Distillation and Data Augmentation

Han, Yanan; Liu, Jianwei

doi:10.1016/j.engappai.2022.104966

Cited by 3 publications

(1 citation statement)

References 55 publications

(72 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…last fully-connected layer). In order to mitigate this drawback, a scheme of multi-scale knowledge distillation is adopted inspired by the work of [14]. More specifically, apart from the model's output each client transmits some intermediate outputs (i.e.…”

Section: Local Supervision Representation Learningmentioning

confidence: 99%

FedRCIL: Federated Knowledge Distillation for Representation based Contrastive Incremental Learning

Psaltis,

Chatzikonstantinou,

Patrikakis

et al. 2023

2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)

View full text Add to dashboard Cite

The present work proposes a holistic approach to address catastrophic forgetting in the field of computer vision during the process of incremental learning. More specifically, it suggests a series of steps for effective learning of models in distributed environments, based on extracting meaningful representations, modeling them into actual knowledge, and transferring it through a continual distillation mechanism. Additionally, it introduces a federated learning algorithm tailored to the problem, eliminating the need for central model transfer, by proposing an approach based on multi-scale representation learning, coupled with a Knowledge Distillation technique. Finally, inspired by the current trend, it modifies a contrastive learning technique combining existing knowledge with previous states, aiming to preserve previously learned knowledge while incorporating new knowledge. Thorough experimentation has been conducted to provide a comprehensive analysis of the issue at hand, highlighting the great potential of the proposed method, achieving great results in a federated environment with reduced communication cost and a robust performance within highly distributed incremental scenarios.

show abstract

Section: Local Supervision Representation Learningmentioning

confidence: 99%