Less-Forgetful Learning for Domain Expansion in Deep Neural Networks

Jung, Heechul; Ju, Jeongwoo; Jung, Minsoo; Kim, Junmo

doi:10.1609/aaai.v32i1.11769

Cited by 50 publications

(15 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Prior-Focused • Requires explicit storage for previous tasks' models Less-Forgetting Learning (LFL) [131] Parameter-Constraint…”

Section: Parameter-isolationmentioning

confidence: 99%

A Review on Plastic Artificial Neural Networks: Exploring the Intersection between Neural Architecture Search and Continual Learning

Shahawy¹,

Benkhelifa²,

White³

2022

Preprint

View full text Add to dashboard Cite

Despite the significant advances achieved in Artificial Neural Networks (ANNs), their design process remains notoriously tedious, depending primarily on intuition, experience and trial-and-error. This human-dependent process is often time-consuming and prone to errors. Furthermore, the models are generally bound to their training contexts, with no considerations of changes to their surrounding environments. Continual adaptability and automation of neural networks is of paramount importance to several domains where model accessibility is limited after deployment (e.g IoT devices, selfdriving vehicles, etc). Additionally, even accessible models require frequent maintenance post-deployment to overcome issues such as Concept/Data Drift, which can be cumbersome and restrictive. The current state of the art on adaptive ANNs is still a premature area of research; nevertheless, Neural Architecture Search (NAS), a form of AutoML, and Continual Learning (CL) have recently gained an increasing momentum in the Deep Learning research field, aiming to provide more robust and adaptive ANN development frameworks. This study is the first extensive review on the intersection between AutoML and CL, outlining research directions for the different methods that can facilitate full automation and lifelong plasticity in ANNs.

show abstract

“…Prior-Focused • Requires explicit storage for previous tasks' models Less-Forgetting Learning (LFL) [131] Parameter-Constraint…”

Section: Parameter-isolationmentioning

confidence: 99%

A Review on Plastic Artificial Neural Networks: Exploring the Intersection between Neural Architecture Search and Continual Learning

Shahawy¹,

Benkhelifa²,

White³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…These methods can further be divided into data-focused and prior-focused methods. Knowledge distillation from a previous model to the trained model on the new data is the primary building block in datafocused methods [36], [37], [38], [39]. Prior-focused methods mitigate forgetting by estimating a distribution over the model parameters used prior when learning from new data [40], [41], [42], [43], [44], [45].…”

Section: B Current State Of the Art And Challengesmentioning

confidence: 99%

Distributed and Adaptive Edge-based AI Models for Sensor Networks (DAISeN)

Boeva¹,

Casalicchio²,

Abghari³

et al. 2022

Annals of Computer Science and Information Systems

View full text Add to dashboard Cite

This position paper describes the aims and preliminary results of the Distributed and Adaptive Edge-based AI Models for Sensor Networks (DAISeN) 1 project. The project ambition is to address today's edge AI challenges by developing advanced AI techniques that model knowledge from the sensor network and the environment to support the deployment of sustainable AI applications. We present one of the use cases being considered in DAISeN and review the state-of-the-art in three research domains related to the use case presented and directly falling into the project scope. We additionally outline the main challenges identified in each domain. The developed Global Navigation Satellite Systems (GNSS) activation model addressing the use case challenges is also briefly introduced. The future research studies planned for the remaining period of the project are finally outlined. I. INTRODUCTIONT HE NUMBER of solutions that provide Artificial Intelligence (AI) and Machine Learning (ML) based systems has been growing recently. These solutions facilitate the creation of new smart products and services in many different fields. In addition, sensor networks are undergoing great expansion and development and the integration of AI and sensor networks benefits many areas such as Industry 4.0, healthcare, mobility, logistics, and many other Internetof-Things (IoT) applications. However, this has also put new challenges in front of researchers and practitioners. New realtime AI and ML algorithms are needed along with different strategies to embed these algorithms in sensor boards and network nodes such as fog/edge nodes. For example, edgebased AI requires robust and adaptive models that take into account the temporal component of a data flow and allow for vertical and horizontal scaling of the decision-making process. These models must employ efficient learning algorithms that are capable of dealing with information varying over time and coping with large scale missing and inaccurate values. In addition, the decision-making models should be composable so that they can be distributed on the edge devices in order to This work is part of the Sony RAP 2020 Project "Distributed and Adaptive Edge-based AI Models for Sensor Networks"1 Daisen is a volcanic mountain located in Tottori Prefecture, Sanin Region of Japan. ensure a trade-off between the decision accuracy, latency, and consumed energy per decision.The IoT is an emerging key technology for future industries and the everyday lives of people, e.g., it has been playing an increasingly important role in healthcare, agriculture, home services, industrial processes, and transportation. Wireless Sensor Network (WSN) is an enabling technology for IoT [1], and, by definition, is the bridge between the physical world and the intelligence residing on the Internet. The integration of AI and sensor networks (by means of the IoT) are now realities that are changing our lives. Sensor networks are widely used to collect environmental parameters, e.g., in homes, buildings, and vehicles, where they a...

show abstract

“…Regularisation‐based methods [10, 16, 18–22] add more constraints to the cost function to maintain performance on the old data set. Typical methods include LwF [10] and elastic weight consolidation (EWC) [16].…”

Section: Related Workmentioning

confidence: 99%

Incremental learning using feature labels for synthetic aperture radar automatic target recognition

Hao

Wang

et al. 2022

IET Radar Sonar & Navi

View full text Add to dashboard Cite

Although deep neural network technology brings high recognition accuracy to the field of synthetic aperture radar image‐based automatic target recognition, it also produces the catastrophic forgetting problem. Here, a new incremental learning method that can extract more information about old data is proposed. Based on the rehearsal method, the authors’ method adds extra linear layers after the feature extractor of the network before training on new incremental data and uses the network to generate distilled labels for incremental training. Through experiments on the moving and stationary target acquisition and recognition data set, we conclude that, when the old model has good performance, our method has better performance than other typical incremental learning methods on small data sets.

show abstract

Less-Forgetful Learning for Domain Expansion in Deep Neural Networks

Cited by 50 publications

References 15 publications

A Review on Plastic Artificial Neural Networks: Exploring the Intersection between Neural Architecture Search and Continual Learning

A Review on Plastic Artificial Neural Networks: Exploring the Intersection between Neural Architecture Search and Continual Learning

Distributed and Adaptive Edge-based AI Models for Sensor Networks (DAISeN)

Incremental learning using feature labels for synthetic aperture radar automatic target recognition

Contact Info

Product

Resources

About