Continual learning of context-dependent processing in neural networks

Zeng, Guangneng; Chen, Yang; Cui, Bo; Yu, Shan

doi:10.1038/s42256-019-0080-x

Cited by 184 publications

(164 citation statements)

References 51 publications

Supporting

Mentioning

159

Contrasting

Order By: Relevance

“…The OWM method was proposed by Zeng et al [ 51 ]. It protected previously learned knowledge by constraining the updated direction of the parameter weights.…”

Section: Methods Descriptionmentioning

confidence: 99%

See 1 more Smart Citation

An Appraisal of Incremental Learning Methods

Luo

Yin

Bai

et al. 2020

Entropy

View full text Add to dashboard Cite

As a special case of machine learning, incremental learning can acquire useful knowledge from incoming data continuously while it does not need to access the original data. It is expected to have the ability of memorization and it is regarded as one of the ultimate goals of artificial intelligence technology. However, incremental learning remains a long term challenge. Modern deep neural network models achieve outstanding performance on stationary data distributions with batch training. This restriction leads to catastrophic forgetting for incremental learning scenarios since the distribution of incoming data is unknown and has a highly different probability from the old data. Therefore, a model must be both plastic to acquire new knowledge and stable to consolidate existing knowledge. This review aims to draw a systematic review of the state of the art of incremental learning methods. Published reports are selected from Web of Science, IEEEXplore, and DBLP databases up to May 2020. Each paper is reviewed according to the types: architectural strategy, regularization strategy and rehearsal and pseudo-rehearsal strategy. We compare and discuss different methods. Moreover, the development trend and research focus are given. It is concluded that incremental learning is still a hot research area and will be for a long period. More attention should be paid to the exploration of both biological systems and computational models.

show abstract

“…The OWM method was proposed by Zeng et al [ 51 ]. It protected previously learned knowledge by constraining the updated direction of the parameter weights.…”

Section: Methods Descriptionmentioning

confidence: 99%

“…Image samples in this dataset are not fixed-size. ImageNet is used in [ 16 , 17 , 51 , 70 , 71 , 76 , 89 , 95 , 96 ].…”

Section: Datasetsmentioning

confidence: 99%

An Appraisal of Incremental Learning Methods

Luo

Yin

Bai

et al. 2020

Entropy

View full text Add to dashboard Cite

show abstract

“…148 It has been shown that input created from a generative model can indeed assist with learning, such as preventing catastrophic forgetting. 149 It should be noted that in most these cases the generative model exists outside the network itself, which is not biologically realistic in the case of the brain, although there are some exceptions. 150 What of those cases where the network itself acts the generative model?…”

Section: The Overfitted Brain Hypothesismentioning

confidence: 99%

The overfitted brain: Dreams evolved to assist generalization

Hoel

2021

Patterns

View full text Add to dashboard Cite

Understanding of the evolved biological function of sleep has advanced considerably in the past decade. However, no equivalent understanding of dreams has emerged. Contemporary neuroscientific theories often view dreams as epiphenomena, and many of the proposals for their biological function are contradicted by the phenomenology of dreams themselves. Now, the recent advent of deep neural networks (DNNs) has finally provided the novel conceptual framework within which to understand the evolved function of dreams. Notably, all DNNs face the issue of overfitting as they learn, which is when performance on one dataset increases but the network's performance fails to generalize (often measured by the divergence of performance on training versus testing datasets). This ubiquitous problem in DNNs is often solved by modelers via ''noise injections'' in the form of noisy or corrupted inputs. The goal of this paper is to argue that the brain faces a similar challenge of overfitting and that nightly dreams evolved to combat the brain's overfitting during its daily learning. That is, dreams are a biological mechanism for increasing generalizability via the creation of corrupted sensory inputs from stochastic activity across the hierarchy of neural structures. Sleep loss, specifically dream loss, leads to an overfitted brain that can still memorize and learn but fails to generalize appropriately. Herein this ''overfitted brain hypothesis'' is explicitly developed and then compared and contrasted with existing contemporary neuroscientific theories of dreams. Existing evidence for the hypothesis is surveyed within both neuroscience and deep learning, and a set of testable predictions is put forward that can be pursued both in vivo and in silico. ll

show abstract

“…A research team from the Institute of Automation, Chinese Academy of Sciences, recently used the idea of orthogonal weights to fully protect the previously acquired information when gradient optimization is performed on deep networks. The team demonstrated a good lifelong learning ability in applications such as handwriting and face recognitions [134]. However, most of the current studies are still limited to single perception modality.…”

Section: Embodied Tactile Learningmentioning

confidence: 99%

Embodied tactile perception and learning

Liu

Guo

Sun

et al. 2020

Brain Science Advances

View full text Add to dashboard Cite

Various living creatures exhibit embodiment intelligence, which is reflected by a collaborative interaction of the brain, body, and environment. The actual behavior of embodiment intelligence is generated by a continuous and dynamic interaction between a subject and the environment through information perception and physical manipulation. The physical interaction between a robot and the environment is the basis for realizing embodied perception and learning. Tactile information plays a critical role in this physical interaction process. It can be used to ensure safety, stability, and compliance, and can provide unique information that is difficult to capture using other perception modalities. However, due to the limitations of existing sensors and perception and learning methods, the development of robotic tactile research lags significantly behind other sensing modalities, such as vision and hearing, thereby seriously restricting the development of robotic embodiment intelligence. This paper presents the current challenges related to robotic tactile embodiment intelligence and reviews the theory and methods of robotic embodied tactile intelligence. Tactile perception and learning methods for embodiment intelligence can be designed based on the development of new large‐scale tactile array sensing devices, with the aim to make breakthroughs in the neuromorphic computing technology of tactile intelligence.

show abstract

Continual learning of context-dependent processing in neural networks

Cited by 184 publications

References 51 publications

An Appraisal of Incremental Learning Methods

An Appraisal of Incremental Learning Methods

The overfitted brain: Dreams evolved to assist generalization

Embodied tactile perception and learning

Contact Info

Product

Resources

About