Sleep-like unsupervised replay reduces catastrophic forgetting in artificial neural networks

Tadros, Timothy; Ramyaa, Ramyaa; Bazhenov, Maxim

doi:10.1038/s41467-022-34938-7

Cited by 12 publications

(5 citation statements)

References 73 publications

(101 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An artificial neural network consists of artificial neurons, which are actually processing elements. Each neuron has several inputs and it assigned a weight to each input 24 , 25 . The average output of each neuron is obtained from the sum of all inputs multiplied by the weights.…”

Section: Methodsmentioning

confidence: 99%

Data driven of underground water level using artificial intelligence hybrid algorithms

Rahimi

Ebrahimi

2023

Sci Rep

View full text Add to dashboard Cite

As the population grows, industry and agriculture have also developed and water resources require quantitative and qualitative management. Currently, the management of water resources is essential in the exploitation and development of these resources. For this reason, it is important to study water level fluctuations to check the amount of underground water storage. It is vital to study the level of underground water in Khuzestan province with a dry climate. The methods which exist for predicting and managing water resources are used in studies according to their strengths and weaknesses and according to the conditions. In recent years, artificial intelligence has been used extensively for groundwater resources worldwide. Since artificial intelligence models have provided good results in water resources up to now, in this study, the hybrid model of three new recombined methods including FF-KNN, ABC-KNN and DL-FF-KNN-ABC-MLP has been used to predict the underground water level in Khuzestan province (Qale-Tol area). The novelty of this technique is that it first does classification by presenting the first block (combination of FF-DWKNN algorithm) and predicts with the second block (combination of ABC-MLP algorithm). The algorithm’s ability to decrease data noise will be enabled by this feature. In order to predict this key and important parameter, a part of the data related to wells 1–5 has been used to build artificial intelligence hybrid models and also to test these models, and to check this model three wells 6–8 have been used for the development of these models. After checking the results, it is clear that the statistical RMSE values of this algorithm including test, train and total data are 0.0451, 0.0597 and 0.0701, respectively. According to the results presented in the table reports, the performance accuracy of DL-FF-KNN-ABC-MLP for predicting this key parameter is very high.

show abstract

Section: Methodsmentioning

confidence: 99%

Data driven of underground water level using artificial intelligence hybrid algorithms

Rahimi

Ebrahimi

2023

Sci Rep

View full text Add to dashboard Cite

show abstract

“…(Hammouamri, Masquelier, and Wilson 2022) achieves continual learning in SNNs by training an external network using evolutionary strategies to generate the firing threshold of classifer. (Tadros et al 2022) implements the local plasticity to help the model to correct bias after the learning process of new tasks by using a conversion algorithm to switch between rate-coding and spike-coding. In addition, ANN-oriented methods, such as (Bricken et al 2023) and (Shen, Dasgupta, and Navlakha 2021), have investigated the mechanisms similar to the selective activated Top-K function and provided efficient solutions for continual learning with ANNs model.…”

Section: Related Workmentioning

confidence: 99%

Efficient Spiking Neural Networks with Sparse Selective Activation for Continual Learning

Shen,

Ni,

et al. 2024

AAAI

View full text Add to dashboard Cite

The next generation of machine intelligence requires the capability of continual learning to acquire new knowledge without forgetting the old one while conserving limited computing resources. Spiking neural networks (SNNs), compared to artificial neural networks (ANNs), have more characteristics that align with biological neurons, which may be helpful as a potential gating function for knowledge maintenance in neural networks. Inspired by the selective sparse activation principle of context gating in biological systems, we present a novel SNN model with selective activation to achieve continual learning. The trace-based K-Winner-Take-All (K-WTA) and variable threshold components are designed to form the sparsity in selective activation in spatial and temporal dimensions of spiking neurons, which promotes the subpopulation of neuron activation to perform specific tasks. As a result, continual learning can be maintained by routing different tasks via different populations of neurons in the network. The experiments are conducted on MNIST and CIFAR10 datasets under the class incremental setting. The results show that the proposed SNN model achieves competitive performance similar to and even surpasses the other regularization-based methods deployed under traditional ANNs.

show abstract

“…A fully-connected ANN with two hidden layers was first trained on a randomly selected subset of MNIST or Fashion MNIST (FMNIST) datasets using backpropagation. Subsequently, the sleep replay consolidation (SRC) algorithm was implemented as previously described in (Tadros et al 2022). Briefly (see Supplementary Material for details), the ANN trained by limited data was mapped to a spiking neural network (SNN) with the same architecture.…”

Section: Algorithmmentioning

confidence: 99%

“…After the sleep phase, the SNN was remapped back to an ANN. In (Tadros et al 2022) SRC was applied after each new task training to avoid catastrophic forgetting, here we applied it once after training with limited data.…”

Section: Algorithmmentioning

confidence: 99%

Sleep-Like Unsupervised Replay Improves Performance When Data Are Limited or Unbalanced (Student Abstract)

Bazhenov,

Dewasurendra,

Krishnan

et al. 2024

AAAI

View full text Add to dashboard Cite

The performance of artificial neural networks (ANNs) degrades when training data are limited or imbalanced. In contrast, the human brain can learn quickly from just a few examples. Here, we investigated the role of sleep in improving the performance of ANNs trained with limited data on the MNIST and Fashion MNIST datasets. Sleep was implemented as an unsupervised phase with local Hebbian type learning rules. We found a significant boost in accuracy after the sleep phase for models trained with limited data in the range of 0.5-10% of total MNIST or Fashion MNIST datasets. When more than 10% of the total data was used, sleep alone had a slight negative impact on performance, but this was remedied by fine-tuning on the original data. This study sheds light on a potential synaptic weight dynamics strategy employed by the brain during sleep to enhance memory performance when training data are limited or imbalanced.

show abstract

Sleep-like unsupervised replay reduces catastrophic forgetting in artificial neural networks

Cited by 12 publications

References 73 publications

Data driven of underground water level using artificial intelligence hybrid algorithms

Data driven of underground water level using artificial intelligence hybrid algorithms

Efficient Spiking Neural Networks with Sparse Selective Activation for Continual Learning

Sleep-Like Unsupervised Replay Improves Performance When Data Are Limited or Unbalanced (Student Abstract)

Contact Info

Product

Resources

About