Catastrophic Forgetting: Still a Problem for DNNs

Pfülb, Benedikt; Gepperth, Alexander; Abdullah, Shuhairimi; Kilian, A

doi:10.1007/978-3-030-01418-6_48

Cited by 16 publications

(7 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although DL approaches have shown good results, they continue to suffer from catastrophic forgetting (Pf €ulb et al, 2018), a reduction in the overall performance when training is done on the new classes. This problem arises due to the requirement of having the complete dataset including the samples from old and new classes.…”

Section: Need For Incremental Learningmentioning

confidence: 99%

RETRACTED: Bayesian approach to incremental batch learning on forest cover sensor data for multiclass classification

D¹,

Venkataramana²,

Saraswathi³

et al. 2021

Concurrent Engineering

View full text Add to dashboard Cite

Deep neural networks can be used to perform nonlinear operations at multiple levels, such as a neural network that is composed of many hidden layers. Although deep learning approaches show good results, they have a drawback called catastrophic forgetting, which is a reduction in performance when a new class is added. Incremental learning is a learning method where existing knowledge should be retained even when new data is acquired. It involves learning with multiple batches of training data and the newer learning sessions do not require the data used in the previous iterations. The Bayesian approach to incremental learning uses the concept of the probability distribution of weights. The key idea of Bayes theorem is to find an updated distribution of weights and biases. In the Bayesian framework, the beliefs can be updated iteratively as the new data comes in. Bayesian framework allows to update the beliefs iteratively in real-time as data comes in. The Bayesian model for incremental learning showed an accuracy of 82%. The execution time for the Bayesian model was lesser on GPU (670 s) when compared to CPU (1165 s).

show abstract

Section: Need For Incremental Learningmentioning

confidence: 99%

RETRACTED: Bayesian approach to incremental batch learning on forest cover sensor data for multiclass classification

D¹,

Venkataramana²,

Saraswathi³

et al. 2021

Concurrent Engineering

View full text Add to dashboard Cite

show abstract

“…Superficially, the ResNet101 that shows superiority in both parameter scale and number of layers achieves the classification performance (mAP of 80.42%) close to that of DenseNet169 (mAP of 80.99%). However, when the scene images are assigned with the hierarchically multiple labels, the issue of data imbalance becomes prominent, which brings the problem of "catastrophic forgetting" [196], [197]. As a result, the networks show weak performance on categories like stadium and apartment When increasing τ to be 0.75, the OPs of all models gain significant improvement.…”

Section: Resultsmentioning

confidence: 99%

Aerial Scene Parsing: From Tile-level Scene Classification to Pixel-wise Semantic Labeling

Yang¹,

Xia²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Given an aerial image, aerial scene parsing (ASP) targets to interpret the semantic structure of the image content, e.g., by assigning a semantic label to every pixel of the image. With the popularization of data-driven methods, the past decades have witnessed promising progress on ASP by approaching the problem with the schemes of tile-level scene classification or segmentationbased image analysis, when using high-resolution aerial images. However, the former scheme often produces results with tile-wise boundaries, while the latter one needs to handle the complex modeling process from pixels to semantics, which often requires largescale and well-annotated image samples with pixel-wise semantic labels. In this paper, we address these issues in aerial scene parsing, with perspectives from tile-level scene classification to pixel-wise semantic labeling. Specifically, we first revisit aerial image interpretation by a literature review. We then present a large-scale scene classification dataset that contains one million aerial images termed Million-AID. With the presented dataset, we also report benchmarking experiments using classical convolutional neural networks (CNNs). Finally, we perform ASP by unifying the tile-level scene classification and object-based image analysis to achieve pixel-wise semantic labeling. Intensive experiments show that Million-AID is a challenging yet useful dataset, which can serve as a benchmark for evaluating newly developed algorithms. When transferring knowledge from Million-AID, fine-tuning CNN models pretrained on Million-AID perform consistently better than those pretrained ImageNet for aerial scene classification, demonstrating the strong generalization ability of the proposed dataset. Moreover, our designed hierarchical multi-task learning method achieves the state-of-the-art pixel-wise classification on the challenging GID, which is a profitable attempt to bridge the tile-level scene classification toward pixel-wise semantic labeling for aerial image interpretation. We hope that our work could serve as a baseline for aerial scene classification and inspire a rethinking of the scene parsing of high-resolution aerial images.

show abstract

“…Loss spikes arise when artificial neural networks (ANNs) encounter difficult examples and can destabilize training with gradient descent [1,2]. Examples may be difficult because an ANN needs more training to generalize, catastrophically forgot previous learning [3] or because an example is complex or unusual. Whatever the cause, applying gradients backpropagated [4] from high losses results in large perturbations to trainable parameters.…”

Section: Introductionmentioning

confidence: 99%

Adaptive learning rate clipping stabilizes learning

Ede

Beanland

2020

Mach. Learn.: Sci. Technol.

View full text Add to dashboard Cite

Artificial neural network training with gradient descent can be destabilized by 'bad batches' with high losses. This is often problematic for training with small batch sizes, high order loss functions or unstably high learning rates. To stabilize learning, we have developed adaptive learning rate clipping (ALRC) to limit backpropagated losses to a number of standard deviations above their running means. ALRC is designed to complement existing learning algorithms: Our algorithm is computationally inexpensive, can be applied to any loss function or batch size, is robust to hyperparameter choices and does not affect backpropagated gradient distributions. Experiments with CIFAR-10 supersampling show that ALCR decreases errors for unstable mean quartic error training while stable mean squared error training is unaffected. We also show that ALRC decreases unstable mean squared errors for scanning transmission electron microscopy supersampling and partial scan completion. Our source code is available at https://github.com/Jeffrey-Ede/ALRC.

show abstract

Catastrophic Forgetting: Still a Problem for DNNs

Cited by 16 publications

References 7 publications

RETRACTED: Bayesian approach to incremental batch learning on forest cover sensor data for multiclass classification

RETRACTED: Bayesian approach to incremental batch learning on forest cover sensor data for multiclass classification

Aerial Scene Parsing: From Tile-level Scene Classification to Pixel-wise Semantic Labeling

Adaptive learning rate clipping stabilizes learning

Contact Info

Product

Resources

About