Addressing catastrophic forgetting for medical domain expansion

Gupta, Sharut; Singh, Praveer; Chang, Ken; Qu, Liangqiong; Aggarwal, Mehak; Arun, Nishanth; Vaswani, Ashwin; Raghavan, Shruti; Agarwal, Vibha; Gidwani, Mishka; Hoebel, Katharina; Patel, Jay; Lu, Charles; Bridge, Christopher P.; Chen, Jimmy; Chan, Robison Vernon Paul; Chiang, Michael F.; Campbell, J. Peter; Rubin, Daniel L.; Kalpathy–Cramer, Jayashree

doi:10.21203/rs.3.rs-1087025/v1

Cited by 3 publications

(1 citation statement)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…All the selected architectures are summarized in Table I along with the type of architecture and number of parameters in the network. Furthermore, we also evaluate different normalization layers, as recent work has shown that they also significantly contribute to catastrophic forgetting [34], [35]. Specifically, we exchange the BatchNorm Layers of ResNet-50 with Continual- [36], Group- [37], Instance- [38], Layer and Batch Re-Normalization layers [39] and evaluate the resulting networks in the two incremental scenarios.…”

Section: A Experimental Setupmentioning

confidence: 99%

Causes of Catastrophic Forgetting in Class-Incremental Semantic Segmentation

Kalb¹,

Beyerer

2023

Computer Vision – ACCV 2022

View full text Add to dashboard Cite

Research in the field of Continual Semantic Segmentation is mainly investigating novel learning algorithms to overcome catastrophic forgetting of neural networks. Most recent publications have focused on improving learning algorithms without distinguishing effects caused by the choice of neural architecture. Therefore, we study how the choice of neural network architecture affects catastrophic forgetting in class-and domainincremental semantic segmentation. Specifically, we compare the well-researched CNNs to recently proposed Transformers and Hybrid architectures, as well as the impact of the choice of novel normalization layers and different decoder heads. We find that traditional CNNs like ResNet have high plasticity but low stability, while transformer architectures are much more stable. When the inductive biases of CNN architectures are combined with transformers in hybrid architectures, it leads to higher plasticity and stability. The stability of these models can be explained by their ability to learn general features that are robust against distribution shifts. Experiments with different normalization layers show that Continual Normalization achieves the best trade-off in terms of adaptability and stability of the model. In the class-incremental setting, the choice of the normalization layer has much less impact. Our experiments suggest that the right choice of architecture can significantly reduce forgetting even with naive fine-tuning and confirm that for real-world applications, the architecture is an important factor in designing a continual learning model.

show abstract

Section: A Experimental Setupmentioning

confidence: 99%

Causes of Catastrophic Forgetting in Class-Incremental Semantic Segmentation

Kalb¹,

Beyerer

2023

Computer Vision – ACCV 2022

View full text Add to dashboard Cite

show abstract

Exploiting data diversity in multi-domain federated learning

Ahmad Madni,

Muhammad Umer,

Luca Foresti

2024

Mach. Learn.: Sci. Technol.

View full text Add to dashboard Cite

Federated Learning (FL) is an evolving machine learning technique that allows collaborative model training without sharing the original data among participants. In real-world scenarios, data residing at multiple clients are often heterogeneous in terms of different resolutions, magnifications, scanners, or imaging protocols, and thus challenging for global FL model convergence in collaborative training. Most of the existing FL methods consider data heterogeneity within one domain by assuming same data variation in each client site. In this paper, we consider data heterogeneity in FL with different domains of heterogeneous data by raising the problems of domain-shift, class-imbalance, and missing data. We propose a method, MDFL (Multi-Domain Federated Learning) as a solution to heterogeneous training data from multiple domains by training robust Transformer model. We use two loss functions, one for correctly predicting class labels and other for encouraging similarity and dissimilarity over latent features, to optimize the global FL model. We perform various experiments using different convolution-based networks and non-convolutional Transformer architectures on multi-domain datasets. We evaluate the proposed approach on benchmark datasets and compare with the existing FL methods. Our results show the superiority of the proposed approach which performs better in term of robust FL global model than the exiting methods.

show abstract

Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learning

Qu¹,

Zhou²,

Liang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Federated learning is an emerging research paradigm enabling collaborative training of machine learning models among different organizations while keeping data private at each institution. Despite recent progress, there remain fundamental challenges such as lack of convergence and potential for catastrophic forgetting in federated learning across real-world heterogeneous devices. In this paper, we demonstrate that attention-based architectures (e.g., Transformers) are fairly robust to distribution shifts and hence improve federated learning over heterogeneous data. Concretely, we conduct the first rigorous empirical investigation of different neural architectures across a range of federated algorithms, real-world benchmarks, and heterogeneous data splits. Our experiments show that simply replacing convolutional networks with Transformers can greatly reduce catastrophic forgetting of previous devices, accelerate convergence, and reach a better global model, especially when dealing with heterogeneous data. We will release our code and pretrained models at https://github.com/Liangqiong/ViT-FL-main to encourage future exploration in robust architectures as an alternative to current research efforts on the optimization front.

show abstract

Addressing catastrophic forgetting for medical domain expansion

Cited by 3 publications

References 36 publications

Causes of Catastrophic Forgetting in Class-Incremental Semantic Segmentation

Causes of Catastrophic Forgetting in Class-Incremental Semantic Segmentation

Exploiting data diversity in multi-domain federated learning

Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learning

Contact Info

Product

Resources

About