Deep Class-Incremental Learning From Decentralized Data

Zhang, Xiaohan; Dong, Songlin; Chen, Jinjie; Tian, Qi; Gong, Yihong; Hong, Xiaopeng

doi:10.1109/tnnls.2022.3214573

Cited by 7 publications

(3 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Rehearsal-based methods tackle catastrophic forgetting either by keeping a small set of old training examples in memory (Tao et al 2020a;Dong et al 2021;Liu et al 2022;Yang et al 2022a,b) or using synthesized data produced by generative models (Shin et al 2017). By using the rehearsal buffer for knowledge distillation and regularization, rehearsal-based methods have achieved state-of-theart results on various benchmarks (Douillard et al 2022;Joseph et al 2022;Zhang et al 2022). However, the performance of rehearsal-based methods generally deteriorates with a smaller buffer size (Mai et al 2022).…”

Section: Related Workmentioning

confidence: 99%

Isolation and Impartial Aggregation: A Paradigm of Incremental Learning without Interference

Wang

Huang

et al. 2023

AAAI

View full text Add to dashboard Cite

This paper focuses on the prevalent stage interference and stage performance imbalance of incremental learning. To avoid obvious stage learning bottlenecks, we propose a new incremental learning framework, which leverages a series of stage-isolated classifiers to perform the learning task at each stage, without interference from others. To be concrete, to aggregate multiple stage classifiers as a uniform one impartially, we first introduce a temperature-controlled energy metric for indicating the confidence score levels of the stage classifiers. We then propose an anchor-based energy self-normalization strategy to ensure the stage classifiers work at the same energy level. Finally, we design a voting-based inference augmentation strategy for robust inference. The proposed method is rehearsal-free and can work for almost all incremental learning scenarios. We evaluate the proposed method on four large datasets. Extensive results demonstrate the superiority of the proposed method in setting up new state-of-the-art overall performance. Code is available at https://github.com/iamwangyabin/ESN.

show abstract

Section: Related Workmentioning

confidence: 99%

Isolation and Impartial Aggregation: A Paradigm of Incremental Learning without Interference

Wang

Huang

et al. 2023

AAAI

View full text Add to dashboard Cite

show abstract

“…Therefore, simply performing element-wise averaging of model parameters, as most FL algorithms adopt, would not produce an ideal global model to serve all clients 8 . In addition, some related schemes have been used in federated learning, e.g., data poisoning attack 9 and federated class-incremental learning 10 , 11 improve on the original federation averaging algorithm, respectively. But they mainly address the problem of model training in specific federated learning scenarios.…”

Section: Introductionmentioning

confidence: 99%

Global prototype distillation for heterogeneous federated learning

Wu,

Chen,

Nie

et al. 2024

Sci Rep

View full text Add to dashboard Cite

Federated learning is a distributed machine learning paradigm where the goal is to collaboratively train a high quality global model while private training data remains local over distributed clients. However, heterogenous data distribution over clients is severely challenging for federated learning system, which severely damage the quality of model. In order to address this challenge, we propose global prototype distillation (FedGPD) for heterogenous federated learning to improve performance of global model. The intuition is to use global class prototypes as knowledge to instruct local training on client side. Eventually, local objectives will be consistent with the global optima so that FedGPD learns an improved global model. Experiments show that FedGPD outperforms previous state-of-art methods by 0.22% ~1.28% in terms of average accuracy on representative benchmark datasets.

show abstract

“…I N the recent years, deep learning has achieved remarkable success in a wide range of applications, including image classification [1], [2], speech recognition [3], [4] and machine translation [5], [6]. These high-capacity models benefit from capturing capture complex patterns of the underlying data distribution.…”

Section: Introductionmentioning

confidence: 99%

Learning From Crowds With Contrastive Representation

Pedrycz

2023

IEEE Access

View full text Add to dashboard Cite

Crowdsourcing provides a practical approach to obtaining annotated data for data-hungry deep models. Due to its simplicity and practicality, simultaneously learning the annotation correction mechanism and the target classifier is widely studied and applied. Existing work has improved performance from the annotator and annotation process modeling perspective. However, the instance representation, which most directly affects model training, has been neglected. In this work, we investigate contrastive representation to improve learning from crowds. Specifically, we first sample confident instances and positive pairs using the pre-trained representation and human annotations. Then, we extend the supervised contrastive loss to obtain a noise-tolerant version that supports continuous consistency between labels. After that, we leverage the learned representations to train the classifier and annotator parameters. The process is generally designed as an end-to-end framework, CrowdCons, compatible with existing crowdsourcing models. Our approach is evaluated on three real-world crowdsourcing datasets; LabelMe, CIFAR10-H and Music. The experimental results show that it can significantly improve prediction accuracy, and the case study demonstrates the robustness of the model regarding noisy annotations.

show abstract

Deep Class-Incremental Learning From Decentralized Data

Cited by 7 publications

References 47 publications

Isolation and Impartial Aggregation: A Paradigm of Incremental Learning without Interference

Isolation and Impartial Aggregation: A Paradigm of Incremental Learning without Interference

Global prototype distillation for heterogeneous federated learning

Learning From Crowds With Contrastive Representation

Contact Info

Product

Resources

About