Ensemble Attention Distillation for Privacy-Preserving Federated Learning

Gong, Xuan; Sharma, Abhishek; Karanam, Srikrishna; Wu, Ziyan; Chen, Terrence; Doermann, David; Innanje, Arun

doi:10.1109/iccv48922.2021.01480

Cited by 73 publications

(37 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, FedDG [13] enhances the generalization ability of the FL framework on the unseen datasets via Fourier transform-based image synthesis and episodic learning strategies. Furthermore, [40] proposed a distillation-based FL method without sharing the model parameters, which further enhances data safety. Although these methods are effective in many medical imaging scenarios, they have not considered the weighting strategies for the global aggregation and local training, which is crucial for FL MS segmentation.…”

Section: B Federated Learningmentioning

confidence: 99%

MS Lesion Segmentation: Revisiting Weighting Mechanisms for Federated Learning

Liu¹,

Cabezas²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Federated learning (FL) has been widely employed for medical image analysis to facilitate multi-client collaborative learning without sharing raw data. Despite great success, FL's performance is limited for multiple sclerosis (MS) lesion segmentation tasks, due to variance in lesion characteristics imparted by different scanners and acquisition parameters. In this work, we propose the first FL MS lesion segmentation framework via two effective re-weighting mechanisms. Specifically, a learnable weight is assigned to each local node during the aggregation process, based on its segmentation performance. In addition, the segmentation loss function in each client is also re-weighted according to the lesion volume for the data during training. Comparison experiments on two FL MS segmentation scenarios using public and clinical datasets have demonstrated the effectiveness of the proposed method by outperforming other FL methods significantly. Furthermore, the segmentation performance of FL incorporating our proposed aggregation mechanism can exceed centralised training with all the raw data. The extensive evaluation also indicated the superiority of our method when estimating brain volume differences estimation after lesion inpainting.

show abstract

Section: B Federated Learningmentioning

confidence: 99%

MS Lesion Segmentation: Revisiting Weighting Mechanisms for Federated Learning

Liu¹,

Cabezas²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…This simple paradigm suffers from performance degradation when there exists data heterogeneity [20,25]. Numerous studies have been conducted for label space heterogeneity, i.e., class distributions are imbalanced across different clients, by regularizing local update with proximal term [26], personalizing client models [2,8,37,27], utilizing shared local data [44,30,10], introducing additional proxy datasets [24,29,11], or performing data-free knowledge distillation [32] in the input space [13,42,43] or the feature space [15,48]. However, there are only limited studies addressing the heterogeneity in feature space, i.e., non-IID features.…”

Section: Related Workmentioning

confidence: 99%

FRAug: Tackling Federated Learning with Non-IID Features via Representation Augmentation

Chen¹,

Frikha²,

Krompaß³

et al. 2022

Preprint

View full text Add to dashboard Cite

Federated Learning (FL) is a decentralized learning paradigm in which multiple clients collaboratively train deep learning models without centralizing their local data and hence preserve data privacy. Real-world applications usually involve a distribution shift across the datasets of the different clients, which hurts the generalization ability of the clients to unseen samples from their respective data distributions. In this work, we address the recently proposed feature shift problem where the clients have different feature distributions while the label distribution is the same. We propose Federated Representation Augmentation (FRAug) to tackle this practical and challenging problem. Our approach generates synthetic client-specific samples in the embedding space to augment the usually small client datasets. For that, we train a shared generative model to fuse the clients' knowledge, learned from different feature distributions, to synthesize client-agnostic embeddings, which are then locally transformed into client-specific embeddings by Representation Transformation Networks (RTNets). By transferring knowledge across the clients, the generated embeddings act as a regularizer for the client models and reduce overfitting to the local original datasets, hence improving generalization. Our empirical evaluation on multiple benchmark datasets demonstrates the effectiveness of the proposed method, which substantially outperforms the current state-of-the-art FL methods for non-IID features, including PartialFed and FedBN.

show abstract

“…Numerous research papers have addressed data heterogeneity (i.e. non-IID data among local clients) in FL [1,7,13,23,31,39,41], such as improve client sampling fairness [27], adaptive optimization [9,28,37,38], and correct the local updation [16,20,33]. Also, federated learning had been extended in real life applications [8,24].…”

Section: Federated Learningmentioning

confidence: 99%

SPATL: Salient Parameter Aggregation and Transfer Learning for Heterogeneous Clients in Federated Learning

Yu¹,

Nguyen²,

Abebe³

et al. 2021

Preprint

View full text Add to dashboard Cite

Efficient federated learning is one of the key challenges for training and deploying AI models on edge devices. However, maintaining data privacy in federated learning raises several challenges including data heterogeneity, expensive communication cost, and limited resources. In this paper, we address the above issues by (a) introducing a salient parameter selection agent based on deep reinforcement learning on local clients, and aggregating the selected salient parameters on the central server, and (b) splitting a normal deep learning model (e.g., CNNs) as a shared encoder and a local predictor, and training the shared encoder through federated learning while transferring its knowledge to Non-IID clients by the local customized predictor. The proposed method (a) significantly reduces the communication overhead of federated learning and accelerates the model inference, while method (b) addresses the data heterogeneity issue in federated learning. Additionally, we leverage the gradient control mechanism to correct the gradient heterogeneity among clients. This makes the training process more stable and converge faster. The experiments show our approach yields a stable training process and achieves notable results compared with the state-of-the-art methods. Our approach significantly reduces the communication cost by up to 108 GB when training VGG-11, and needed 7.6× less communication overhead when training ResNet-20, while accelerating the local inference by reducing up to 39.7% FLOPs on VGG-11.

show abstract

Ensemble Attention Distillation for Privacy-Preserving Federated Learning

Cited by 73 publications

References 30 publications

MS Lesion Segmentation: Revisiting Weighting Mechanisms for Federated Learning

MS Lesion Segmentation: Revisiting Weighting Mechanisms for Federated Learning

FRAug: Tackling Federated Learning with Non-IID Features via Representation Augmentation

SPATL: Salient Parameter Aggregation and Transfer Learning for Heterogeneous Clients in Federated Learning

Contact Info

Product

Resources

About