FedBKD: Heterogenous Federated Learning via Bidirectional Knowledge Distillation for Modulation Classification in IoT-Edge System

Qi, Peihan; Zhou, Xiaoyu; Ding, Yuanlei; Zhang, Zhengyu; Zheng, Shilian; Li, Zan

doi:10.1109/jstsp.2022.3224597

Cited by 19 publications

(12 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently serval works [23,24,33,34] has been proposed to implement data-free knowledge distillation for FL. FedGKD [34] prevents the local model drift by guiding local model training through knowledge distillation between historical global and local models.…”

Section: Knowledge Distillation In Federated Learningmentioning

confidence: 99%

Data-Free Knowledge Distillation via Generator-Free Data Generation for Non-IID Federated Learning

Zhao,

Liao,

et al. 2023

Preprint

View full text Add to dashboard Cite

Data heterogeneity (Non-IID) on Federated Learning (FL) is currently a widely publicized problem, which leads to local model drift and performance degradation. Because of the advantage of knowledge distillation, it has been explored in some recent work to refine global models. However, these approaches rely on a proxy dataset or a data generator. First, in many FL scenarios, proxy dataset do not necessarily exist on the server. Second, the quality of data generated by the generator is unstable and the generator depends on the computing resources of the server. In this work, we propose a novel data-Free knowledge distillation approach via generator-Free Data Generation for Non-IID FL, dubbed as FedF2DG. Specifically, FedF2DG requires only local models to generate pseudo datasets for each client, and can generate hard samples by adding an additional regularization term that exploit disagreements between local model and global model. Meanwhile, FedF2DG enables flexible utilization of computational resources by generating pseudo dataset locally or on the server. And to address the label distribution shift in Non-IID FL, we propose a Data Generation Principle that can adaptively control the label distribution and number of pseudo dataset based on client current state, and this allows for the extraction of more client knowledge. Then knowledge distillation is performed to transfer the knowledge in local models to the global model. Extensive experiments demonstrate that our proposed method significantly outperforms the state-of-the-art FL methods and can serve as plugin for existing Federated Learning methds such as FedAvg, FedProx, etc, and improve their performance.

show abstract

Section: Knowledge Distillation In Federated Learningmentioning

confidence: 99%

Data-Free Knowledge Distillation via Generator-Free Data Generation for Non-IID Federated Learning

Zhao,

Liao,

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Recent trends suggest that KD has great potential to apply to various learning processes in FEL as an important tool for knowledge transfer or model collaborative training in diversity-constrained mobile edge networks. Specifically, the technical characteristics of KD meet the core demands of FEL, and the roles it can play include but not limited to compressing large-scale edge models for on-device deployment [25], transferring local adaptive knowledge to on-device models for personalization [39,17,44], and helping establish novel FL frameworks for enabling heterogeneous device supports [30,43,41]. Representative works that apply KD in FEL are summarized in Table 1, 2.…”

Section: Why Concern Knowledge Distillation In Federated Edgementioning

confidence: 99%

“…Edge-end Collaboration FD [14] FedGEMS [6] Limited Computation FedGKT [10] FedBKD [30] Heterogeneous Computation CMFD [34] Decentralization…”

Section: Model Representationmentioning

confidence: 99%

“…Yu [41] adopts a variant of co-distillation [3] at the network termination for collaboratively optimizing on-device models and the downloaded edge model, while integrating the knowledge from heterogeneous devices via ensemble KD in the edge. In addition, Qi [30] employs KD after on-device model uploading and edge model downloading, and uses a conditional variational autoencoder to complement both the required public data and private data. It is also worth noting that previously mentioned approaches taking KD as the end-edge model representation exchange protocol [10,6,7,39,13,14] also support heterogeneous on-device models, which are partly applicable on heterogeneous devices as well.…”

Section: Heterogeneous Resourcesmentioning

confidence: 99%

“…Recent trends suggest the great potential to address the above challenges in the context of FEL. Previous works integrating KD into the training process of FEL have been successful in tackling constrained device resources [13,14,31,32,6,10,7], adapting to heterogeneous devices and user requirements [39,17,44,43,41,30,25], and adapting to complex communication channels as well as network topologies [1,2,28,19,34]. Therefore, a survey is urgently needed to review how KD applies to FEL.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Survey of Knowledge Distillation in Federated Edge Learning

Wu¹,

Sun²,

Wang³

et al. 2023

Preprint

View full text Add to dashboard Cite

The increasing demand for intelligent services and privacy protection of mobile and Internet of Things (IoT) devices motivates the wide application of Federated Edge Learning (FEL), in which devices collaboratively train on-device Machine Learning (ML) models without sharing their private data. Limited by device hardware, diverse user behaviors and network infrastructure, the algorithm design of FEL faces challenges related to resources, personalization and network environments. Fortunately, Knowledge Distillation (KD) has been leveraged as an important technique to tackle the above challenges in FEL. In this paper, we investigate the works that KD applies to FEL, discuss the limitations and open problems of existing KD-based FEL approaches, and provide guidance for their real deployment.

show abstract

Light-M: An efficient lightweight medical image segmentation framework for resource-constrained IoMT

Zhang,

Chen,

Yang

2024

Computers in Biology and Medicine

View full text Add to dashboard Cite

FedBKD: Heterogenous Federated Learning via Bidirectional Knowledge Distillation for Modulation Classification in IoT-Edge System

Cited by 19 publications

References 40 publications

Data-Free Knowledge Distillation via Generator-Free Data Generation for Non-IID Federated Learning

Data-Free Knowledge Distillation via Generator-Free Data Generation for Non-IID Federated Learning

Survey of Knowledge Distillation in Federated Edge Learning

Light-M: An efficient lightweight medical image segmentation framework for resource-constrained IoMT

Contact Info

Product

Resources

About