FedHe: Heterogeneous Models and Communication-Efficient Federated Learning

Chan, Yun Hin; Ngai, Edith C.-H.

doi:10.1109/msn53354.2021.00043

Cited by 14 publications

(9 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The client selection scheme is used to select the clients that can contribute more to enhance the global model, which results in a reduction in the communication rounds [147,185,191]. Using asynchronous communication can enhance the global model performance by allowing the aggregation of the received model without waiting for all clients [146,171,211]. Select model update is a technique that uploads the trained model that can help model coverage and ignore irrelevant updates [157,189].…”

Section: Discussionmentioning

confidence: 99%

“…Quantization [142][143][144][145]148,152,156,158,168,169,174,176,183,184,188,[191][192][193]198,199] Sparsification [140,141,151,153,155,165,174,186,200,202,204] Client Selection [147,166,172,185,191,198,207] Asynchronous [146,171,190,203,211] Two-Level Aggregation [164,175,180,182,185] Select Model Updates [149,157,…”

Section: Techniques Studies Referencedmentioning

confidence: 99%

“…CNN MNIST [146,149,152,154,165,[169][170][171]180,182,185,189,192,202,203,[208][209][210]212] Cifar -10 [140,146,167,171,176,184,188-191,204,210] FMNIST [143,152,[167][168][169]190,203,212] FEMNIST [140,142,167,185] EMNIST [150,169,203,210] Cifar-100 [190] ResNet Cifar-10 [143,144,148,159,161,168,174,179,180,182,[184][185][186]…”

Section: Model Dataset Studies Referencedmentioning

confidence: 99%

See 2 more Smart Citations

Communication Efficiency and Non-Independent and Identically Distributed Data Challenge in Federated Learning: A Systematic Mapping Study

Alotaibi,

Khan,

Mahmood

2024

Applied Sciences

View full text Add to dashboard Cite

Federated learning has emerged as a promising approach for collaborative model training across distributed devices. Federated learning faces challenges such as Non-Independent and Identically Distributed (non-IID) data and communication challenges. This study aims to provide in-depth knowledge in the federated learning environment by identifying the most used techniques for overcoming non-IID data challenges and techniques that provide communication-efficient solutions in federated learning. The study highlights the most used non-IID data types, learning models, and datasets in federated learning. A systematic mapping study was performed using six digital libraries, and 193 studies were identified and analyzed after the inclusion and exclusion criteria were applied. We identified that enhancing the aggregation method and clustering are the most widely used techniques for non-IID data problems (used in 18% and 16% of the selected studies), and a quantization technique was the most common technique in studies that provide communication-efficient solutions in federated learning (used in 27% and 15% of the selected studies). Additionally, our work shows that label distribution skew is the most used case to simulate a non-IID environment, specifically, the quantity label imbalance. The supervised learning model CNN model is the most commonly used learning model, and the image datasets MNIST and Cifar-10 are the most widely used datasets when evaluating the proposed approaches. Furthermore, we believe the research community needs to consider the client’s limited resources and the importance of their updates when addressing non-IID and communication challenges to prevent the loss of valuable and unique information. The outcome of this systematic study will benefit federated learning users, researchers, and providers.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Techniques Studies Referencedmentioning

confidence: 99%

Section: Model Dataset Studies Referencedmentioning

confidence: 99%

See 1 more Smart Citation

Communication Efficiency and Non-Independent and Identically Distributed Data Challenge in Federated Learning: A Systematic Mapping Study

Alotaibi,

Khan,

Mahmood

2024

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…FedHe [ 33 ]: This method does not require a public dataset. It also uses the average logit information of each category as a loss to assist client model training.…”

Section: Methodsmentioning

confidence: 99%

FedTKD: A Trustworthy Heterogeneous Federated Learning Based on Adaptive Knowledge Distillation

Chen,

Zhang,

Dong

et al. 2024

Entropy

View full text Add to dashboard Cite

Federated learning allows multiple parties to train models while jointly protecting user privacy. However, traditional federated learning requires each client to have the same model structure to fuse the global model. In real-world scenarios, each client may need to develop personalized models based on its environment, making it difficult to perform federated learning in a heterogeneous model environment. Some knowledge distillation methods address the problem of heterogeneous model fusion to some extent. However, these methods assume that each client is trustworthy. Some clients may produce malicious or low-quality knowledge, making it difficult to aggregate trustworthy knowledge in a heterogeneous environment. To address these challenges, we propose a trustworthy heterogeneous federated learning framework (FedTKD) to achieve client identification and trustworthy knowledge fusion. Firstly, we propose a malicious client identification method based on client logit features, which can exclude malicious information in fusing global logit. Then, we propose a selectivity knowledge fusion method to achieve high-quality global logit computation. Additionally, we propose an adaptive knowledge distillation method to improve the accuracy of knowledge transfer from the server side to the client side. Finally, we design different attack and data distribution scenarios to validate our method. The experiment shows that our method outperforms the baseline methods, showing stable performance in all attack scenarios and achieving an accuracy improvement of 2% to 3% in different data distributions.

show abstract

“…To address this, the transfer of single per-class representations is discussed. Hin and Edith present FedHe [39], which, similar to previous approaches, allows for different model architectures per device. Contrary to FedMD or Cronus, FedHe does not use a public dataset for distillation.…”

Section: Mixture Of Distillation and Fedavgmentioning

confidence: 99%

Federated Learning for Computationally Constrained Heterogeneous Devices: A Survey

et al. 2023

View full text Add to dashboard Cite

With an increasing number of smart devices like internet of things (IoT) devices deployed in the field, offloading training of neural networks (NNs) to a central server becomes more and more infeasible. Recent efforts to improve users’ privacy have led to on-device learning emerging as an alternative. However, a model trained only on a single device, using only local data, is unlikely to reach a high accuracy. Federated learning (FL) has been introduced as a solution, offering a privacy-preserving trade-off between communication overhead and model accuracy by sharing knowledge between devices but disclosing the devices’ private data. The applicability and the benefit of applying baseline FL are, however, limited in many relevant use cases due to the heterogeneity present in such environments. In this survey, we outline the heterogeneity challenges FL has to overcome to be widely applicable in real-world applications. We especially focus on the aspect of computation heterogeneity among the participating devices and provide a comprehensive overview of recent works on heterogeneity-aware FL. We discuss two groups: works that adapt the NN architecture and works that approach heterogeneity on a system level, covering Federated Averaging (FedAvg), distillation, and split learning-based approaches, as well as synchronous and asynchronous aggregation schemes.

show abstract

FedHe: Heterogeneous Models and Communication-Efficient Federated Learning

Cited by 14 publications

References 28 publications

Communication Efficiency and Non-Independent and Identically Distributed Data Challenge in Federated Learning: A Systematic Mapping Study

Communication Efficiency and Non-Independent and Identically Distributed Data Challenge in Federated Learning: A Systematic Mapping Study

FedTKD: A Trustworthy Heterogeneous Federated Learning Based on Adaptive Knowledge Distillation

Federated Learning for Computationally Constrained Heterogeneous Devices: A Survey

Contact Info

Product

Resources

About