Communication-Efficient and Personalized Federated Lottery Ticket Learning

Seo, Sejin; Ko, Seung-Woo; Park, Jihong; Kim, Seong-Lyun; Bennis, Mehdi

doi:10.1109/spawc51858.2021.9593126

Cited by 6 publications

(7 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Some researchers use the category information of the client's private dataset to transfer knowledge. For example, the FedDistill+ method [18] requires each client to calculate the average logit information of each category, and the server-side fuses this information to form the global logit information of each category and finally guides the client model training through this information. Chan et al proposed the FedHe method, which also aggregates the logit information of private data to achieve knowledge aggregation and then uses this knowledge to guide the model training of each client.…”

Section: Federated Knowledge Distillationmentioning

confidence: 99%

FedTKD: A Trustworthy Heterogeneous Federated Learning Based on Adaptive Knowledge Distillation

Chen,

Zhang,

Dong

et al. 2024

Entropy

View full text Add to dashboard Cite

Federated learning allows multiple parties to train models while jointly protecting user privacy. However, traditional federated learning requires each client to have the same model structure to fuse the global model. In real-world scenarios, each client may need to develop personalized models based on its environment, making it difficult to perform federated learning in a heterogeneous model environment. Some knowledge distillation methods address the problem of heterogeneous model fusion to some extent. However, these methods assume that each client is trustworthy. Some clients may produce malicious or low-quality knowledge, making it difficult to aggregate trustworthy knowledge in a heterogeneous environment. To address these challenges, we propose a trustworthy heterogeneous federated learning framework (FedTKD) to achieve client identification and trustworthy knowledge fusion. Firstly, we propose a malicious client identification method based on client logit features, which can exclude malicious information in fusing global logit. Then, we propose a selectivity knowledge fusion method to achieve high-quality global logit computation. Additionally, we propose an adaptive knowledge distillation method to improve the accuracy of knowledge transfer from the server side to the client side. Finally, we design different attack and data distribution scenarios to validate our method. The experiment shows that our method outperforms the baseline methods, showing stable performance in all attack scenarios and achieving an accuracy improvement of 2% to 3% in different data distributions.

show abstract

Section: Federated Knowledge Distillationmentioning

confidence: 99%

FedTKD: A Trustworthy Heterogeneous Federated Learning Based on Adaptive Knowledge Distillation

Chen,

Zhang,

Dong

et al. 2024

Entropy

View full text Add to dashboard Cite

show abstract

“…Quantization [142][143][144][145]148,152,156,158,168,169,174,176,183,184,188,[191][192][193]198,199] Sparsification [140,141,151,153,155,165,174,186,200,202,204] Client Selection [147,166,172,185,191,198,207] Asynchronous [146,171,190,203,211] Two-Level Aggregation [164,175,180,182,185] Select Model Updates [149,157,…”

Section: Techniques Studies Referencedmentioning

confidence: 99%

Communication Efficiency and Non-Independent and Identically Distributed Data Challenge in Federated Learning: A Systematic Mapping Study

Alotaibi,

Khan,

Mahmood

2024

Applied Sciences

View full text Add to dashboard Cite

Federated learning has emerged as a promising approach for collaborative model training across distributed devices. Federated learning faces challenges such as Non-Independent and Identically Distributed (non-IID) data and communication challenges. This study aims to provide in-depth knowledge in the federated learning environment by identifying the most used techniques for overcoming non-IID data challenges and techniques that provide communication-efficient solutions in federated learning. The study highlights the most used non-IID data types, learning models, and datasets in federated learning. A systematic mapping study was performed using six digital libraries, and 193 studies were identified and analyzed after the inclusion and exclusion criteria were applied. We identified that enhancing the aggregation method and clustering are the most widely used techniques for non-IID data problems (used in 18% and 16% of the selected studies), and a quantization technique was the most common technique in studies that provide communication-efficient solutions in federated learning (used in 27% and 15% of the selected studies). Additionally, our work shows that label distribution skew is the most used case to simulate a non-IID environment, specifically, the quantity label imbalance. The supervised learning model CNN model is the most commonly used learning model, and the image datasets MNIST and Cifar-10 are the most widely used datasets when evaluating the proposed approaches. Furthermore, we believe the research community needs to consider the client’s limited resources and the importance of their updates when addressing non-IID and communication challenges to prevent the loss of valuable and unique information. The outcome of this systematic study will benefit federated learning users, researchers, and providers.

show abstract

“…The data-based optimization methods can leverage data generation techniques to generate data that makes the data among different participants closer to being identically distributed, thus alleviating the performance loss of the global model caused by data heterogeneity between the clients. FedDistill [35] is a data-free knowledge distillation method, where the participants share the average of label-based logit vectors. FedGAN [36] trains a generative adversarial network (GAN) to handle non-IID data challenges in an efficient communication manner, but inevitably introduces bias.…”

Section: Federated Learningmentioning

confidence: 99%

“…We combine the federated learning framework with the classical RGCN method and apply the knowledge distillation method to optimize the classical federated learning algorithm to address the problem of data heterogeneity among clients. Therefore, the baselines we apply include a detection model trained only locally (named Local), and federated learning methods FedAvg [25], FedProx [32], and FedDistill [35] that address the data heterogeneity problem. We evaluate the performance of these methods under different degrees of data heterogeneity.…”

Section: Datasetsmentioning

confidence: 99%

FedKG: A Knowledge Distillation-Based Federated Graph Method for Social Bot Detection

Wang,

Chen,

Wang

et al. 2024

Sensors

View full text Add to dashboard Cite

Malicious social bots pose a serious threat to social network security by spreading false information and guiding bad opinions in social networks. The singularity and scarcity of single organization data and the high cost of labeling social bots have given rise to the construction of federated models that combine federated learning with social bot detection. In this paper, we first combine the federated learning framework with the Relational Graph Convolutional Neural Network (RGCN) model to achieve federated social bot detection. A class-level cross entropy loss function is applied in the local model training to mitigate the effects of the class imbalance problem in local data. To address the data heterogeneity issue from multiple participants, we optimize the classical federated learning algorithm by applying knowledge distillation methods. Specifically, we adjust the client-side and server-side models separately: training a global generator to generate pseudo-samples based on the local data distribution knowledge to correct the optimization direction of client-side classification models, and integrating client-side classification models’ knowledge on the server side to guide the training of the global classification model. We conduct extensive experiments on widely used datasets, and the results demonstrate the effectiveness of our approach in social bot detection in heterogeneous data scenarios. Compared to baseline methods, our approach achieves a nearly 3–10% improvement in detection accuracy when the data heterogeneity is larger. Additionally, our method achieves the specified accuracy with minimal communication rounds.

show abstract

Communication-Efficient and Personalized Federated Lottery Ticket Learning

Cited by 6 publications

References 7 publications

FedTKD: A Trustworthy Heterogeneous Federated Learning Based on Adaptive Knowledge Distillation

FedTKD: A Trustworthy Heterogeneous Federated Learning Based on Adaptive Knowledge Distillation

Communication Efficiency and Non-Independent and Identically Distributed Data Challenge in Federated Learning: A Systematic Mapping Study

FedKG: A Knowledge Distillation-Based Federated Graph Method for Social Bot Detection

Contact Info

Product

Resources

About