Survey of Knowledge Distillation in Federated Edge Learning

Wu, Zhiyuan; Sun, Shuhui; Wang, Yuwei; Liu, Min; Jiang, Xuefeng; Li, Runhan

doi:10.48550/arxiv.2301.05849

Cited by 2 publications

(4 citation statements)

References 35 publications

(112 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The clients then use their labeled and unlabeled data to train their respective student models, which are subsequently sent back to the server for aggregation. Last but not least, the study in [25] provides a review of the use of knowledge distillation in federated edge learning, which is a distributed learning approach that involves extending the FL paradigm to edge devices like smartphones and IoT devices. The review focuses on previous research on knowledge distillation in this context and highlights the potential benefits, challenges, and future directions of the proposed approaches.…”

Section: B Knowledge Distillation For Iot Traffic Classificationmentioning

confidence: 99%

Unleashing the Potential of Knowledge Distillation for IoT Traffic Classification

Abbasi,

Shahraki,

Prieto

et al. 2024

Trans. Mach. Learn. Comm. Netw.

View full text Add to dashboard Cite

The Internet of Things (IoT) has revolutionized our lives by generating large amounts of data, however, the data needs to be collected, processed, and analyzed in real-time. Network Traffic Classification (NTC) in IoT is a crucial step for optimizing network performance, enhancing security, and improving user experience. Different methods are introduced for NTC, but recently Machine Learning solutions have received high attention in this field, however, Traditional Machine Learning (ML) methods struggle with the complexity and heterogeneity of IoT traffic, as well as the limited resources of IoT devices. Deep learning shows promise but is computationally intensive for resource-constrained IoT devices. Knowledge distillation is a solution to help ML by compressing complex models into smaller ones suitable for IoT devices. In this paper, we examine the use of knowledge distillation for IoT traffic classification. Through experiments, we show that the student model achieves a balance between accuracy and efficiency. It exhibits similar accuracy to the larger teacher model while maintaining a smaller size. This makes it a suitable alternative for resource-constrained scenarios like mobile or IoT traffic classification. We find that the knowledge distillation technique effectively transfers knowledge from the teacher model to the student model, even with reduced training data. The results also demonstrate the robustness of the approach, as the student model performs well even with the removal of certain classes. Additionally, we highlight the trade-off between model capacity and computational cost, suggesting that increasing model size beyond a certain point may not be beneficial. The findings emphasize the value of soft labels in training student models with limited data resources.

show abstract

Section: B Knowledge Distillation For Iot Traffic Classificationmentioning

confidence: 99%

Unleashing the Potential of Knowledge Distillation for IoT Traffic Classification

Abbasi,

Shahraki,

Prieto

et al. 2024

Trans. Mach. Learn. Comm. Netw.

View full text Add to dashboard Cite

show abstract

“…As noted in [25], knowledge distillation has emerged as an important technique for addressing challenges in federated edge learning. [23], [29] leverage knowledge distillation as a communication protocol for exchanging model representations among devices and the edge server, enabling communication-efficient training over heterogeneous models.…”

Section: Knowledge Distillation In Federated Edge Learningmentioning

confidence: 99%

“…In addition, we consider 4 different model architectures, where {A C 1 , A C 2 , A C 3 } are for clients, and A S is for the server, and the main configurations of four adopted models are shown in TABLE 4. It is worth noting that the model on the server does not contain the foremost Conv+Batch+ReLU layers to fit the training requirements of [24], [25]. Moreover, both client-side model homogeneity and heterogeneity are considered in our experiments.…”

Section: Modelsmentioning

confidence: 99%

“…However, existing methods based on SLIA endeavor to achieve satisfactory performance either relying on additional client-side training on unrealistic public datasets [20], [21], or requiring the transfer of embedded features with non-negligible sizes in addition to logits [22], [24]. They appear to be unfriendly to devices by reason of the induction of intensive computation, tremendous communication, or public datasets reliance, making them unsuitable for practical applications in EI [25].…”

mentioning

confidence: 99%

See 1 more Smart Citation

FedCache: A Knowledge Cache-driven Federated Learning Architecture for Personalized Edge Intelligence

Wu,

Sun,

Wang

et al. 2024

Preprint

View full text Add to dashboard Cite

Edge Intelligence (EI) enables Artificial Intelligence (AI) applications to run at the edge, where data analysis and decision-making can be performed in real-time and close to data sources. To protect data privacy and unify data silos distributed among end devices in EI, Federated Learning (FL) is proposed for collaborative training shared AI models across multiple devices without compromising data security. However, the prevailing FL approaches cannot guarantee model generalization and adaptation on heterogeneous clients. Recently, Personalized Federated Learning (PFL) has drawn growing awareness in EI, as it enables striking a productive balance between local-specific training requirements inherent in devices and global-generalized optimization objectives for satisfactory performance. However, most existing PFL methods are based on the Parameters Interaction-based Architecture (PIA) represented by FedAvg, which causes unaffordable communication burdens due to large-scale parameters transmission between devices and the edge server. In contrast, Logits Interaction-based Architecture (LIA) enables to update model parameters with logits transfer, and gains the advantages of communication lightweight and heterogeneous on-device model allowance compared to PIA. Nevertheless, previous LIA methods attempt to achieve satisfactory performance either relying on unrealistic public datasets or increasing communication overhead for additional information transmission other than logits. To tackle this dilemma, we propose a knowledge cache-driven PFL architecture, named FedCache, which reserves a knowledge cache on the server for fetching personalized knowledge from the samples with similar hashes to each given on-device sample. During the training phase, ensemble distillation is applied to on-device models for constructive optimization with personalized knowledge transferred from the server-side knowledge cache. Empirical experiments on four datasets demonstrate the comparable performance of FedCache with state-of-art PFL approaches, with more than two orders of magnitude improvements in communication efficiency. Our code and DEMO are available at https://github.com/wuzhiyuan2000/FedCache.

show abstract

Survey of Knowledge Distillation in Federated Edge Learning

Cited by 2 publications

References 35 publications

Unleashing the Potential of Knowledge Distillation for IoT Traffic Classification

Unleashing the Potential of Knowledge Distillation for IoT Traffic Classification

FedCache: A Knowledge Cache-driven Federated Learning Architecture for Personalized Edge Intelligence

Contact Info

Product

Resources

About