Knowledge Distillation for Federated Learning: a Practical Guide

Abstract

Federated Learning (FL) enables the training of Deep Learning models without centrally collecting possibly sensitive raw data. This paves the way for stronger privacy guarantees when building predictive models. The most used algorithms for FL are parameter-averaging based schemes (e.g., Federated Averaging) that, however, have well known limits: (i) Clients must implement the same model architecture; (ii) Transmitting model weights and model updates implies high communication cost, which scales up with the number of model parameters; (iii) In presence of non-IID data distributions, parameter-averaging aggregation schemes perform poorly due to client model drifts. Federated adaptations of regular Knowledge Distillation (KD) can solve and/or mitigate the weaknesses of parameter-averaging FL algorithms while possibly introducing other trade-offs. In this article, we provide a review of KD-based algorithms tailored for specific FL issues.

Federated Learning in Computer Vision

“…Knowledge Distillation (KD) was introduced in [84] for model compression: it allows to transfer knowledge from a larger network (teacher) to a smaller one (student). It has been widely used in continual learning and recently has been increasingly employed in FL algorithms [162] to reduce catastrophic forgetting, tackle data heterogeneity, and enable model heterogeneity.…”

Survey of Knowledge Distillation in Federated Edge Learning

“…To the best of our knowledge, this paper is the first work to investigate the application of knowledge distillation in federated edge learning. Different from existing surveys [33,40,21,26], we take the challenges faced by FEL as the main clue, introducing existing FEL approaches based on diverse forms of KD techniques and providing guidance for both future research directions and real deployment. Specifically, the reminders of this paper are organized as follows.…”