Cooperative Learning VIA Federated Distillation OVER Fading Channels

Abstract

Cooperative training methods for distributed machine learning are typically based on the exchange of local gradients or local model parameters. The latter approach is known as Federated Learning (FL). An alternative solution with reduced communication overhead, referred to as Federated Distillation (FD), was recently proposed that exchanges only averaged model outputs. While prior work studied implementations of FL over wireless fading channels, here we propose wireless protocols for FD and for an enhanced version thereof that leverages an offline communication phase to communicate "mixed-up" covariate vectors. The proposed implementations consist of different combinations of digital schemes based on separate source-channel coding and of over-the-air computing strategies based on analog joint source-channel coding. It is shown that the enhanced version FD has the potential to significantly outperform FL in the presence of limited spectral resources.

Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training With Non-IID Private Data

“…While FD performs well when the mobile devicegenerated data is identically and independently distributed, FD exhibits lower performance than the FL benchmark with model parameter exchange in non-IID data distributions. This was experimentally verified in [6], [7], [8], [9] and the experiments presented in Section 4. To fill this gap, we design an FL framework with model output exchange achieving similar or higher performance than previously proposed approaches even when subjected to non-IID data distributions.…”

“…Federated learning with model output exchange over mobile device-generated dataset. FD is proposed in [6], [7], [8], [9] as an FL framework with model output exchange that trains ML models considering mobile devicegenerated dataset. Unlike CD and PATE that trains distributed ML models using a shared dataset, each mobile device trains each ML model using a local dataset, enabling ML model training with mobile device-generated data.…”

“…Leveraging unlabeled data towards performance similar to that of benchmark federated learning under non-IID. Typical FL with model output exchange termed federated distillation (FD) [6], [7], [8], [9] can achieve scalability of the model size; however, these methods provide poor models in general. In FD, each mobile device re-trains local model based on both the local labeled data and the global logits.…”

Rate-Adapted Decentralized Learning Over Wireless Networks

“…4(b) (where 0 < λ < 1) into Eq. (7). The dependence of the training loss is insignificant at any K, even when the transmission rate was reduced from 6 Mbps to 3 Mbps.…”

“…Some researchers have investigated methods for quantization and sparsification of data that must be communicated to reduce the communication load. In particular, compressed sensing [6], [29] and digital data coding [7], [8] have been explored for data reduction when communicating with a central server. These approaches have been extended to be applicable to wireless systems.…”