Data-Free Knowledge Distillation via Generator-Free Data Generation for Non-IID Federated Learning

Abstract

Data heterogeneity (Non-IID) on Federated Learning (FL) is currently a widely publicized problem, which leads to local model drift and performance degradation. Because of the advantage of knowledge distillation, it has been explored in some recent work to refine global models. However, these approaches rely on a proxy dataset or a data generator. First, in many FL scenarios, proxy dataset do not necessarily exist on the server. Second, the quality of data generated by the generator is unstable and the generator depends on the computing resources of the server. In this work, we propose a novel data-Free knowledge distillation approach via generator-Free Data Generation for Non-IID FL, dubbed as FedF2DG. Specifically, FedF2DG requires only local models to generate pseudo datasets for each client, and can generate hard samples by adding an additional regularization term that exploit disagreements between local model and global model. Meanwhile, FedF2DG enables flexible utilization of computational resources by generating pseudo dataset locally or on the server. And to address the label distribution shift in Non-IID FL, we propose a Data Generation Principle that can adaptively control the label distribution and number of pseudo dataset based on client current state, and this allows for the extraction of more client knowledge. Then knowledge distillation is performed to transfer the knowledge in local models to the global model. Extensive experiments demonstrate that our proposed method significantly outperforms the state-of-the-art FL methods and can serve as plugin for existing Federated Learning methds such as FedAvg, FedProx, etc, and improve their performance.

Geography of Technology Transfer in China

“…To improve the performance of FL, existing methods can be classified into two categories, i.e., homogeneous methods and heterogeneous methods. Homogeneous FL methods [5], [14]- [16] still use the same model as the global model for local training. This goal aims to use a wisely model training mechanism [14], [16], device selection mechanism [17]- [20], or a data processing mechanism [21]- [23] to improve the inference performance of the global model.…”

“…Homogeneous FL methods [5], [14]- [16] still use the same model as the global model for local training. This goal aims to use a wisely model training mechanism [14], [16], device selection mechanism [17]- [20], or a data processing mechanism [21]- [23] to improve the inference performance of the global model. Although homogeneous FL methods can alleviate performance degradation caused by non-IID data, their performance is still limited due to existing lowperformance devices.…”

Bayesian federated inference for estimating statistical models based on non‐shared multicenter data sets

“…The MAP estimators θ𝓁 and the matrices Â𝓁 are found analytically or numerically, depending on the form of the likelihood functions in the local centers. The matrix  is positive definite by definition, so its approximation in (7) as well (if the approximation is sufficiently accurate) and, thus, is invertible.…”

“…The formulae in (7) do not say anything about the plausibility of the subsets describing similar subpopulations. However, once we have computed the estimates ( θ, Â) we should find that θ is compatible with each 'local' estimate θ𝓁 , given the error bars coded in the matrices  and Â𝓁 .…”

“…In recent years much progress has been made in FL (eg, improved optimization in each center, aggregation of the local models at the central server and dealing with heterogeneity of the local populations [5][6][7] ). An overview of the recent developments including references is given in Liu et al 8 Multiple researchers have proposed FL in a Bayesian setting for deep learning models.…”