Class-Wise Adaptive Self Distillation for Federated Learning on Non-IID Data (Student Abstract)

Abstract

Federated learning (FL) enables multiple clients to collaboratively train a globally generalized model while keeping local data decentralized. A key challenge in FL is to handle the heterogeneity of data distributions among clients. The local model will shift the global feature when fitting local data, which results in forgetting the global knowledge. Following the idea of knowledge distillation, the global model's prediction can be utilized to help local models preserve the global knowledge in FL. However, when the global model hasn't converged completely, its predictions tend to be less reliable on certain classes, which may results in distillation's misleading of local models. In this paper, we propose a class-wise adaptive self distillation (FedCAD) mechanism to ameliorate this problem. We design class-wise adaptive terms to soften the influence of distillation loss according to the global model's performance on each class and therefore avoid the misleading. Experiments show that our method outperforms other state-of-the-art FL algorithms on benchmark datasets.

Knowledge Distillation for Federated Learning: a Practical Guide

“…KD-based solutions can be used to handle data heterogeneity either at server side, rectifying FedAvg's global model via ensemble distillation on a proxy dataset [30,41,7] or using a data-free generator [54,53], or at client side, distilling global knowledge via on-device regularizers [52,25,17,16] or synthetically-generated data [57], directly controlling the phenomenon of client drift.…”

“…Inspired by the work of Lukasik et al [33], He et al further observe that, in the framework of Fig. 1, leveraging an inaccurate global model (i.e., inaccurate teacher) on specific classification classes might mislead local training [17]. To alleviate such phenomenon, a class-wise adaptive weight is proposed in FedCAD [17] to control the impact of distillation: when the global model is accurate on a certain class, local models learn more from the distilled knowledge.…”

“…Then, a set of strategies proposed to enhance the aggregation step of FedAvg with a server-side ensemble distillation phase to enable model heterogeneity and/or improve model fusion in presence of heterogeneous data [30,41,7,54,53]. Recently, two KD-based lines of work focused on mitigating the phenomenon of client model drift -which makes averaging-based aggregations inefficient -either using regularization terms in clients' objective functions [52,25,17,16,56] or leveraging globally learned data-free generator [57].…”

FedAAR: A Novel Federated Learning Framework for Animal Activity Recognition with Wearable Sensors

“…First, the movement patterns of individual animals are often drawn from distinct distributions, which inevitably results in data heterogeneity between clients. Such data heterogeneity enlarges the inconsistency of learned features across clients, easily raising drift concerns between client updates since each client model is optimised towards its local objective instead of global optima during local training [ 13 , 14 , 15 ]. To address this issue, some existing methods [ 14 , 15 , 16 , 17 ] impose constraints on the local optimisation by exploiting a model-level regularisation term, which aims to facilitate all local models to approach consistent views.…”

“…Such data heterogeneity enlarges the inconsistency of learned features across clients, easily raising drift concerns between client updates since each client model is optimised towards its local objective instead of global optima during local training [ 13 , 14 , 15 ]. To address this issue, some existing methods [ 14 , 15 , 16 , 17 ] impose constraints on the local optimisation by exploiting a model-level regularisation term, which aims to facilitate all local models to approach consistent views. For instance, FedProx restricted local model parameters to be close to global parameters by adding a proximal term in the local training [ 16 ].…”