Implicit model specialization through dag-based decentralized federated learning

Abstract

Federated learning allows a group of distributed clients to train a common machine learning model on private data. The exchange of model updates is managed either by a central entity or in a decentralized way, e.g. by a blockchain. However, the strong generalization across all clients makes these approaches unsuited for non-independent and identically distributed (non-IID) data.We propose a unified approach to decentralization and personalization in federated learning that is based on a directed acyclic graph (DAG) of model updates. Instead of training a single global model, clients specialize on their local data while using the model updates from other clients dependent on the similarity of their respective data. This specialization implicitly emerges from the DAG-based communication and selection of model updates. Thus, we enable the evolution of specialized models, which focus on a subset of the data and therefore cover non-IID data better than federated learning in a centralized or blockchain-based setup.To the best of our knowledge, the proposed solution is the first to unite personalization and poisoning robustness in fully decentralized federated learning. Our evaluation shows that the specialization of models emerges directly from the DAG-based communication of model updates on three different datasets. Furthermore, we show stable model accuracy and less variance across clients when compared to federated averaging.

An Energy Efficient Specializing DAG Federated Learning Based on Event-Triggered Communication

“…These features of DAG-DLT make it a suitable choice for decentralized FL in edge device environments where resource limitations and connectivity constraints are prevalent. Among the existing studies [20], [30]- [33], an Implicit Model Specialization FL called Specializing DAG FL (SDAFGL) [20] proposed by Beilharz et al is attention-gaining. SDAFGL focuses on improving model accuracy locally on the client side.…”

“…And for the CNN model, we use the last two layers of the model for similarity measures. To assess the performance of our framework, we compared it with five baselines: FedAvg [4], FlexCFL [11], IFCA [12], FeSEM [13], and SDAFGL [20]. The details are as follows.…”

“…• FeSEM [13]: FeSEM is a framework that clusters clients by initializing multiple models in the server and minimizing the l 2 distance between the client model and the cluster model. • SDAGFL [20]: SDAGFL is a DAG-DLT based FL framework. It utilizes an accuracy-based biased random walk tip selection algorithm to find models that perform well on local data, enabling implicit model specialization and client cluster.…”

“…What's better is that our DAG-ACFL framework needs not to know the cluster information of clients in advance, which is more practical in real-world applications. [4], FLEXCFL [11], IFCA [12], FESEM [13] AND SDAGFL [20]…”

“…A decentralized clustered FL framework using Distributed Ledger Techniques (DLT) based on Directed Acyclic Graphs (DAG) [19], called SDAFGL [20], addresses the abovementioned challenge by using the advantages of the DAG-DLT. SDAFGL handles clustered-wise non-iid datasets using an accuracy-biased random walk tip selection algorithm.…”