Federated Evaluation of On-device Personalization

Wang, Kangkang; Mathews, Rajiv; Kiddon, Chloé; Eichner, Hubert; Beaufays, Françoise; Ramage, Daniel

doi:10.48550/arxiv.1910.10252

Cited by 60 publications

(93 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The benefit of this personalization has been shown for the language model of a virtual smartphone keyboard. [65] One personalized federated averaging algorithm, called Per-FedAvg [16] uses model-agnostic metalearning [17] methods to find an initial shared model that users can adapt to their local data with comparably little training. Mansour et al [44] propose three different approaches for the personalization of models: user clustering, data interpolation, and model interpolation.…”

Section: Specialization In Federated Learningmentioning

confidence: 99%

Implicit model specialization through dag-based decentralized federated learning

Beilharz

Pfitzner

Schmid

et al. 2021

Proceedings of the 22nd International Middleware Conference

View full text Add to dashboard Cite

Federated learning allows a group of distributed clients to train a common machine learning model on private data. The exchange of model updates is managed either by a central entity or in a decentralized way, e.g. by a blockchain. However, the strong generalization across all clients makes these approaches unsuited for non-independent and identically distributed (non-IID) data.We propose a unified approach to decentralization and personalization in federated learning that is based on a directed acyclic graph (DAG) of model updates. Instead of training a single global model, clients specialize on their local data while using the model updates from other clients dependent on the similarity of their respective data. This specialization implicitly emerges from the DAG-based communication and selection of model updates. Thus, we enable the evolution of specialized models, which focus on a subset of the data and therefore cover non-IID data better than federated learning in a centralized or blockchain-based setup.To the best of our knowledge, the proposed solution is the first to unite personalization and poisoning robustness in fully decentralized federated learning. Our evaluation shows that the specialization of models emerges directly from the DAG-based communication of model updates on three different datasets. Furthermore, we show stable model accuracy and less variance across clients when compared to federated averaging.

show abstract

Section: Specialization In Federated Learningmentioning

confidence: 99%

Implicit model specialization through dag-based decentralized federated learning

Beilharz

Pfitzner

Schmid

et al. 2021

Proceedings of the 22nd International Middleware Conference

View full text Add to dashboard Cite

show abstract

“…Recently, various approaches have been proposed to realize personalized FL with homogeneous local model structure, which can be categorized into three types according to the number of global models applied in the server, i.e., single global model, multiple global models and no global model. single global model type is a close variety of conventional FL, e.g., FedAvg [1], that combine global model optimization process with additional local model customization, and consist of four different kinds of approaches: local fine-tuning [20][21][22][23], regularization (e.g., pFedMe [6], L2SGD [7,24], Ditto [25]), hybrid local and global models [11,26,27] and meta learning [9,28]. All of these pFL methods apply a single global model, and thus limit the customized level of the local model at the client side.…”

Section: Related Workmentioning

confidence: 99%

Parameterized Knowledge Transfer for Personalized Federated Learning

Zhang¹,

Guo²,

Ma³

et al. 2021

Preprint

View full text Add to dashboard Cite

In recent years, personalized federated learning (pFL) has attracted increasing attention for its potential in dealing with statistical heterogeneity among clients. However, the state-of-the-art pFL methods rely on model parameters aggregation at the server side, which require all models to have the same structure and size, and thus limits the application for more heterogeneous scenarios. To deal with such model constraints, we exploit the potentials of heterogeneous model settings and propose a novel training framework to employ personalized models for different clients. Specifically, we formulate the aggregation procedure in original pFL into a personalized group knowledge transfer training algorithm, namely, KT-pFL, which enables each client to maintain a personalized soft prediction at the server side to guide the others' local training. KT-pFL updates the personalized soft prediction of each client by a linear combination of all local soft predictions using a knowledge coefficient matrix, which can adaptively reinforce the collaboration among clients who own similar data distribution. Furthermore, to quantify the contributions of each client to others' personalized training, the knowledge coefficient matrix is parameterized so that it can be trained simultaneously with the models. The knowledge coefficient matrix and the model parameters are alternatively updated in each round following the gradient descent way. Extensive experiments on various datasets (EMNIST, Fashion_MNIST, CIFAR-10) are conducted under different settings (heterogeneous models and data distributions). It is demonstrated that the proposed framework is the first federated learning paradigm that realizes personalized model training via parameterized group knowledge transfer while achieving significant performance gain comparing with state-of-the-art algorithms.

show abstract

“…The first type of solutions include proposals such as Transfer Learning techniques to adapt a pre-trained model over a public dataset to a bunch of devices [36], although it can also be applied without needing a public dataset [37]. Another idea, performed in [38,39] is adapting algorithms from the Model-Agnostic Meta-Learning (MAML) setting, such as Reptile, to a federated setting.…”

Section: Client-level Personalizationmentioning

confidence: 99%

“…As far as we are concerned, this strategy has only been performed in centralized settings. Nonetheless, in a decentralized setting each participant could Changes in the input space through clients Domain transformation Domains with particular features [62,63,64] Domain factorization [65,66,67,68] Personalization [36,37,38,39,45,56] Domain adaptation [69,70,71] Dissimilarity methods [72,73,74,75] Sample reweighting [76,77] Generative adversarial networks [78,79,80,81,82] Figure 1: Classification of the different approaches that are able to solve the problem of spatial heterogeneity in the input spaces.…”

Section: Changes In the Input Space Throughout Clientsmentioning

confidence: 99%

See 1 more Smart Citation

Non-IID data and Continual Learning processes in Federated Learning: A long road ahead

Criado¹,

Casado²,

Iglesias³

et al. 2021

Preprint

View full text Add to dashboard Cite

Federated Learning is a novel framework that allows multiple devices or institutions to train a machine learning model collaboratively while preserving their data private. This decentralized approach is prone to suffer the consequences of data statistical heterogeneity, both across the different entities and over time, which may lead to a lack of convergence. To avoid such issues, different methods have been proposed in the past few years. However, data may be heterogeneous in lots of different ways, and current proposals do not always determine the kind of heterogeneity they are considering. In this work, we formally classify data statistical heterogeneity and review the most remarkable learning strategies that are able to face it. At the same time, we introduce approaches from other machine learning frameworks, such as Continual Learning, that also deal with data heterogeneity and could be easily adapted to the Federated Learning settings.

show abstract

Federated Evaluation of On-device Personalization

Cited by 60 publications

References 0 publications

Implicit model specialization through dag-based decentralized federated learning

Implicit model specialization through dag-based decentralized federated learning

Parameterized Knowledge Transfer for Personalized Federated Learning

Non-IID data and Continual Learning processes in Federated Learning: A long road ahead

Contact Info

Product

Resources

About