Federated Optimization in Heterogeneous Networks

Abstract

Federated Learning is a distributed learning paradigm with two key challenges that differentiate it from traditional distributed optimization: (1) significant variability in terms of the systems characteristics on each device in the network (systems heterogeneity), and (2) non-identically distributed data across the network (statistical heterogeneity). In this work, we introduce a framework, FedProx, to tackle heterogeneity in federated networks. FedProx can be viewed as a generalization and re-parametrization of FedAvg, the current state-of-the-art method for federated learning. While FedProx makes only minor algorithmic modifications to FedAvg, these modifications have important ramifications both in theory and in practice. Theoretically, we provide convergence guarantees for our framework when learning over data from non-identical distributions (statistical heterogeneity), and while adhering to device-level systems constraints by allowing each participating device to perform a variable amount of work (systems heterogeneity). Practically, we demonstrate that FedProx allows for more robust convergence than FedAvg across a suite of federated datasets. In particular, in highly heterogeneous settings, FedProx demonstrates significantly more stable and accurate convergence behavior relative to FedAvg-improving absolute test accuracy by 22% on average.1 Privacy is a third key challenge in the federated setting. While not the focus of this work, standard privacy-preserving approaches such as differential privacy and secure multiparty communication can naturally be combined with the methods proposed herein-particularly since our framework proposes only lightweight algorithmic modifications to prior work.

Flexible Parallel Learning in Edge Scenarios: Communication, Computational and Energy Cost

“…4, forcing the same model to process all images may be a suboptimal approach. Using advanced strategies like FedProx [17] to combine the updates coming from different nodes does improve the performance compared to FedAvg, due to FedProx's ability to deal with non-i.i.d. data, but does not close the performance gap.…”

“…Dealing with low-quality, non-i.i.d. dataset is the focus of several FL studies, including [5], [15]- [17], and the main strategy they use is assigning to learning nodes weights reflecting their data quality, thus, the contribution they can give to the learning process. FPL achieves the same objective through the junction layer: the values of the parameters therein -hence, the importance to assign to different data sourcesare found as a part of the DNN training process.…”

“…This reproduces and generalizes FL. Since the input datasets are not i.i.d., the FedProx strategy [17] is used in lieu of the more common FedAvg to combine the updates coming from different nodes. One averaging round is executed after each local computation one.…”

An Efficient and Robust System for Vertically Federated Random Forest

“…As federated learning has raised lots of attention over the recent years, researchers also start to investigate building efficient libraries (He et al, 2020;Beutel et al, 2020) and systems (Bonawitz et al, 2019b;a;Hiessl et al, 2020). Several seminal work has also been done on designing better communication protocol (Konečnỳ et al, 2016), optimization algorithms (Li et al, 2018), and improving model robustness (Konstantinidis & Ramamoorthy, 2021). However, most of the existing research is targeting horizontal federated learning systems and vertical federated learning systems are still waiting to be tailored for industrial production.…”

Multi-Model Federated Learning

“…The policies were tested on Synthetic(1,1) [15], [16] and Synthetic-IID [15], [16] datasets with logistic regression as the classification model. Synthetic(α, β) [15] is a dataset involving labelled feature vectors that can be non-identically or identically distributed across the clients. The variation in the underlying local models is decided by α while β introduces non-i.i.d.…”