Clustered Federated Learning: Model-Agnostic Distributed Multitask Optimization Under Privacy Constraints

Abstract

Federated learning (FL) is currently the most widely adopted framework for collaborative training of (deep) machine learning models under privacy constraints. Albeit its popularity, it has been observed that FL yields suboptimal results if the local clients' data distributions diverge. To address this issue, we present clustered FL (CFL), a novel federated multitask learning (FMTL) framework, which exploits geometric properties of the FL loss surface to group the client population into clusters with jointly trainable data distributions. In contrast to existing FMTL approaches, CFL does not require any modifications to the FL communication protocol to be made, is applicable to general nonconvex objectives (in particular, deep neural networks), does not require the number of clusters to be known a priori, and comes with strong mathematical guarantees on the clustering quality. CFL is flexible enough to handle client populations that vary over time and can be implemented in a privacy-preserving way. As clustering is only performed after FL has converged to a stationary point, CFL can be viewed as a postprocessing method that will always achieve greater or equal performance than conventional FL by allowing clients to arrive at more specialized models. We verify our theoretical analysis in experiments with deep convolutional and recurrent neural networks on commonly used FL data sets.

Dynamic-Fusion-Based Federated Learning for COVID-19 Detection

“…Federated learning can address statistical and system heterogeneity issues since models are trained locally [26]. However, challenges still exist in dealing with non-IID data Many researchers have worked on training data clustering [27], multi-stage local training [28], and multi-task learning [26]. Also, some works [29], [30] focus on incentive mechanism design to motivate clients to participate in the machine learning jobs.…”

Estimation of Microphone Clusters in Acoustic Sensor Networks Using Unsupervised Federated Learning

“…Our proposed approach builds upon the concepts introduced by the aforementioned works and focuses on adding a privacypreserving layer using a variation of FL [10,11], namely CFL [9]. Spectral features which, although aggregated still retain privacysensitive information [17], are replaced with more privacy-aware, locally learned, neural network parameter updates.…”

“…It is shown in [9,19] that for the cases where clients' data comes from different (incongruent) distributions, there is no single θ * that can optimally minimize the loss of all clients at the same time. For this reason, the authors suggest clustering the clients that have similar (congruent) distributions and training separate server models for each resulting cluster.…”

“…Hierarchical clustering using bi-partitioning can be recursively applied using A. The resulting two clusters c1 and c2 of each bipartitioning step are derived such that the maximum cross-cluster cosine similarity is always smaller than the minimum of either intracluster cosine similarities [9]:…”

“…Moreover, in a world where privacy concerns have taken center-stage [7] and privacy policies like the European Union General Data Protection Regulation (EU GDPR) [8] aim to enforce principles such as "privacy-by-design", a more privacy-sensible solution is recommended. Therefore, this work steers away from using feature representations derived directly from the raw audio data in favor of a more privacy-aware solution based on clustered federated learning (CFL) [9]. In this way, ASN nodes (clients) need only share locally learned neural network parameter updates with a central node (server).…”