Asynchronous gossip principal components analysis

Fellus, Jérôme; Picard, David; Gosselin, Philippe-Henri

doi:10.1016/j.neucom.2014.11.076

Cited by 8 publications

(5 citation statements)

References 21 publications

(38 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A plethora of algorithms has been designed for distributed sensor networks dealing with both horizontal data partitioning and vertical partitioning, including but not limited to work described in Bertrand and Moonen (2014) , Fellus et al (2015) , Jelasity et al (2007) , Schizas and Aduroja (2015 ) and Wu et al (2018) . These algorithms cover cross-device FL.…”

Section: Methodsmentioning

confidence: 99%

Federated horizontally partitioned principal component analysis for biomedical applications

Hartebrodt

Röttger

2022

Bioinformatics Advances

View full text Add to dashboard Cite

Motivation Federated learning enables privacy preserving machine learning in the medical domain because the sensitive patient data remains with the owner and only parameters are exchanged between the data holders. The federated scenario introduces specific challenges related to the decentralized nature of the data, such as batch effects and differences in study population between the sites. Here, we investigate the challenges of moving classical analysis methods to the federated domain, specifically principal component analysis, a versatile and widely used tool, often serving as an initial step in machine learning and visualization workflows. We provide implementations of different federated PCA algorithms and evaluate them regarding their accuracy for high dimensional biological data using realistic sample distributions over multiple data sites, and their ability to preserve downstream analyses. Results Federated subspace iteration converges to the centralized solution even for unfavorable data distributions, while approximate methods introduce error. Larger sample sizes at the study sites lead to better accuracy of the approximate methods. Approximate methods may be sufficient for coarse data visualization, but are vulnerable to outliers and batch effects. Before the analysis, the PCA algorithm, as well as the number of eigenvectors should be considered carefully to avoid unnecessary communication overhead. Availability Simulation code and notebooks for federated PCA can be found at https://gitlab.com/roettgerlab/federatedPCA; the code for the federated app is available at https://github.com/AnneHartebrodt/fc-federated-pca Supplementary information Supplementary data are available at Bioinformatics Advances online.

show abstract

Section: Methodsmentioning

confidence: 99%

Federated horizontally partitioned principal component analysis for biomedical applications

Hartebrodt

Röttger

2022

Bioinformatics Advances

View full text Add to dashboard Cite

show abstract

“…Another class of methods is called block (updating) algorithms, as represented by the classic method of simultaneous subspace iteration (SSI) [39,44,45], that generally have higher scalability because their main computations are large matrix times (relatively) small and dense block matrices. Indeed there are various practical implementations of SSI under distributed settings, such as [10,13,25,37,47].…”

Section: Overview Of Related Workmentioning

confidence: 99%

A Distributed and Secure Algorithm for Computing Dominant SVD Based on Projection Splitting

Wang¹,

Liu²,

Zhang³

2020

Preprint

View full text Add to dashboard Cite

In this paper, we propose and study a distributed and secure algorithm for computing dominant (or truncated) singular value decompositions (SVD) of large and distributed data matrices. We consider the scenario where each node privately holds a subset of columns and only exchanges "safe" information with other nodes in a collaborative effort to calculate a dominant SVD for the whole matrix. In the framework of alternating direction methods of multipliers (ADMM), we propose a novel formulation for building consensus by equalizing subspaces spanned by splitting variables instead of equalizing themselves. This technique greatly relaxes feasibility restrictions and accelerates convergence significantly, while at the same time yielding simple subproblems. We design several algorithmic features, including a low-rank multiplier formula and mechanisms for controlling subproblem solution accuracies, to increase the algorithm's computational efficiency and reduce its communication overhead. More importantly, unlike many existing distributed or parallelized algorithms, our algorithm preserves the privacy of locally-held data; that is, none of the nodes can recover the data stored in another node through information exchanged during communications. We present convergence analysis results, including a worst-case complexity estimate, and extensive experimental results indicating that the proposed algorithm, while safely guarding data privacy, has a strong potential to deliver a cutting-edge performance, especially when communication costs are high.

show abstract

“…A popular way of tackling this issue in distributed optimization involves exchanging information in a peer to peer fashion. In this category, Gossip protocols [11] have been successfully applied to many machine learning problems such as kernel methods [12] , PCA [13] and k-means clustering [14,15] . We propose to combine Gossip protocols with SGD to efficiently train deep CNN without the need of a central node.…”

Section: Introductionmentioning

confidence: 99%

Distributed optimization for deep learning with gossip exchange

et al. 2019

Self Cite

View full text Add to dashboard Cite

We address the issue of speeding up the training of convolutional neural networks by studying a distributed method adapted to stochastic gradient descent. Our parallel optimization setup uses several threads, each applying individual gradient descents on a local variable. We propose a new way of sharing information between different threads based on gossip algorithms that show good consensus convergence properties. Our method called GoSGD has the advantage to be fully asynchronous and decentralized.

show abstract

Asynchronous gossip principal components analysis

Cited by 8 publications

References 21 publications

Federated horizontally partitioned principal component analysis for biomedical applications

Federated horizontally partitioned principal component analysis for biomedical applications

A Distributed and Secure Algorithm for Computing Dominant SVD Based on Projection Splitting

Distributed optimization for deep learning with gossip exchange

Contact Info

Product

Resources

About