A Distributed and Secure Algorithm for Computing Dominant SVD Based on Projection Splitting

Abstract

In this paper, we propose and study a distributed and secure algorithm for computing dominant (or truncated) singular value decompositions (SVD) of large and distributed data matrices. We consider the scenario where each node privately holds a subset of columns and only exchanges "safe" information with other nodes in a collaborative effort to calculate a dominant SVD for the whole matrix. In the framework of alternating direction methods of multipliers (ADMM), we propose a novel formulation for building consensus by equalizing subspaces spanned by splitting variables instead of equalizing themselves. This technique greatly relaxes feasibility restrictions and accelerates convergence significantly, while at the same time yielding simple subproblems. We design several algorithmic features, including a low-rank multiplier formula and mechanisms for controlling subproblem solution accuracies, to increase the algorithm's computational efficiency and reduce its communication overhead. More importantly, unlike many existing distributed or parallelized algorithms, our algorithm preserves the privacy of locally-held data; that is, none of the nodes can recover the data stored in another node through information exchanged during communications. We present convergence analysis results, including a worst-case complexity estimate, and extensive experimental results indicating that the proposed algorithm, while safely guarding data privacy, has a strong potential to deliver a cutting-edge performance, especially when communication costs are high.

A Communication-Efficient and Privacy-Aware Distributed Algorithm for Sparse PCA

“…Recently, distributed ADMM algorithms have been introduced to PCA and related problems to handle massive datasets, such as [21,39]. In general, these methods achieve algorithm level parallelization, which are much more secure than those based on linear algebra level parallelization.…”

“…However, [21] only deals with the special case of p = 1. And [39] can only tackle the optimization problem over the Grassmann manifold.…”

“…Conventional distributed algorithms will impose consensus constraints X i = Z for a global variable Z. Since each local function f i (X) is invariant under the orthogonal transformation, the subspace constraint X i X i = ZZ introduced in [39] can be exploited to split the problem (2). This strategy relaxes the feasibility restrictions and accelerates the convergence.…”

Decentralized Optimization Over the Stiefel Manifold by an Approximate Augmented Lagrangian Function

“…Recently, several distributed ADMM algorithms designed for centralized networks have emerged to solve various variants of PCA problems [39,40], which pursue a consensus on the subspaces spanned by splitting variables. This strategy relaxes feasibility restrictions and significantly improves the convergence.…”

Solving Optimization Problems over the Stiefel Manifold by Smooth Exact Penalty Function

“…Optimization problems with orthogonality constraints have wide applications in statistics [36,13], scientific computation [27? ], image processing [5] and many other related areas [17,31,49]. Interested readers could refer to some recent works [15,25,46,41], a recent survey [23], and several books [3,9] for details.…”