Distributed Differentially Private Algorithms for Matrix and Tensor Factorization

Abstract

In many signal processing and machine learning applications, datasets containing private information are held at different locations, requiring the development of distributed privacy-preserving algorithms. Tensor and matrix factorizations are key components of many processing pipelines. In the distributed setting, differentially private algorithms suffer because they introduce noise to guarantee privacy. This paper designs new and improved distributed and differentially private algorithms for two popular matrix and tensor factorization methods: principal component analysis (PCA) and orthogonal tensor decomposition (OTD). The new algorithms employ a correlated noise design scheme to alleviate the effects of noise and can achieve the same noise level as the centralized scenario. Experiments on synthetic and real data illustrate the regimes in which the correlated noise allows performance matching with the centralized setting, outperforming previous methods and demonstrating that meaningful utility is possible while guaranteeing differential privacy.

When Machine Learning Meets Privacy

“…Apart from the above-mentioned works, there are other research works in the closely relevant area, such as tensor/matrix factorizations and functional optimization schemes. In more detail, the authors of [59,60] discussed differentially private algorithms for tensor decomposition, in both centralized and distributed settings [60]. The authors of [40] applied a DP framework in the matrix factorization process with four different possible perturbation: input perturbation, private stochastic gradient perturbation, alternating least squares (ALS) with output perturbation, and output perturbation.…”

Column-Wise Element Selection for Computationally Efficient Nonnegative Coupled Matrix Tensor Factorization

“…To effectively parallelize or distribute the calculation of MTTKRP, the majority of researchers rely on the parallel and distributed setups [12], [14], [15], [16], [17], [26]. For instance, the parallel and distributed stochastic gradientbased FlexiFact algorithm was proposed to reduce the computational cost [14].…”

A Survey on Differentially Private Machine Learning [Review Article]

“…The differentially private algo rithms perform worse in the distributed environment since the introduction of a larger volume of noise. However, a dis tributed differentially private algorithm for PCA is proposed in [47], which employs a correlated noise design scheme to alleviate the effects of noise and achieves the same noise level as the cen tralized scenario. This method defines a noise generator to generate the D × D matrix Es i.i.d.…”

“…Some techniques are designed to reduce time complexity and there remains room to be further improved. In addition, a dis tributed differentially private algorithm [47] for PCA is developed.…”