PLANC: Parallel Low Rank Approximation with Non-negativity Constraints

Abstract

We consider the problem of low-rank approximation of massive dense non-negative tensor data, for example to discover latent patterns in video and imaging applications. As the size of data sets grows, single workstations are hitting bottlenecks in both computation time and available memory. We propose a distributed-memory parallel computing solution to handle massive data sets, loading the input data across the memories of multiple nodes and performing efficient and scalable parallel algorithms to compute the low-rank approximation. We present a software package called PLANC (Parallel Low Rank Approximation with Non-negativity Constraints), which implements our solution and allows for extension in terms of data (dense or sparse, matrices or tensors of any order), algorithm (e.g., from multiplicative updating techniques to alternating direction method of multipliers), and architecture (we exploit GPUs to accelerate the computation in this work). We describe our parallel distributions and algorithms, which are careful to avoid unnecessary communication and computation, show how to extend the software to include new algorithms and/or constraints, and report efficiency and scalability results for both synthetic and real-world data sets.

Efficient parallel CP decomposition with pairwise perturbation and multi-sweep dimension tree

“…First, we propose the multi-sweep dimension tree (MSDT) algorithm, which requires the TTM between an order-N input tensor with dimension size s and the first-contracted input matrix once every N −1 N sweeps and reduce the leading persweep computational cost of a rank-R CP-ALS to 2 N N −1 s N R. This algorithm can produce exactly the same results as the standard dimension tree, i.e., it has no accuracy loss. Leveraging a parallelization strategy similar to previous work [3], [10] that performs the dimension tree calculations locally, our benchmark results show a speed-up of 1.25X compared to the state-of-art dimension tree running on 1024 processors.…”

“…Our parallel algorithms for CP-ALS on dense tensors are based on Algorithm 3, which is introduced in [3], [10]. The input tensor T T T with order N is uniformly distributed across an order N processor grid P, and all the factor matrices are initially distributed such that each processor owns a subset of the rows.…”

“…Distributing the work in the solve reduced the computational and bandwidth costs, while raising the latency cost. Our performance evaluation in Section V-B also includes the PLANC implementation of CP-ALS [10], which makes use of a sequential linear system solve. As we will show, the cost of solving linear systems is often small for both approaches, considering that the MTTKRP calculations are the major bottleneck.…”

“…For an order N tensor with modes of dimension s and CP rank R, N MTTKRPs are necessary in each sweep, each costing 2s N R to the leading order. The state-of-art dimension tree based construction of MTTKRP [16], [29], [3] uses amortization to save the cost, and has been implemented in multiple tensor computation libraries [10], [22]. However, it still requires the computational cost of at least 4s N R for each sweep.…”

“…Second, we propose a communication-efficient pairwise perturbation algorithm. The implementation also uses a parallelization strategy similar to [3], [10], and reduces the communication cost by constructing the first-order local PP operators in the PP initialization step as well as constructing the local MTTKRP approximations in the PP approximated step. Our benchmark results show that the PP approximated step achieves a speed-up of 1.94X compared to the state-of-art dimension tree running on 1024 processors.…”