Parallel Nonnegative CP Decomposition of Dense Tensors

Abstract

The CP tensor decomposition is a low-rank approximation of a tensor. We present a distributed-memory parallel algorithm and implementation of an alternating optimization method for computing a CP decomposition of dense tensor data that can enforce nonnegativity of the computed low-rank factors. The principal task is to parallelize the matricized-tensor times Khatri-Rao product (MTTKRP) bottleneck subcomputation. The algorithm is computation efficient, using dimension trees to avoid redundant computation across MTTKRPs within the alternating method. Our approach is also communication efficient, using a data distribution and parallel algorithm across a multidimensional processor grid that can be tuned to minimize communication. We benchmark our software on synthetic as well as hyperspectral image and neuroscience dynamic functional connectivity data, demonstrating that our algorithm scales well to 100s of nodes (up to 4096 cores) and is faster and more general than the currently available parallel software.

Efficient parallel CP decomposition with pairwise perturbation and multi-sweep dimension tree

“…First, we propose the multi-sweep dimension tree (MSDT) algorithm, which requires the TTM between an order-N input tensor with dimension size s and the first-contracted input matrix once every N −1 N sweeps and reduce the leading persweep computational cost of a rank-R CP-ALS to 2 N N −1 s N R. This algorithm can produce exactly the same results as the standard dimension tree, i.e., it has no accuracy loss. Leveraging a parallelization strategy similar to previous work [3], [10] that performs the dimension tree calculations locally, our benchmark results show a speed-up of 1.25X compared to the state-of-art dimension tree running on 1024 processors.…”

“…For an order N tensor with modes of dimension s and CP rank R, N MTTKRPs are necessary in each sweep, each costing 2s N R to the leading order. The state-of-art dimension tree based construction of MTTKRP [16], [29], [3] uses amortization to save the cost, and has been implemented in multiple tensor computation libraries [10], [22]. However, it still requires the computational cost of at least 4s N R for each sweep.…”

“…To further accelerate CP-ALS, many researchers leverage different techniques to parallelize and approximate MTTKRP calculations. The parallelization strategies have been developed both for dense tensors on GPUs [12] and distributed memory systems [3], [20], and for sparse tensors on GPUs [27] and distributed memory systems [19], [35], [34], [17]. The communication lower bounds for a single dense MTTKRP computation has been discussed in [5], [4].…”

“…Second, we propose a communication-efficient pairwise perturbation algorithm. The implementation also uses a parallelization strategy similar to [3], [10], and reduces the communication cost by constructing the first-order local PP operators in the PP initialization step as well as constructing the local MTTKRP approximations in the PP approximated step. Our benchmark results show that the PP approximated step achieves a speed-up of 1.94X compared to the state-of-art dimension tree running on 1024 processors.…”

Accelerating block coordinate descent for nonnegative tensor factorization

“…With modern machine learning applications of NTF in mind, for which input tensor sizes can be extremely large and NTF should be provided as a low-level routine, there would be a definite economical and scientific gain to speeding up NTF algorithms. Radically different approaches exist in the literature to speed up existing algorithms for solving NTF, such as parallel computing, 67,68 compression, and sketching. 69,70 The combinations and relationships between these methods is poorly understood.…”