Partitioning Models for General Medium-Grain Parallel Sparse Tensor Decomposition

Abstract

The focus of this article is efficient parallelization of the canonical polyadic decomposition algorithm utilizing the alternating least squares method for sparse tensors on distributed-memory architectures. We propose a hypergraph model for general mediumgrain partitioning which does not enforce any topological constraint on the partitioning. The proposed model is based on splitting the given tensor into nonzero-disjoint component tensors. Then a mode-dependent coarse-grain hypergraph is constructed for each component tensor. A net amalgamation operation is proposed to form a composite medium-grain hypergraph from these modedependent coarse-grain hypergraphs to correctly encapsulate the minimization of the communication volume. We propose a heuristic which splits the nonzeros of dense slices to obtain sparse slices in component tensors. So we partially attain slice coherency at (sub)slice level since partitioning is performed on (sub)slices instead of individual nonzeros. We also utilize the well-known recursivebipartitioning framework to improve the quality of the splitting heuristic. Finally, we propose a medium-grain tripartite graph model with the aim of a faster partitioning at the expense of increasing the total communication volume. Parallel experiments conducted on 10 realworld tensors on up to 1024 processors confirm the validity of the proposed hypergraph and graph models.

True Load Balancing for Matricized Tensor Times Khatri-Rao Product

“…To devise intelligent tensor partitioning models, sparse matrix partitioning community adapted well-known sparse matrix partitioning models for tensors. These models came in different granularities: coarse-grain [9], multi-dimensional cartesian model [11], fine-grain [9] and medium-grain [12]. The multidimensional cartesian model is derived from the hypergraph model proposed earlier for 2D checkerboard partitioning of sparse matrices [33], [34].…”

“…The fine-grain model can be considered as an extension of the fine-grain hypergraph model for 2D nonzero-based sparse matrix partitioning [33], [35], [36] to multi-dimensional tensor partitioning. The recent general medium-grain model [12] can be considered as an extension of the medium-grain model for 2D sparse matrix partitioning [37] to tensors. Among these, fine-grain model achieves the minimum communication volume as well as the best computational balance on the tensor nonzeros assigned to processors.…”

“…To address (i), several successful partitioning models [9], [10], [11], [12] have been proposed with the goal of reducing the communication cost of MTTKRP while maintaining a balance on its computational costs on all processors. To address (ii) and (iii), several storage formats [13], [14], [15] have been proposed, usually together with a new method to realize MTTKRP on the proposed format.…”

Accelerated Stochastic Gradient for Nonnegative Tensor Completion and Parallel Implementation

“…The works in [15], [16], and [17] make use of either the Map-Reduce programming model or the Spark engine. In [18], a hypergraph model for general medium-grain partitioning has been presented.…”