Software for Sparse Tensor Decomposition on Emerging Computing Architectures

Abstract

In this paper, we develop software for decomposing sparse tensors that is portable to and performant on a variety of multicore, manycore, and GPU computing architectures. The result is a single code whose performance matches optimized architecture-specific implementations.The key to a portable approach is to determine multiple levels of parallelism that can be mapped in different ways to different architectures, and we explain how to do this for the matricized tensor times Khatri-Rao product (MTTKRP) which is the key kernel in canonical polyadic tensor decomposition. Our implementation leverages the Kokkos framework, which enables a single code to achieve high performance across multiple architectures that differ in how they approach fine-grained parallelism. We also introduce a new construct for portable thread-local arrays, which we call compile-time polymorphic arrays. Not only are the specifics of our approaches and implementation interesting for tuning tensor computations, but they also provide a roadmap for developing other portable highperformance codes. As a last step in optimizing performance, we modify the MTTKRP algorithm itself to do a permuted traversal of tensor nonzeros to reduce atomic-write contention. We test the performance of our implementation on 16-and 68-core Intel CPUs and the K80 and P100 NVIDIA GPUs, showing that we are competitive with state-of-the-art architecture-specific codes while having the advantage of being able to run on a variety of architectures.

True Load Balancing for Matricized Tensor Times Khatri-Rao Product

“…In the literature, there are various CP-ALS implementations adopting different parallelism paradigms [13], [17], [28], [29], [30], [31], [32]. On distributed-memory systems, DMS [17] is the most commonly-used implementation.…”

Stochastic Gradients for Large-Scale Tensor Decomposition

“…Bader and Kolda [3] consider both dense and sparse Y tensors, showing that the cost is O(rn d ) for dense Y and O(r nnz(Y)) for sparse Y. Phan, Tichavsky, and Cichocki [39] propose methods to reuse partial computations when computing the MTTKRP for all d modes in sequence. Much recent work has focused on more efficient representations of sparse tensors and parallel MTTKRP computations [44,24,29,40]. There is also continued work on improving the efficiency of dense MTTKRP calculations [20,5].…”

“…In terms of implementations, an interesting consequence of sampling in the context of parallel tensor decomposition [44,24,29,40] is that we can reduce the computation and/or communication by sampling only a subset of the entries. Moreover, we may be able to stratify the samples in such a way that is amenable to more structured communications.…”

PASTA: a parallel sparse tensor algorithm benchmark suite

“…Besides, their performance e ciency is low because of MATLAB environment. Recently, many other highly performance e cient libraries emerge, such as SPLATT [130], Cyclops Tensor Framework (CTF) [132], DFacTo [24], GigaTensor [65], HyperTensor [69], GenTen [110], to name a few. However, these libraries are speci c to one or two particular sparse tensor operations, this violates the application diversity requirement.…”