Tensor-matrix products with a compressed sparse tensor

Sparse matricized tensor times Khatri-Rao product (MTTKRP) is one of the most computationally expensive kernels in sparse tensor computations. This work focuses on optimizing the MTTKRP operation on GPUs, addressing both performance and storage requirements. We begin by identifying the performance bottlenecks in directly extending the state-ofthe-art CSF (compressed sparse fiber) format from CPUs to GPUs. A significant challenge with GPUs compared to multicore CPUs is that of utilizing the much greater degree of parallelism in a load-balanced fashion for irregular computations like sparse MTTKRP. To address this issue, we develop a new storage-efficient representation for tensors that enables highperformance, load-balanced execution of MTTKRP on GPUs. A GPU implementation of sparse MTTKRP using the new sparse tensor representation is shown to outperform all currently known parallel sparse CPU and GPU MTTKRP implementations.

show abstract

“…Equation 6 reformulates Equation 4 for a row of Y . Smith et al [22] further optimize the computation by factoring out C as shown in 8.…”

Section: Sparse Mttkrpmentioning

confidence: 99%

Load-Balanced Sparse MTTKRP on GPUs

Nisa

Sukumaran-Rajam

et al. 2019

2019 IEEE International Parallel and Distributed Processing Symposium (IPDPS)

View full text Add to dashboard Cite

show abstract

“…Unless the factors are very sparse, this product is totally dense due to high fill-in and can easily require many times more memory than the original tensor. Due to the performance characteristics of the MTTKRP, researchers have focused on designing efficient implementations with respect to both execution time and memory [11], [2].…”

Section: Tensor Decompositionmentioning

confidence: 99%

“…SPLATT 2 is an open source software toolbox for sparse tensor factorization and related kernels [11], [12], [13]. SPLATT includes routines for computing least-squares CP, as well as constrained CP and CP with missing values (i.e., tensor completion).…”

mentioning

confidence: 99%

“…In this paper, we focus on SPLATT's shared-memory implementation of CP-ALS and leave the distributed implementation for future work. 2 SPLATT source code is available from https://github.com/ShadenSmith/splatt SPLATT uses a novel data structure for the storage of sparse tensors, a compressed sparse fiber (CSF), that addresses the memory/computation trade-off of computing tensor-matrix products [11]. To efficiently perform MTTKRP on tensors stored in CSF, SPLATT uses a custom shared-memory parallelized algorithm.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Parallel Sparse Tensor Decomposition in Chapel

Rolinger

Simon

Krieger

2018

2018 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)

View full text Add to dashboard Cite

In big-data analytics, using tensor decomposition to extract patterns from large, sparse multivariate data is a popular technique. Many challenges exist for designing parallel, high performance tensor decomposition algorithms due to irregular data accesses and the growing size of tensors that are processed. There have been many efforts at implementing shared-memory algorithms for tensor decomposition, most of which have focused on the traditional C/C++ with OpenMP framework. However, Chapel is becoming an increasingly popular programing language due to its expressiveness and simplicity for writing scalable parallel programs. In this work, we port a state of the art C/OpenMP parallel sparse tensor decomposition tool, SPLATT, to Chapel. We present a performance study that investigates bottlenecks in our Chapel code and discusses approaches for improving its performance. Also, we discuss features in Chapel that would have been beneficial to our porting effort. We demonstrate that our Chapel code is competitive with the C/OpenMP code for both runtime and scalability, achieving 83%-96% performance of the original code and near linear scalability up to 32 cores.

show abstract

“…Choi and Vishwanathan addressed this problem by using a compressed representation of a tensor for each mode to reduce the amount of floating point operations, especially for sparse tensors. An efficient parallel sparse tensor‐matrix multiplication that uses different compressed representation was also proposed by Smith et al Recently, Chen et al proposed an improved version of Phan and Cichocki's parallel PARAFAC that better reflects the dynamics of large tensors, which can be implemented on a GPU cluster.…”

Section: Introductionmentioning

confidence: 99%

Distributed geometric nonnegative matrix factorization and hierarchical alternating least squares–based nonnegative tensor factorization with the MapReduce paradigm

Zdunek

Fonał

2018

Concurrency and Computation

View full text Add to dashboard Cite

Summary Nonnegative matrix factorization and its multilinear extension known as nonnegative tensor factorization are commonly used methods in machine learning and data analysis for feature extraction and dimensionality reduction for nonnegative high‐dimensional data. Dimensionality reduction for massive amounts of data usually involves distributed computation across multi‐node computer architectures. In this study, we propose various computational strategies for parallel and distributed computation of the latent factors in both factorization models, all of which are based on partitioning the computational tasks according to the MapReduce paradigm. We extend the previously reported distributed hierarchical alternating least squares algorithm to the multi‐way array factorization model, where we assume that the observed multi‐way data can be partitioned into chunks along one mode. Moreover, we propose a new geometry‐based distributed computational strategy for solving nonnegative matrix factorization problems. Numerical experiments performed using various large‐scale data sets demonstrated that these algorithms are efficient and robust to noisy data.

show abstract

Tensor-matrix products with a compressed sparse tensor

Cited by 111 publications

References 14 publications

Load-Balanced Sparse MTTKRP on GPUs

Load-Balanced Sparse MTTKRP on GPUs

Parallel Sparse Tensor Decomposition in Chapel

Distributed geometric nonnegative matrix factorization and hierarchical alternating least squares–based nonnegative tensor factorization with the MapReduce paradigm

Contact Info

Product

Resources

About