Adaptive sparse tiling for sparse matrix multiplication

Abstract

Tiling is a key technique for data locality optimization and is widely used in high-performance implementations of dense matrix-matrix multiplication for multicore/manycore CPUs and GPUs. However, the irregular and matrix-dependent data access pattern of sparse matrix multiplication makes it challenging to use tiling to enhance data reuse. In this paper, we devise an adaptive tiling strategy and apply it to enhance the performance of two primitives: SpMM (product of sparse matrix and dense matrix) and SDDMM (sampled dense-dense matrix multiplication). In contrast to studies that have resorted to non-standard sparse-matrix representations to enhance performance, we use the standard Compressed Sparse Row (CSR) representation, within which intra-row reordering is performed to enable adaptive tiling. Experimental evaluation using an extensive set of matrices from the Sparse Suite collection demonstrates significant performance improvement over currently available state-ofthe-art alternatives.

Accelerating Sparse CNN Inference on GPUs with Performance-Aware Weight Pruning

“…Third, it introduces load imbalance as different kernels are likely to have different workloads. In fact, the throughput of cuBLAS's single-precision GEMM can be up to 8,000 GFLOPS [2] on an Nvidia P100 GPU, whereas the throughput of the state-of-theart implementation of single-precision SpMM is about 800 GFLOPS on the same device [23,25]. Because of the huge throughput gap, it is a major challenge to improve the performance of CNN inference on GPUs while retaining its accuracy.…”

“…As explained in the background section, sparse convolutions can be implemented as SpMM. Although previous works have studied SpMM on GPUs [22,23,25], their optimization techniques mainly target large sparse matrices with at least 10,000 rows and columns that are found in scientific computing applications, and they can not deliver good performance for sparse convolutions where the number of convolution kernels is usually smaller than 1000. In fact, we adopted a state-of-the-art implementation of SpMM from [25] for sparse convolution with real-world pruned models from [46], and we found that the sparse convolutions do not run much faster (and can even be slower) compared with the original dense convolutions implemented as GEMM.…”

“…After the sparse matrix is divided into multiple dense blocks and a sparse block, we use GEMM for the computation with the dense blocks and use a row-wise implementation of SpMM [23] for the computation with the sparse block. The CUDA code can be easily generated with a GEMM-based implementation of dense convolutions and an SpMM kernel.…”

“…If an element in the removed row happens to be the only element in its column, then the performance benefit is increased by 2 . For the sparse block, with the row-wise SpMM implementation [23], the results in each row are accumulated in registers with perfect data reuse, but there is little data reuse when accessing the input among the rows. If we remove a weight in the sparse block, we save the read of a row from the input matrix, and the benefit is 2 .…”

“…However, existing SpMM implementations cannot deliver satisfactory performance for sparse convolutions. The reason is that these implementations mainly target large sparse matrices where data locality can be effectively improved by data reorganization [22,23,25], whereas the sparse matrices of the pruned convolution kernels are small and have less freedom of data reorganization. We observe that the main performance bottleneck in small sparse matrix multiplications is the control-flow instructions instead of data locality.…”