A three-dimensional approach to parallel matrix multiplication

Abstract

A three-dimensional (3D) matrix multiplication algorithm for massively parallel processing systems is presented. The P processors are con gured as a \virtual" processing cube with dimensions p 1 , p 2 , and p 3 proportional to the matrices' dimensions|M, N, and K. Each processor performs a single local matrix multiplication of size M=p 1 N=p 2 K=p 3. Before the local computation can be carried out, each subcube must receive a single submatrix of A and B. After the single matrix multiplication has completed, K=p 3 submatrices of this product must be sent to their respective destination processors and then summed together with the resulting matrix C. The 3D parallel matrix multiplication approach has a factor P 1=6 less communication than the 2D parallel algorithms. This algorithm has been implemented on IBM POWERparallel T M SP2 T M systems (up to 216 nodes) and has yielded close to the peak performance of the machine. The algorithm has been combined with Winograd's variant of Strassen's algorithm to achieve performance which exceeds the theoretical peak of the system. (We assume the MFLOPS rate of matrix multiplication to be 2MNK.

A massively parallel tensor contraction framework for coupled-cluster computations

“…If the matrices are square and of size S, and there is enough memory for only O(1) copies of the matrices (S = Θ(M · p)), matrix multiplication can be performed on a 2D grid of processors in a communication-optimal fashion. In particular, blocked Cannon's algorithm [9] and SUMMA [1,53] achieve the bandwidth cost…”

“…Given an unlimited amount of memory, it is possible to replicate the data to reduce the communication cost, which is realized by a 3D matrix multiplication algorithm [32,12,1,2,23] with a resulting bandwidth cost of…”

“…The distributed algorithm for tensor contractions used by CTF is a combination of replication, as done in 3D and 2.5D algorithms [50] and a nested SUMMA [1,53] algorithm. If the dimensions of two tensors with the same contraction index are mapped onto different torus dimensions, a SUMMA algorithm is done on the plane defined by the two torus dimensions.…”

“…Additionally CC is rigorously size-extensive [5] and easily extensible to excited states [51], derivatives [45,52], and properties [33]. This paper focuses on the fundamental kernels of coupled-cluster calculations -tensor contractions -and documents a completely new set of parallel algorithms implemented as a distributed-memory tensor contraction framework, Cyclops Tensor Framework (CTF) 1 . CTF has enabled coupled-cluster with excitations of up to three electrons (CCSDT) to scale on state-of-the-art architectures while achieving a high degree of efficiency in computation, communication and storage.…”

Matrix Multiplication on Multidimensional Torus Networks

“…We do not consider algorithms that replicate data (e.g. 3D matrix multiplication [1]). However, the extension is natural, given the replication approach presented in [11].…”