High Performance and Portable Convolution Operators for Multicore Processors

Abstract

The considerable impact of Convolutional Neural Networks on many Artificial Intelligence tasks has led to the development of various high performance algorithms for the convolution operator present in this type of networks. One of these approaches leverages the im2col transform followed by a general matrix multiplication (gemm) in order to take advantage of the highly optimized realizations of the gemm kernel in many linear algebra libraries. The main problems of this approach are 1) the large memory workspace required to host the intermediate matrices generated by the im2col transform; and 2) the time to perform the im2col transform, which is not negligible for complex neural networks. This paper presents a portable high performance convolution algorithm based on the BLIS realization of the gemm kernel that avoids the use of the intermediate memory by taking advantage of the BLIS structure. In addition, the proposed algorithm eliminates the cost of the explicit im2col transform, while maintaining the portability and performance of the underlying realization of gemm in BLIS.

Communication Bounds for Convolutional Neural Networks

“…We will focus on four algorithms in particular: im2col, blocking, Winograd convolutions, and FFT convolutions. im2col [14], Winograd [13], and FFT techniques [17] for performing convolutions are all well documented in the literature. We will focus on designing improved blocking algorithms.…”

Efficient and portable GEMM-based convolution operators for deep neural network training on multicore processors

“…In this paper, we extend our previous work in [26] to obtain an efficient integration of the convolution operators in a framework for distributed training of DNNs on clusters of computers equipped with multicore processors. In particular, this work makes the following contributions:…”

“…Unfortunately, there are two major problems with this approach: 1) a large memory workspace is required to host the intermediate matrix generated by the im2col transform; and, especially for training, 2) the time to apply this transform is not negligible for complex CNNs. In [26], we presented a portable high performance convolution algorithm based on the BLIS [33] realization of gemm, named convgemm, that practically eliminates the memory and time cost of the im2col transform, while maintaining the portability and performance of the underlying realization of the BLIS gemm for multicore processors.…”

“…• For the computation of the downstream gradients with respect to the inputs, we adopt an approach similar to that in [26] to integrate a col2im operator within the BLIS realization of gemm, yielding a deconvgemm operator that avoids the creation of a large intermediate matrix, while maintaining the portability and, to a certain extent, performance.…”

“…In Section 2, we offer a short review of supervised iterative training for DNNs via the stochastic gradient descent (SGD), paying special attention to the convolution layers appearing in the forward-backward iteration. In Section 3, we first discuss the BLIS kernel for gemm and explain how this was leveraged in [26] to obtain an efficient realization of the convolution operator in the forward pass of the training process. In the same section, we next discuss how to extend that idea to the more complex backward propagation stage, which constitutes one of the major contributions of this work.…”