Anatomy of High-Performance Deep Learning Convolutions on SIMD Architectures

Abstract

Convolution layers are prevalent in many classes of deep neural networks, including Convolutional Neural Networks (CNNs) which provide state-of-the-art results for tasks like image recognition, neural machine translation and speech recognition. The computationally expensive nature of a convolution operation has led to the proliferation of implementations including matrix-matrix multiplication formulation, and direct convolution primarily targeting GPUs. In this paper, we introduce direct convolution kernels for x86 architectures, in particular for Xeon and Xeon Phi systems, which are implemented via a dynamic compilation approach. Our JIT-based implementation shows close to theoretical peak performance, depending on the setting and the CPU architecture at hand. We additionally demonstrate how these JIT-optimized kernels can be integrated into a lightweight multi-node graph execution model. This illustrates that single-and multi-node runs yield high efficiencies and high imagethroughputs when executing state-of-the-art image recognition tasks on CPUs.

A survey on evaluating and optimizing performance of Intel Xeon Phi

“…Loop-optimizations unrolling, 9,17,23,24,29,50,84,90 collapsing, 4,6,7,13,20,21,44,54 splitting 22,28 Blocking (tiling) in cache, 14,15,18,[20][21][22]27,39,44,52,54,69 registers 68,69 Compile-time optimizations using pre-computed values, 35,52 specifying array and loop bounds at compile time 6,54 Compute-related optimizations Reusing intermediate variables, 22,35 using conflict-detection instruction of AVX-512, 52,85 performing redundant computation to avoid data-communication or atomic operations 52,82 Array transpose 6, 79…”

“…Georganas et al 69 note that implementing convolution as GEMM leads to large memory footprint and memory bandwidth dependency. Hence, they implement convolution using direct convolution which avoids expensive memory accesses due to shuffle/scatter/gather operations.…”

“…Hence, they implement convolution using direct convolution which avoids expensive memory accesses due to shuffle/scatter/gather operations. Notice that Das et al 68 implement convolution as matrix-multiplication whereas Georganas et al 69 implement it as direct convolution. Figure 17A shows the forward-propagation code using direct convolution.…”

“…Crimi et al 87 Cache-related excessive cache misses, 17,54 lack of shared last level cache 14,29,69 Slow communication on Memory, 29,35,50 PCIe bus, 14,19,24,36,50,82,95 between two Phis in the same node 43 Core and performance-scalability related factors…”

Machine Learning Systems are Stuck in a Rut

“…where • denotes scalar multiplication, and O, I , and K are all 4-dimensional arrays of scalars. The resulting code is little more than 7 nested loops around a multiplyaccumulate operation, but array layout, vectorization, parallelization and caching are extremely important for performance [5].…”