Advancing Direct Convolution using Convolution Slicing Optimization and ISA Extensions

Abstract

Convolution is one of the most computationally intensive operations that must be performed for machine-learning model inference. A traditional approach to compute convolutions is known as the Im2Col + BLAS method. This paper proposes SConv: a direct-convolution algorithm based on a MLIR/LLVM code-generation toolchain that can be integrated into machine-learning compilers . This algorithm introduces: (a) Convolution Slicing Analysis (CSA) -a convolution-specific 3D cache-blocking analysis pass that focuses on tile reuse over the cache hierarchy; (b) Convolution Slicing Optimization (CSO) -a code-generation pass that uses CSA to generate a tiled direct-convolution macro-kernel; and (c) Vector-Based Packing (VBP) -an architecture-specific optimized input-tensor packing solution based on vector-register shift instructions for convolutions with unitary stride. Experiments conducted on 393 convolutions from full ONNX-MLIR machine-learning models indicate that the elimination of the Im2Col transformation and the use of fast packing routines result in a total packing time reduction, on full model inference, of 2.0x -3.9x on Intel x86 and 3.6x -7.2x on IBM POWER10. The speed-up over an Im2Col + BLAS method based on current BLAS implementations for end-to-end machine-learning model inference is in the range of 9% -25% for Intel x86 and 10% -42% for IBM POWER10 architectures. The total convolution speedup for model inference is 12% -27% on Intel x86 and 26% -46% on IBM POWER10. SConv also outperforms BLAS GEMM, when computing pointwise convolutions, in more than 83% of the 219 tested instances.

Optimizing Direct Convolutions on ARM Multi-Cores

“…While promising, LIBXSMM has two fundamental drawbacks. First, its data layout design is incompatible with the common data layouts (i.e., 𝑁𝐶𝐻𝑊 or 𝑁 𝐻𝑊 𝐶) 1 used in mainstreamed deep learning (DL) frameworks [8,13,30]. Therefore, integrating the BRGEMM routines into DL frameworks requires either code refactoring to the underlying DL framework or introducing a format conversion stage at the user code when calling and exiting each CONV operator.…”