Characterising Across-Stack Optimisations for Deep Convolutional Neural Networks

Abstract

Convolutional Neural Networks (CNNs) are extremely computationally demanding, presenting a large barrier to their deployment on resource-constrained devices. Since such systems are where some of their most useful applications lie (e.g. obstacle detection for mobile robots, vision-based medical assistive technology), significant bodies of work from both machine learning and systems communities have attempted to provide optimisations that will make CNNs available to edge devices. In this paper we unify the two viewpoints in a Deep Learning Inference Stack and take an across-stack approach by implementing and evaluating the most common neural network compression techniques (weight pruning, channel pruning, and quantisation) and optimising their parallel execution with a range of programming approaches (OpenMP, OpenCL) and hardware architectures (CPU, GPU). We provide comprehensive Pareto curves to instruct trade-offs under constraints of accuracy, execution time, and memory space.

Optimising Hardware Accelerated Neural Networks with Quantisation and a Knowledge Distillation Evolutionary Algorithm

“…Previous work shows that combining compression methods can achieve superior performance compared with using them in isolation, e.g., combining pruning and knowledge distillation [46]. The approach in [47] shows that distilling knowledge to shallower quantised architectures can achieve accuracy comparable with state-of-the-art full-precision models.…”

Optimizing Grouped Convolutions on Edge Devices

“…In this paper, we propose grouped spatial pack convolutions (GSPC), for the common NCHW data layout 1 . We modify and extend the spatial pack convolutions (SPC) algorithm described in [11], which does not cover grouped convolutions.…”

“…Figure 3 illustrates the GSPC algorithm with an example. We use tile sizes T O = T I = 2, as these are the maximum values allowed by the constraints (1). The initial data layout is shown on the left, with the channels split by group for clarity.…”

“…That is, many neural architecture compression techniques may not work as expected at the system level where one of the main metrics considered is the inference time. Turner et al [1] demonstrated that compression at the neural architecture level may have negative effects further down the Deep Learning Inference Stack, depending on the choices of algorithmic transformation and the target hardware device.…”

DNNTune

“…DNN benchmarking. Turner et al [57] implemented several common DNN compression techniques (weight pruning, channel pruning, and quantization) and evaluated the accuracy, execution time, and memory space on both CPU and GPU. They found that channel pruning can greatly reduce the execution time while weight pruning cannot.…”