Block convolution: Towards memory-efficient inference of large-scale CNNs on FPGA

Abstract

Deep convolutional neural networks have achieved remarkable progress in recent years. However, the large volume of intermediate results generated during inference poses a significant challenge to the accelerator design for resourceconstraint FPGA. Due to the limited on-chip storage, partial results of intermediate layers are frequently transferred back and forth between on-chip memory and off-chip DRAM, leading to a non-negligible increase in latency and energy consumption. In this paper, we propose block convolution, a hardware-friendly, simple, yet efficient convolution operation that can completely avoid the off-chip transfer of intermediate feature maps at runtime. The fundamental idea of block convolution is to eliminate the dependency of feature map tiles in the spatial dimension when spatial tiling is used, which is realized by splitting a feature map into independent blocks so that convolution can be performed separately on individual blocks. We conduct extensive experiments to demonstrate the efficacy of the proposed block convolution on both the algorithm side and the hardware side. Specifically, we evaluate block convolution on 1) VGG-16, ResNet-18, ResNet-50, and MobileNet-V1 for ImageNet classification task; 2) SSD, FPN for COCO object detection task, and 3) VDSR for Set5 single image super-resolution task. Experimental results demonstrate that comparable or higher accuracy can be achieved with block convolution. We also showcase two CNN accelerators via algorithm/hardware co-design based on block convolution on memory-limited FPGAs, and evaluation shows that both accelerators substantially outperform the baseline without offchip transfer of intermediate feature maps.

Recent advances in efficient computation of deep convolutional neural networks

“…Since the majority of the computations in a network are matrix-matrix/matrix-vector multiplication, it is critical to deal with the massive nested loops to achieve high throughput. Loop optimization is one of the most frequently adopted techniques in accelerator design [92,56,73,2,88,49], including loop tiling, loop unrolling, loop interchange, etc. Loop tiling is used to divide all of the data into multiple small blocks in order to alleviate the pressure of onchip storage [56,2,64], while loop unrolling attempts to improve the parallelism of the computing engine for high speed [56,64].…”

“…[70] designed a flexible data buffing scheme to reduce bandwidth requirements, and [2] and [88] proposed a fusion-based method to reduce off-chip traffic. Most recently, [49] presented a block-based convolution that can completely avoid offchip transfers of intermediate data in VGG-16 with high throughput.…”

Energy-Efficient Architecture for CNNs Inference on Heterogeneous FPGA

“…However, as discussed in [33], in state-of-the-art deep CNNs, CONVs consume most of the computational time, thus becoming one of the most critical tasks responsible for limiting reachable speed performances. For this reason, the design of hardware parallel convolutional engines suitable for the inference of deep CNNs in high-performance low-power applications has recently received a great deal of attention [26][27][28][29][30][31]34]. The most exploited design techniques aim to boost the achievable performances by increasing the level of parallelism with which data is processed [28][29][30][31]34].…”

“…For this reason, the design of hardware parallel convolutional engines suitable for the inference of deep CNNs in high-performance low-power applications has recently received a great deal of attention [26][27][28][29][30][31]34]. The most exploited design techniques aim to boost the achievable performances by increasing the level of parallelism with which data is processed [28][29][30][31]34]. Indeed, as is visible in Figure 1a, most of the computations involved in a convolutional layer are independent from each other, offering the possibility of parallelizing the operations within the kernel and across both ifmaps and ofmaps.…”

“…This allows energy efficiencies higher than traditional GPUs to be achieved [25]. In the literature, several FPGA-based designs have been presented to accelerate the inference of 16- [26][27][28][29] and 8-bit [30,31] fixed-point quantized CNNs. However, while some of them [28][29][30][31] were optimized to achieve high performanc by directly increasing the level of parallelism with which feature maps are processed, others [26,27] were mainly oriented towards low-power applications.…”