End-to-End Optimization of Deep Learning Applications

Abstract

The irregularity of recent Convolutional Neural Network (CNN) models such as less data reuse and parallelism due to the extensive network pruning and simplification creates new challenges for FPGA acceleration. Furthermore, without proper optimization, there could be significant overheads when integrating FPGAs into existing machine learning frameworks like TensorFlow. Such a problem is mostly overlooked by previous studies. However, our study shows that a naive FPGA integration into TensorFlow could lead to up to 8.45× performance degradation. To address the challenges mentioned above, we propose several SW/HW co-design approaches to perform the end-to-end optimization of deep learning applications. We present a flexible and composable architecture called FlexCNN. It can deliver high computation efficiency for different types of convolution layers using techniques including dynamic tiling and data layout optimization. FlexCNN is further integrated into the TensorFlow framework with a fully-pipelined software-hardware integration flow. This alleviates the high overheads of TensorFlow-FPGA handshake and other non-CNN processing stages. We use OpenPose, a popular CNN-based application for human pose recognition, as a case study. Experimental results show that with the FlexCNN architecture optimizations, we can achieve 2.3× performance improvement. The pipelined integration stack leads to a further 5× speedup. Overall, the SW/HW co-optimization produces a speedup of 11.5× and results in an end-to-end performance of 23.8FPS for OpenPose with floating-point precision, which is the highest performance reported for this application on FPGA in the literature.

AutoDSE: Enabling Software Programmers to Design Efficient FPGA Accelerators

“…However, they assume that the performance/area changes monotonically by modifying an individual design parameter, which is not a valid assumption as we explained in Challenge 2 of Section 1. To increase the accuracy of the estimation model, a number of other studies restrict the target application to those that have a welldefined accelerator micro-architecture template [9,14,15,40,45,58], a specific application [55,61], or a particular computation pattern [10,28,37]; hence, they lose generality.…”

“…The main enabler of this feature is the ability to iteratively re-optimize the micro-architecture quickly just by inserting synthesis directives in the form of pragmas instead of re-writing the low-level behavioral description of the design. Because of the reduced code development cycle and the shorter turn-around times, HLS has been rapidly adopted by both academia and industry [3,20,30,45,49,65]. In fact, Code 1 shows an intuitive HLS C implementation of one forward path of a Convolutional Neural Network (CNN) on Xilinx FPGAs.…”

EF-Train: Enable Efficient On-device CNN Training on FPGA through Data Reshaping for Online Adaptation or Personalization

“…Cafeine [39] combined both on-chip and of-chip data reorganizations for the convolutional matrixmultiplication representation to maximize the underlying memory bandwidth utilization. FlexCNNe [26] further optimized data layout optimizations on the concatenation layers. However, all these works [6]- [26] are based on the computation and memory access pattern in the inference phase which only has FP.…”

“…FlexCNNe [26] further optimized data layout optimizations on the concatenation layers. However, all these works [6]- [26] are based on the computation and memory access pattern in the inference phase which only has FP. The training phase involves FP, BP, and WU where their data access pattern for output features, input features, and weights are diferent.…”

HALF: Holistic Auto Machine Learning for FPGAs

“…Contrary, FINN uses an High-level Synthesis (HLS) hardware library [20] of hardware layers and components that are used to generate streaming architectures customized for each network. Other tools for automatic hardware generation are FlexCNN [21], integrating an FPGA implementation framework into Tensorflow and DNNBuilder [22], which uses software-hardware co-design to perform an end-to-end optimization of deep learning applications.…”