No abstract

No abstract

深度学习这一概念在 2006 年被人提出, 通过增加人工神经网络的深度增加算法提取数据特征 的能力. 卷积神经网络 (convolutional neural network, CNN) 作为深度学习中被广泛采用的算法, 被 人们实现在大量的应用中, 例如人脸识别、车道检测、立体视觉匹配、语音识别 [1∼4] , 如图 1 所示. 但是卷积神经网络的计算量庞大, 这就要求设计与其计算量相对应的加速器. 图形处理器 (graphic processing units, GPU)、现场可编程逻辑门阵列 (field-programmable gate array, FPGA)、专用集成电 路 (application specific integrated circuit, ASIC) 是常用的神经网络加速器. 在这些加速器中, FPGA 因为它的高度并行计算、低功耗、可重复编程的特性而被人们广泛采用. 此外, 高层次综合 (high level synthesis, HLS) 为软件开发人员提供了良好的编程环境, 降低了 FPGA 的开发周期, 使得 FPGA 编程 更加有效率.

Deep neural networks (DNNs) are increasingly deployed in various image recognition and natural language processing applications. The continuous demand for accuracy and high performance has led to innovations in DNN design and a proliferation of new operators. However, existing DNN training frameworks such as PyTorch and TensorFlow only support a limited range of operators and rely on hand-optimized libraries to provide efficient implementations for these operators. To evaluate novel neural networks with new operators, the programmers have to either replace the holistic new operators with existing operators or provide low-level implementations manually. Therefore, a critical requirement for DNN training frameworks is to provide high-performance implementations for the neural networks containing new operators automatically in the absence of efficient library support. In this paper, we introduce NeoFlow, which is a flexible framework for enabling efficient compilation for high-performance DNN training. NeoFlow allows the programmers to directly write customized expressions as new operators to be mapped to graph representation and low-level implementations automatically, providing both high programming productivity and high performance. First, NeoFlow provides expression-based automatic differentiation to support customized model definitions with new operators. Then, NeoFlow proposes an efficient compilation system that partitions the neural network graph into subgraphs, explores optimized schedules, and generates high-performance libraries for subgraphs automatically. Finally, NeoFlow develops an efficient runtime system to combine the compilation and training as a whole by overlapping their execution. In the experiments, we examine the numerical accuracy and performance of NeoFlow. The results show that NeoFlow can achieve similar or even better performance at the operator and whole graph level for DNNs compared to deep learning frameworks. Especially, for novel networks training, the geometric mean speedups of NeoFlow to PyTorch, TensorFlow, and CuDNN are 3.16X, 2.43X, and 1.92X, respectively.