A Scalable Near-Memory Architecture for Training Deep Neural Networks on Large In-Memory Datasets

Abstract

Most investigations into near-memory hardware accelerators for deep neural networks have primarily focused on inference, while the potential of accelerating training has received relatively little attention so far. Based on an in-depth analysis of the key computational patterns in state-of-the-art gradient-based training methods, we propose an efficient near-memory acceleration engine called NTX that can be used to train state-of-the-art deep convolutional neural networks at scale. Our main contributions are: (i) a loose coupling of RISC-V cores and NTX co-processors reducing offloading overhead by 7× over previously published results; (ii) an optimized IEEE 754 compliant data path for fast high-precision convolutions and gradient propagation; (iii) evaluation of near-memory computing with NTX embedded into residual area on the Logic Base die of a Hybrid Memory Cube; and (iv) a scaling analysis to meshes of HMCs in a data center scenario. We demonstrate a 2.7× energy efficiency improvement of NTX over contemporary GPUs at 4.4× less silicon area, and a compute performance of 1.2 Tflop/s for training large state-of-the-art networks with full floating-point precision. At the data center scale, a mesh of NTX achieves above 95% parallel and energy efficiency, while providing 2.1× energy savings or 3.1× performance improvement over a GPU-based system.

Stream Semantic Registers: A Lightweight RISC-V ISA Extension Achieving Full Compute Utilization in Single-Issue Cores

“…Each lane consists of a First-In First-Out (FIFO) queue to buffer read and write data. An address generator based on the one presented by Schuiki et al [8] and Conti et al [9] assigns memory addresses to the stream-based accesses performed by the core. The lane can be put into read mode, in which case the address generator is used to fetch data from memory and store it in the FIFO.…”

NTX: An Energy-efficient Streaming Accelerator for Floating-point Generalized Reduction Workloads in 22 nm FD-SOI

“…In the following we provide a short overview of the NTX architecture outlined in more details in [12]. The Logic Base (LoB) of a HMC offers a unique opportunity to introduce a PiM as depicted in Figure 1.…”

“…We combine a small 32 bit RISC-V processor core (RV32IMC) [18] with multiple NTX co-processors. Both operate on shared 64 kB TCDM (reduced from 128 kB in [12]). The memory is divided into 32 banks that are connected to the processors via an interconnect offering single-cycle access latency.…”

“…The FMAC unit allows NTX to handle common matrix operations such as inner/outer products and vector additions/multiplications. An additional comparator, index counter, and ALU register enable various additional commands such as finding minima/maxima, ReLU, thresholding and masking, and memcpy/memset [12].…”

Eden

“…Improvements to accelerator efficiency [27,85], DNN-optimized GPU kernels [23,28], and libraries designed to efficiently leverage instruction set extensions [23,83] have improved the computational efficiency of DNN evaluation. However, improving the memory efficiency of DNN evaluation is an on-going challenge [13,29,37,54,107,148]. The memory intensity of DNN inference is increasing, and the sizes of state-of-art DNNs have grown dramatically in recent years.…”