A Highly Parallel and Energy Efficient Three-Dimensional Multilayer CMOS-RRAM Accelerator for Tensorized Neural Network

Abstract

It is a grand challenge to develop highly-parallel yet energy-efficient machine learning hardware accelerator. This paper introduces a 3D multi-layer CMOS-RRAM accelerator for tensorized neural network (TNN). Highly parallel matrix-vector multiplication can be performed with low power in the proposed 3D multi-layer CMOS-RRAM accelerator. The adoption of tensorization can significantly compress the weight matrix of neural network using much fewer parameters. Simulation results using the benchmark MNIST show that the proposed accelerator has 1.283× speed-up, 4.276× energy-saving and 9.339× area-saving compared to 3D CMOS-ASIC implementation; and 6.37× speedup and 2612× energy-saving compared to 2D CPU implementation. In addition, 14.85× model compression can be achieved by tensorization with acceptable accuracy loss.

Neuromorphic Spiking Neural Networks and Their Memristor-CMOS Hardware Implementations

“…Many works have proposed including ReRAM memristive memory crossbars to implement Matrix-Vector-Multiplication Units in computer architectures to accelerate Neural Network applications [150,151,152,153,154,155] demonstrating great benefits in power consumption levels. PRIME [151] and RESPARC [150] report simulations of energy savings compared to fully CMOS Neural Processors Units in the order of 103 depending on the particular neural network architecture.…”

Editorial: Spiking Neural Network Learning, Benchmarking, Programming and Executing

“…In recent years, hardware oriented DNN compression techniques have been proposed that offer significant memory saving and hardware acceleration (Han et al, 2015a(Han et al, , 2016Zhang et al, 2016;Huang et al, 2017;Aimar et al, 2018). At present, many compression techniques are proposed that provide a trade-off between processing efficiency and application accuracy (Han et al, 2015b;Novikov et al, 2015;Zhou et al, 2016).…”

A Survey of ReRAM-Based Architectures for Processing-In-Memory and Neural Networks

“…Algorithm genetic algorithm [20], greedy algorithm [18,69], simulated annealing [70], weighted bipartite-matching algorithm [47], graph clustering [67] Comparison of ReRAM-based design with FPGA execution [52,56,62,70] GPU execution [19,24,41,46,52,53,57,59,61,62] CPU execution [14,19,41,49,53,54,57,62,65,71,72] [19,41]…”

“…Specifically, their NN-based designs achieve magnitude-order higher energy and area efficiency compared to CPU and GPU. Huang et al [19] presented a 3D CMOS-ReRAM based accelerator for TNNs (tensor neural networks). As shown in Figure 6, their design has three layers: two-layers of ReRAM crossbar and one-layer of CMOS circuitry.…”

“…For example, since the CONV (convolution) operation in convolutional neural networks (CNNs) involves MVM, and CONV layers account for more than 95% of computations in CNNs, ReRAM-based processing engine can boost the efficiency of CNNs significantly [12,13]. These factors have motivated researchers to implement a variety of ML/NN architectures on ReRAM, such as multi-layer perceptron [14,15], CNN [16][17][18], tensorized NN [19] and auto-associative memory [14].…”