Making Memristive Neural Network Accelerators Reliable

Feinberg, Benjamin A.; Wang, Shibo; Ïpek, Engin

doi:10.1109/hpca.2018.00015

Cited by 97 publications

(58 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, ASICs are based on CMOS technology and therefore suffer from the interconnect problem-even in highly optimized architectures where data is stored in register files close to the logic units, a majority of the energy consumption comes from data movement, not logic [13,16]. Analog crossbar arrays based on CMOS gates [18] or memristors [19,20] promise better performance, but as analog electronic devices, they suffer from calibration issues and limited accuracy [21].Photonic approaches can greatly reduce both the logic and data-movement energy by performing (the linear part of) each neural-network layer in a passive, linear optical circuit. This allows the linear step is performed at high speed with no energy consumption beyond transmitter and receiver energies.…”

mentioning

confidence: 99%

Large-Scale Optical Neural Networks Based on Photoelectric Multiplication

Hamerly¹,

Bernstein²,

Sludds³

et al. 2019

Phys. Rev. X

284

243

View full text Add to dashboard Cite

Recent success in deep neural networks has generated strong interest in hardware accelerators to improve speed and energy consumption. This paper presents a new type of photonic accelerator based on coherent detection that is scalable to large (N 10 6 ) networks and can be operated at high (GHz) speeds and very low (sub-aJ) energies per multiply-and-accumulate (MAC), using the massive spatial multiplexing enabled by standard free-space optical components. In contrast to previous approaches, both weights and inputs are optically encoded so that the network can be reprogrammed and trained on the fly. Simulations of the network using models for digit-and image-classification reveal a "standard quantum limit" for optical neural networks, set by photodetector shot noise. This bound, which can be as low as 50 zJ/MAC, suggests performance below the thermodynamic (Landauer) limit for digital irreversible computation is theoretically possible in this device. The proposed accelerator can implement both fully-connected and convolutional networks. We also present a scheme for back-propagation and training that can be performed in the same hardware. This architecture will enable a new class of ultra-low-energy processors for deep learning.In recent years, deep neural networks have tackled a wide range of problems including image analysis [1], natural language processing [2], game playing [3], physical chemistry [4], and medicine [5]. This is not a new field, however. The theoretical tools underpinning deep learning have been around for several decades [6,7,8]; the recent resurgence is driven primarily by (1) the availability of large training datasets [9], and (2) substantial growth in computing power [10] and the ability to train networks on GPUs [11]. Moving to more complex problems and higher network accuracies requires larger and deeper neural networks, which in turn require even more computing power [12]. This motivates the development of special-purpose hardware optimized to perform neural-network inference and training [13].To outperform a GPU, a neural-network accelerator must significantly lower the energy consumption, since the performance of modern microprocessors is limited by on-chip power [14]. In addition, the system must be fast, programmable, scalable to many neurons, compact, and ideally compatible with training as well as inference. Application-specific integrated circuits (ASICs) are one obvious candidate for this task. Stateof-the-art ASICs can reduce the energy per multiply-and-accumulate (MAC) from 20 pJ/MAC for modern GPUs [15] to around 1 pJ/MAC [16,17]. However, ASICs are based on CMOS technology and therefore suffer from the interconnect problem-even in highly optimized architectures where data is stored in register files close to the logic units, a majority of the energy consumption comes from data movement, not logic [13,16]. Analog crossbar arrays based on CMOS gates [18] or memristors [19,20] promise better performance, but as analog electronic devices, they suffer from calibration issues and li...

show abstract

mentioning

confidence: 99%

Large-Scale Optical Neural Networks Based on Photoelectric Multiplication

Hamerly¹,

Bernstein²,

Sludds³

et al. 2019

Phys. Rev. X

284

243

View full text Add to dashboard Cite

show abstract

“…While we try to optimize synaptic resources and reduce "wastage" by minimizing unused synapses, it might be possible to reuse these synapses to provide a degree of fault tolerance in the hardware by providing redundancy. Current explorations of fault tolerance mostly show reduction in performance degradation after retraining a neural network with faults (Lee et al, 2014;Feinberg et al, 2018); however, there might be scope to optimize the fault tolerance by providing some extra synapses. We feel this is an important avenue of future work.…”

Section: Discussionmentioning

confidence: 99%

HFNet: A CNN Architecture Co-designed for Neuromorphic Hardware With a Crossbar Array of Synapses

et al. 2020

View full text Add to dashboard Cite

The hardware-software co-optimization of neural network architectures is a field of research that emerged with the advent of commercial neuromorphic chips, such as the IBM TrueNorth and Intel Loihi. Development of simulation and automated mapping software tools in tandem with the design of neuromorphic hardware, whilst taking into consideration the hardware constraints, will play an increasingly significant role in deployment of system-level applications. This paper illustrates the importance and benefits of co-design of convolutional neural networks (CNN) that are to be mapped onto neuromorphic hardware with a crossbar array of synapses. Toward this end, we first study which convolution techniques are more hardware friendly and propose different mapping techniques for different convolutions. We show that, for a seven-layered CNN, our proposed mapping technique can reduce the number of cores used by 4.9-13.8 times for crossbar sizes ranging from 128 × 256 to 1,024 × 1,024, and this can be compared to the toeplitz method of mapping. We next develop an iterative co-design process for the systematic design of more hardware-friendly CNNs whilst considering hardware constraints, such as core sizes. A python wrapper, developed for the mapping process, is also useful for validating hardware design and studies on traffic volume and energy consumption. Finally, a new neural network dubbed HFNet is proposed using the above co-design process; it achieves a classification accuracy of 71.3% on the IMAGENET dataset (comparable to the VGG-16) but uses 11 times less cores for neuromorphic hardware with core size of 1,024 × 1,024. We also modified the HFNet to fit onto different core sizes and report on the corresponding classification accuracies. Various aspects of the paper are patent pending.

show abstract

“…Real CMOS hardware follows the σ N = 0 noise level. Further, recent research have explored coding schemes for reliable memristor computation at high precision [38,92].…”

Section: Design Space Explorationmentioning

confidence: 99%

“…Many machine learning accelerators have been proposed that leverage memristor crossbars [9,10,12,22,23,53,58,66,73,88,95,100]. These accelerators have been demonstrated on several types of workloads including BSBs [53], MLPs [22,39,73,88], SNNs [9,66], BMs [12], and CNNs [22,23,95,100,109]. Some accelerators support inference only [9,23,53,73,88,95] while others also support training [12,22,66,100].…”

Section: Related Workmentioning

confidence: 99%

Puma

Ankit

Hajj

Chalamalasetti

et al. 2019

Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Syst

262

View full text Add to dashboard Cite

Memristor crossbars are circuits capable of performing analog matrix-vector multiplications, overcoming the fundamental energy efficiency limitations of digital logic. They have been shown to be effective in special-purpose accelerators for a limited set of neural network applications.We present the Programmable Ultra-efficient Memristorbased Accelerator (PUMA) which enhances memristor crossbars with general purpose execution units to enable the acceleration of a wide variety of Machine Learning (ML) inference workloads. PUMA's microarchitecture techniques exposed through a specialized Instruction Set Architecture (ISA) retain the efficiency of in-memory computing and analog circuitry, without compromising programmability.We also present the PUMA compiler which translates high-level code to PUMA ISA. The compiler partitions the computational graph and optimizes instruction scheduling and register allocation to generate code for large and complex workloads to run on thousands of spatial cores.We have developed a detailed architecture simulator that incorporates the functionality, timing, and power models of * Work done while at University of Illinois at Urbana-Champaign PUMA's components to evaluate performance and energy consumption. A PUMA accelerator running at 1 GHz can reach area and power efficiency of 577 GOPS/s/mm 2 and 837 GOPS/s/W, respectively. Our evaluation of diverse ML applications from image recognition, machine translation, and language modelling (5M-800M synapses) shows that PUMA achieves up to 2,446× energy and 66× latency improvement for inference compared to state-of-the-art GPUs. Compared to an application-specific memristor-based accelerator, PUMA incurs small energy overheads at similar inference latency and added programmability.

show abstract

Making Memristive Neural Network Accelerators Reliable

Cited by 97 publications

References 46 publications

Large-Scale Optical Neural Networks Based on Photoelectric Multiplication

Large-Scale Optical Neural Networks Based on Photoelectric Multiplication

HFNet: A CNN Architecture Co-designed for Neuromorphic Hardware With a Crossbar Array of Synapses

Puma

Contact Info

Product

Resources

About