Mixed-Precision Deep Learning Based on Computational Memory

Nandakumar, S. R.; Gallo, Manuel Le; Piveteau, Christophe; Joshi, Vinay; Mariani, Giovanni; Boybat, Irem; Karunaratne, Geethan; Khaddam-Aljameh, Riduan; Egger, Urs; Πετρόπουλος, Αναστάσιος; Antonakopoulos, Theodore; Rajendran, Bipin; Sebastian, Abu; Eleftheriou, Evangelos

doi:10.3389/fnins.2020.00406

Cited by 87 publications

(86 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In order to reduce the data transfers to a minimum in inference accelerators, a promising avenue is to employ in-memory computing using non-volatile memory devices [3][4][5] . Both charge-based storage devices, such as Flash memory 6 , and resistance-based (memristive) storage devices, such as metal-oxide resistive randomaccess memory [7][8][9][10] and phase-change memory (PCM) [11][12][13][14] are being investigated for this. In this approach, the network weights are encoded as the analog charge state or conductance state of these devices organized in crossbar arrays, and the matrix-vector multiplications during inference can be performed in-situ in a single time step by exploiting Kirchhoff's circuit laws.…”

mentioning

confidence: 99%

Accurate deep neural network inference using computational phase-change memory

Joshi¹,

Gallo²,

Haefeli³

et al. 2020

Nat Commun

Self Cite

342

323

View full text Add to dashboard Cite

In-memory computing using resistive memory devices is a promising non-von Neumann approach for making energy-efficient deep learning inference hardware. However, due to device variability and noise, the network needs to be trained in a specific way so that transferring the digitally trained weights to the analog resistive memory devices will not result in significant loss of accuracy. Here, we introduce a methodology to train ResNet-type convolutional neural networks that results in no appreciable accuracy loss when transferring weights to phase-change memory (PCM) devices. We also propose a compensation technique that exploits the batch normalization parameters to improve the accuracy retention over time. We achieve a classification accuracy of 93.7% on CIFAR-10 and a top-1 accuracy of 71.6% on ImageNet benchmarks after mapping the trained weights to PCM. Our hardware results on CIFAR-10 with ResNet-32 demonstrate an accuracy above 93.5% retained over a one-day period, where each of the 361,722 synaptic weights is programmed on just two PCM devices organized in a differential configuration.

show abstract

mentioning

confidence: 99%

Accurate deep neural network inference using computational phase-change memory

Joshi¹,

Gallo²,

Haefeli³

et al. 2020

Nat Commun

Self Cite

342

323

View full text Add to dashboard Cite

show abstract

Section: Acceleratorsmentioning

confidence: 99%

Memristors—From In‐Memory Computing, Deep Learning Acceleration, and Spiking Neural Networks to the Future of Neuromorphic and Bio‐Inspired Computing

Mehonić¹,

Sebastian

Rajendran

et al. 2020

Advanced Intelligent Systems

Self Cite

183

121

View full text Add to dashboard Cite

Three factors are currently driving the main developments in artificial intelligence (AI): availability of vast amounts of data, continuous growth in computing power, and algorithmic innovations. Graphics processing units (GPUs) have been demonstrated as effective co-processors for the implementation of machine learning (ML) algorithms based on deep learning (DL). Solutions based on DL and GPU implementations have led to massive improvements in many AI tasks, but have also caused an exponential increase in demand for computing power. Recent analyses show that the demand for computing power has increased by a factor of 300 000 since 2012, and the estimate is that this demand will double every 3.4 months-at a much faster rate than improvements made historically through Moore's scaling (a sevenfold improvement over the same period of time). [1] At the same time, Moore's law has been slowing down significantly for the last few years, [2] as there are strong indications that we will not be able to continue scaling down complementary metal-oxide-semiconductor (CMOS) transistors. This calls for the exploration of alternative technology roadmaps for the development of scalable and efficient AI solutions. Transistor scaling is not the only way to improve computing performance. Architectural innovations, such as GPUs, field-programmable arrays (FPGAs), and application-specific integrated circuits (ASICs), have all significantly advanced the ML field. [3] A common aspect of modern computing architectures for ML is a move away from the classical von-Neumann architecture that physically separates memory and computing. This approach yields a performance bottleneck that is often the main reason for both energy and speed inefficiency of ML implementations on conventional hardware platforms due to

show abstract

“…Under this circumstance, some DNNs architectures with the novel learning algorithms were proposed during last 5 years. One intriguing feature of the DNN training is reportedly to perform the forward and backward propagations imprecisely while the gradients need to be accumulated in high precision, which induced the debut of a mixed-precision in-memory computing approach [166]. The key idea is to store the synaptic weights in phase-change devices where the forward and backward passes are performed, whereas the weight changes are accumulated in high precision, as shown in Figures 24(a)-(e).…”

Section: Phase-change Neuro Networkmentioning

confidence: 99%

“…The synaptic weights are changed by the pulses applied to the memory devices once the accumulated weight exceeds the threshold value. Inspired by this idea, a two-layered neural network having 2-phase-change devices in the differential configuration that indicate the synaptic weights was utilized to solve the handwritten digit classification problem [166]. The test accuracy after 20 epochs of the training was approximately 98%.…”

Section: Phase-change Neuro Networkmentioning

confidence: 99%

“…A hardware model incorporating PCM conductance drift (ν = 0.06), device-to-device drift exponent variability with 0.1ν standard deviation, and instantaneous Gaussian read noise with 0.6 µS standard deviation is able to capture the experimental results fairly well. (a)-(e) are reprinted with permission from [166]; (f) is reprinted with permission from [168]; (g) and (h) are reprinted with permission from [167].…”

Section: Challenges Of Phase-change Devicesmentioning

confidence: 99%

See 1 more Smart Citation

Applications of Phase Change Materials in Electrical Regime From Conventional Storage Memory to Novel Neuromorphic Computing

Liu

Wang

2020

IEEE Access

View full text Add to dashboard Cite

Phase-change materials, also well known as the Chalcogenide alloy, have received considerable attention during last two decades owing to its widespread applications in the field of the electrical storage market such as phase-change random access memory and phase-change probe memory. In addition to the storage devices, its unique electrical properties that can be dynamically tunable with respect to the electrical excitations lead to numerous novel applications represented by memristor and memristor-based neuromorphic electrical circuits. These emerging applications undoubtedly allows for a further exploitation of the potential of phase-change materials, and thus makes it advantageous over other storage medium like ferroelectric and magnetic materials. In order to help researchers understand the role of phase-change materials in these novel applications as well as their importance for citizen's daily life, a comprehensive review that not only covers the traditional storage applications, but also the applications on these exotic devices becomes imperative. In this review, the chemical structure of phase-change materials and their remarkable electrical properties are first reviewed, followed by an introduction of their applications on the storage fields. The physical principles of various emerging electrical devices using phase-change materials are subsequently overviewed in association with their state-of-the-art progress. The prospect of phase-change materials for the future non-volatile electrical applications that are yet to be unraveled is finally envisaged.

show abstract

Mixed-Precision Deep Learning Based on Computational Memory

Cited by 87 publications

References 54 publications

Accurate deep neural network inference using computational phase-change memory

Accurate deep neural network inference using computational phase-change memory

Memristors—From In‐Memory Computing, Deep Learning Acceleration, and Spiking Neural Networks to the Future of Neuromorphic and Bio‐Inspired Computing

Applications of Phase Change Materials in Electrical Regime From Conventional Storage Memory to Novel Neuromorphic Computing

Contact Info

Product

Resources

About