Resistive memory device requirements for a neural algorithm accelerator

Agarwal, Sapan; Plimpton, Steven J.; Hughart, David Russell; Hsia, Alexander H.; Richter, Isaac; Cox, James C.; James, Conrad D.; Marinella, Matthew

doi:10.1109/ijcnn.2016.7727298

Cited by 182 publications

(142 citation statements)

References 36 publications

Supporting

Mentioning

138

Contrasting

Order By: Relevance

“…To take conductance saturation into account, which is expected to be present in actual RPU devices, the bounds on the weights values, | w ij |, is assumed be 0.6 on average with a 30% device-to-device variation. We did not introduce any non-linearity in the weight update as this effect has been shown to be insignificant as long as the updates are reasonably balanced (symmetric) between up and down changes (Agrawal et al, 2016a ; Gokmen and Vlasov, 2016 ). During the forward and backward cycles the vector-matrix multiplications performed on an RPU array are prone to analog noise and signal saturation due to the peripheral circuitry.…”

Section: Resultsmentioning

confidence: 99%

“…We note that for all of the simulation results described above we do not include any non-linearity in the weight update as this effect is shown to be not important as long as the updates are symmetric in positive and negative directions (Agrawal et al, 2016a ; Gokmen and Vlasov, 2016 ). In order to check the validity of this behavior for the above CNN architecture, we performed simulations using the blue model of Figure 6 while including a weight dependent update rule with different functional forms Δ w min ( w ij ) that included a linear or a quadratic dependence on weight value.…”

Section: Resultsmentioning

confidence: 99%

“…The concept of using resistive cross-point device arrays (Chen et al, 2015b ; Agrawal et al, 2016b ; Gokmen and Vlasov, 2016 ; Fuller et al, 2017 ) as DNN accelerators have been tested, to some extent, by performing simulations for the specific case of fully connected neural networks. The effect of various device properties and system parameters on training performance has been evaluated to derive the required device and system level specifications for a successful implementation of an accelerator chip for DNN compute efficient training (Agrawal et al, 2016a ; Gokmen and Vlasov, 2016 ). A key requirement is that these analog resistive devices must change conductance symmetrically when subjected to positive or negative pulse stimuli.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Training Deep Convolutional Neural Networks with Resistive Cross-Point Devices

2017

View full text Add to dashboard Cite

In a previous work we have detailed the requirements for obtaining maximal deep learning performance benefit by implementing fully connected deep neural networks (DNN) in the form of arrays of resistive devices. Here we extend the concept of Resistive Processing Unit (RPU) devices to convolutional neural networks (CNNs). We show how to map the convolutional layers to fully connected RPU arrays such that the parallelism of the hardware can be fully utilized in all three cycles of the backpropagation algorithm. We find that the noise and bound limitations imposed by the analog nature of the computations performed on the arrays significantly affect the training accuracy of the CNNs. Noise and bound management techniques are presented that mitigate these problems without introducing any additional complexity in the analog circuits and that can be addressed by the digital circuits. In addition, we discuss digitally programmable update management and device variability reduction techniques that can be used selectively for some of the layers in a CNN. We show that a combination of all those techniques enables a successful application of the RPU concept for training CNNs. The techniques discussed here are more general and can be applied beyond CNN architectures and therefore enables applicability of the RPU approach to a large class of neural network architectures.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Training Deep Convolutional Neural Networks with Resistive Cross-Point Devices

2017

View full text Add to dashboard Cite

show abstract

“…Considering state-of-the-art learning algorithms, to par training accuracy compared to the conventional digital hardware a restrictive set of RPU device specifications must be met. As shown empirically [12] [19] [20], a key requirement is that these analog resistive devices must change conductance symmetrically when subjected to positive or negative voltage pulse stimuli. This requirement differs significantly from those needed for memory elements and accomplishing such symmetrically switching analog devices is a difficult task.…”

Section: Introductionmentioning

confidence: 99%

Algorithm for Training Neural Networks on Resistive Device Arrays

Gokmen

Haensch

2020

Front. Neurosci.

View full text Add to dashboard Cite

Hardware architectures composed of resistive cross-point device arrays can provide significant power and speed benefits for deep neural network training workloads using stochastic gradient descent (SGD) and backpropagation (BP) algorithm. The training accuracy on this imminent analog hardware however strongly depends on the switching characteristics of the cross-point elements. One of the key requirements is that these resistive devices must change conductance in a symmetrical fashion when subjected to positive or negative pulse stimuli. Here, we present a new training algorithm, so-called the "Tiki-Taka" algorithm, that eliminates this stringent symmetry requirement. We show that device asymmetry introduces an unintentional implicit cost term into the SGD algorithm, whereas in the "Tiki-Taka" algorithm a coupled dynamical system simultaneously minimizes the original objective function of the neural network and the unintentional cost term due to device asymmetry in a self-consistent fashion. We tested the validity of this new algorithm on a range of network architectures such as fully connected, convolutional and LSTM networks. Simulation results on these various networks show that whatever accuracy is achieved using the conventional SGD algorithm with symmetric (ideal) device switching characteristics the same accuracy is also achieved using the "Tiki-Taka" algorithm with non-symmetric (non-ideal) device switching characteristics. Moreover, all the operations performed on the arrays are still parallel and therefore the implementation cost of this new algorithm on array architectures is minimal; and it maintains the aforementioned power and speed benefits. These algorithmic improvements are crucial to relax the material specification and to realize technologically viable resistive crossbar arrays that outperform digital accelerators for similar training tasks.

show abstract

“…More importantly, our study demonstrates that the overall current noise level stays at ≤1% of the mean current even in truly atomic scale Ag filaments which neither compromises reproducible resistive switching nor impedes most practical applications. 53,54 We studied memristive nanojunctions created between a mechanically sharpened PtIr tip of a custom designed scanning tunneling microscope (STM) and Ag/Ag 2 S thin films. The latter were fabricated by the electron-beam evaporation of an 80 nm thick Ag layer onto a Si substrate followed by a 5-minute long sulfurisation carried out at 60°C and 5 × 10 −6 mbar resulting in a 30 nm thick stoichiometric Ag 2 S layer on top of the Ag electrode.…”

mentioning

confidence: 99%