An approximate backpropagation learning rule for memristor based neural networks using synaptic plasticity

Negrov, Dmitrii; Karandashev, Iakov M.; Shakirov, Vladimir; Matveyev, Yu.; Dunin-Barkowski, Witali L.; Zenkevich, A.

doi:10.1016/j.neucom.2016.10.061

Cited by 39 publications

(17 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Furthermore, it is efficient to improve the characteristics of memristor synapses depending on individual neuromorphic networks, because a desirable memristor synapse capable of being employed into neuromorphic systems is yet to be reported. Supervised learning-based networks [35,[40][41][42][43][44], for example, are less vulnerable to cycle-to-cycle and device-to-device variations. This is because memristor synapses are updated according to calculated errors under known target values.…”

Section: (A) Cation-based Devices: Through Electrochemical Reaction mentioning

confidence: 99%

See 1 more Smart Citation

Memristor Synapses for Neuromorphic Computing

Choi¹,

Ham²

2019

Memristors - Circuits and Applications of Memristor Devices [Working Title]

View full text Add to dashboard Cite

Neuromorphic computing, which imitates the principle behind biological synapses with a high degree of parallelism, has recently emerged as a promising candidate for novel and sustainable computing technologies. The first step toward realizing a massively parallel neuromorphic system is to develop an artificial synapse capable of emulating synapse functionality, such as analog modulation, with ultralow power consumption and robust controllability. We begin this chapter with a simple description of neuromorphic systems and memristor synapses. Further, we introduce and evaluate the state-of-the-art neuromorphic hardware technology in terms of novel functional materials and device architectures toward the implementation of fully neuromorphic computers, which have been extensively explored in recent years. Finally, we briefly describe artificial neural networks based on memristor synapse in forms of crossbar arrays.Conventional computing architecture, that is, von Neumann architecture, forms the groundwork for modern computing technologies [3,18]. Despite tremendous growth in computing performance, classical architecture currently suffers from the von Neumann bottleneck, which results from data movements between the processor and the memory unit [4,5]. The memory wall issue, causing high power consumption and low speed, hinders the continuous development of computing technologies [4,5,9]. Moreover, artificial neural network (ANN) algorithms, such as deep learning [19], deal with image classification [20,21], sound recognition [22,23], specific complex tasks (e.g., the AlphaGo [24]) and so on. Although the ANN algorithms have exhibited superior performance over the conventional computing technologies, they are, at present, constructed on the von Neumann architecture; hence, considerable time and energy resources are required for their operation [8,9]. Neuromorphic architecture [6,7], a bio-inspired computing architecture, is one of the most promising candidates to resolve these problems. The neuromorphic systems take advantage of the cerebral nervous system, which consists of a massive parallel connectivity between the neurons (i.e., processor) and the synapses (i.e., memory), indicating the absence of the von Neumann bottleneck [8,9]. Figure 1 shows the shift of the computing architecture from von Neumann architecture (Figure 1a) to neuromorphic architecture (Figure 1b). The von Neumann architecture shows that the processor and memory are separate, leading to the von Neumann bottleneck. In contrast, in the case of neuromorphic architecture, the neurons and synapses are combined, alleviating the bottleneck issue. The neurons are uncomplicated computing units, the synapses are local memory units, and the communication channels (red line) connect numerous neurons and synapses. It should be noted that the practical purpose of neuromorphic systems is not to replace the von Neumann architecture completely, but to supplement the conventional architecture to make up its leeway, especially for intelligent tasks such as image r...

show abstract

Section: (A) Cation-based Devices: Through Electrochemical Reaction mentioning

confidence: 99%

“…This influences output currents, leading to degradation of learning performance. The non-idealities in array-level could be overcome by device functions [35,44], operational scheme [39,[58][59][60], or learning algorithms [35,[40][41][42][43][44] to some degree.…”

Section: Neuromorphic Systems Based On Crossbar Array Of Memristor Symentioning

confidence: 99%

Memristor Synapses for Neuromorphic Computing

Choi¹,

Ham²

2019

Memristors - Circuits and Applications of Memristor Devices [Working Title]

View full text Add to dashboard Cite

show abstract

“…There are several works proposing the implementation of memristive neural network with backpropagation algorithm in digital and mixed-signal domain domain [4], [5], [6], [7], [8], [9], [10]. However, the analog learning circuits based on conventional backpropagation learning algorithm [11], [12], [8], [13], [14] in memristive crossbars have not been fully implemented. The implementation of such learning algorithm opens up an opportunity to create an analog hardware-based learning architecture.…”

Section: Introductionmentioning

confidence: 99%

Learning in Memristive Neural Network Architectures Using Analog Backpropagation Circuits

Krestinskaya

Salama

James

2019

IEEE Trans. Circuits Syst. I

121

View full text Add to dashboard Cite

The on-chip implementation of learning algorithms would speed-up the training of neural networks in crossbar arrays. The circuit level design and implementation of backpropagation algorithm using gradient descent operation for neural network architectures is an open problem. In this paper, we proposed the analog backpropagation learning circuits for various memristive learning architectures, such as Deep Neural Network (DNN), Binary Neural Network (BNN), Multiple Neural Network (MNN), Hierarchical Temporal Memory (HTM) and Long-Short Term Memory (LSTM). The circuit design and verification is done using TSMC 180nm CMOS process models, and TiO2 based memristor models. The application level validations of the system are done using XOR problem, MNIST character and Yale face image databases.

show abstract

“…The possibility of using memristors as synapses in neural networks has been extensively studied. The wealth of proposals in this field can be broadly split into two groups: one related to spike-timing-dependent plasticity (STDP) and spiking neural networks (SNN) [26][27][28][29][30], and the other to more traditional neural network models [31][32][33][34][35][36][37][38][39][40][41][42][43]. The first group has a more biological focus, with its main goal being the reproduction of effects occurring in natural neural networks, rather than algorith-mic improvements.…”

Section: Introductionmentioning

confidence: 99%

Perceptrons from memristors

et al. 2020

View full text Add to dashboard Cite

Memristors, resistors with memory whose outputs depend on the history of their inputs, have been used with success in neuromorphic architectures, particularly as synapses and non-volatile memories. However, to the best of our knowledge, no model for a network in which both the synapses and the neurons are implemented using memristors has been proposed so far. In the present work we introduce models for single and multilayer perceptrons based exclusively on memristors. We adapt the delta rule to the memristor-based single-layer perceptron and the backpropagation algorithm to the memristor-based multilayer perceptron. Our results show that both perform as expected for perceptrons, including satisfying Minsky-Paperts theorem. As a consequence of the Universal Approximation Theorem, they also show that memristors are universal function approximators. By using memristors for both the neurons and the synapses, our models pave the way for novel memristor-based neural network architectures and algorithms. A neural network based on memristors could show advantages in terms of energy conservation and open up possibilities for other learning systems to be adapted to a memristor-based paradigm, both in the classical and quantum learning realms.

show abstract

An approximate backpropagation learning rule for memristor based neural networks using synaptic plasticity

Cited by 39 publications

References 23 publications

Memristor Synapses for Neuromorphic Computing

Memristor Synapses for Neuromorphic Computing

Learning in Memristive Neural Network Architectures Using Analog Backpropagation Circuits

Perceptrons from memristors

Contact Info

Product

Resources

About