Randomized unregulated step descent for limited precision synaptic elements

Müller, Lorenz; Nair, Manu V; Indiveri, Giacomo

doi:10.1109/iscas.2017.8050217

Cited by 7 publications

(9 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, k-means clustering has been used to map high resolution weights to low resolution values. One of the significant low resolution learning methods for on-chip learning is randomized unregulated step descent (RUSD) introduced in [34]. This method combines unregulated step descent (USD) with randomized rounding i.e.…”

Section: B Backpropagation and Variantsmentioning

confidence: 99%

Low-Power, Adaptive Neuromorphic Systems: Recent Progress and Future Directions

Basu

Acharya

Karnik

et al. 2018

IEEE J. Emerg. Sel. Topics Circuits Syst.

100

View full text Add to dashboard Cite

In this paper, we present a survey of recent works in developing neuromorphic or neuro-inspired hardware systems.In particular, we focus on those systems which can either learn from data in an unsupervised or online supervised manner. We present algorithms and architectures developed specially to support on-chip learning. Emphasis is placed on hardware friendly modifications of standard algorithms, such as backpropagation, as well as novel algorithms, such as structural plasticity, developed specially for low-resolution synapses. We cover works related to both spike-based and more traditional non-spike-based algorithms. This is followed by developments in novel devices, such as floating-gate MOS, memristors, and spintronic devices. CMOS circuit innovations for on-chip learning and CMOS interface circuits for post-CMOS devices, such as memristors, are presented. Common architectures, such as crossbar or island style arrays, are discussed, along with their relative merits and demerits. Finally, we present some possible applications of neuromorphic hardware, such as brain-machine interfaces, robotics, etc., and identify future research trends in the field.

show abstract

Section: B Backpropagation and Variantsmentioning

confidence: 99%

Low-Power, Adaptive Neuromorphic Systems: Recent Progress and Future Directions

Basu

Acharya

Karnik

et al. 2018

IEEE J. Emerg. Sel. Topics Circuits Syst.

100

View full text Add to dashboard Cite

show abstract

“…The former allows exploiting the full potential of memristive devices tuneability to achieve a real-time on-line adaptive operation. Among the spike-based training procedures, supervised learning rules inspired by the back-propagation exist (Urbanczik and Senn, 2014 ; Müller et al, 2017 ; Donati et al, 2019 ), which are seldom investigated for systems including realistic simulations of memristive devices (Nair et al, 2017 ; Payvand et al, 2018 ). On the contrary, the literature is extremely rich of reports dealing with networks trained by supervised (Brivio et al, 2019a ) and unsupervised versions of the so-called Spike Timing Dependent Plasticity (STDP) (Diehl and Cook, 2015 ; Garbin et al, 2015 ; Querlioz et al, 2015 ; Ambrogio et al, 2016 ; La Barbera et al, 2018 ).…”

Section: Introductionmentioning

confidence: 99%

Non-linear Memristive Synaptic Dynamics for Efficient Unsupervised Learning in Spiking Neural Networks

Brivio

Vianello

et al. 2021

Front. Neurosci.

View full text Add to dashboard Cite

Spiking neural networks (SNNs) are a computational tool in which the information is coded into spikes, as in some parts of the brain, differently from conventional neural networks (NNs) that compute over real-numbers. Therefore, SNNs can implement intelligent information extraction in real-time at the edge of data acquisition and correspond to a complementary solution to conventional NNs working for cloud-computing. Both NN classes face hardware constraints due to limited computing parallelism and separation of logic and memory. Emerging memory devices, like resistive switching memories, phase change memories, or memristive devices in general are strong candidates to remove these hurdles for NN applications. The well-established training procedures of conventional NNs helped in defining the desiderata for memristive device dynamics implementing synaptic units. The generally agreed requirements are a linear evolution of memristive conductance upon stimulation with train of identical pulses and a symmetric conductance change for conductance increase and decrease. Conversely, little work has been done to understand the main properties of memristive devices supporting efficient SNN operation. The reason lies in the lack of a background theory for their training. As a consequence, requirements for NNs have been taken as a reference to develop memristive devices for SNNs. In the present work, we show that, for efficient CMOS/memristive SNNs, the requirements for synaptic memristive dynamics are very different from the needs of a conventional NN. System-level simulations of a SNN trained to classify hand-written digit images through a spike timing dependent plasticity protocol are performed considering various linear and non-linear plausible synaptic memristive dynamics. We consider memristive dynamics bounded by artificial hard conductance values and limited by the natural dynamics evolution toward asymptotic values (soft-boundaries). We quantitatively analyze the impact of resolution and non-linearity properties of the synapses on the network training and classification performance. Finally, we demonstrate that the non-linear synapses with hard boundary values enable higher classification performance and realize the best trade-off between classification accuracy and required training time. With reference to the obtained results, we discuss how memristive devices with non-linear dynamics constitute a technologically convenient solution for the development of on-line SNN training.

show abstract

“…This circuit is inspired by the biologically plausible learning rule presented in [49] and gradient-descent based methods applied to memristive devices [35,47]. We use these learning circuits to implement a randomized unregulated step descent algorithm, which has been shown to be effective for synaptic elements with limited precision [34]. In the following Section, we present the network architecture that is compatible with the proposed differential memristive synapse circuit.…”

Section: Introductionmentioning

confidence: 99%

A differential memristive synapse circuit for on-line learning in neuromorphic computing systems

2017

Self Cite

View full text Add to dashboard Cite

Spike-based learning with memristive devices in neuromorphic computing architectures typically uses learning circuits that require overlapping pulses from preand post-synaptic nodes. This imposes severe constraints on the length of the pulses transmitted in the network, and on the network's throughput. Furthermore, most of these circuits do not decouple the currents flowing through memristive devices from the one stimulating the target neuron. This can be a problem when using devices with high conductance values, because of the resulting large currents. In this paper we propose a novel circuit that decouples the current produced by the memristive device from the one used to stimulate the post-synaptic neuron, by using a novel differential scheme based on the Gilbert normalizer circuit. We show how this circuit is useful for reducing the effect of variability in the memristive devices, and how it is ideally suited for spike-based learning mechanisms that do not require overlapping pre-and post-synaptic pulses. We demonstrate the features of the proposed synapse circuit with SPICE simulations, and validate its learning properties with high-level behavioral network simulations which use a stochastic gradient descent learning rule in two classification tasks.

show abstract

Randomized unregulated step descent for limited precision synaptic elements

Cited by 7 publications

References 14 publications

Low-Power, Adaptive Neuromorphic Systems: Recent Progress and Future Directions

Low-Power, Adaptive Neuromorphic Systems: Recent Progress and Future Directions

Non-linear Memristive Synaptic Dynamics for Efficient Unsupervised Learning in Spiking Neural Networks

A differential memristive synapse circuit for on-line learning in neuromorphic computing systems

Contact Info

Product

Resources

About