Experimental Demonstration of Multilevel Resistive Random Access Memory Programming for up to Two Months Stable Neural Networks Inference Accuracy

Esmanhotto, Eduardo; Hirtzlin, Tifenn; Bonnet, Djohan; Castellani, N.; Portal, Jean‐Michel; Querlioz, Damien; Vianello, Elisa

doi:10.1002/aisy.202200145

Cited by 10 publications

(13 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The model of the noise is derived by conductance measurements of our programmed RRAM devices after a relaxation period (Fig. 2 f) and set to 10% of the maximum conductance value in the layer 59 . This approach makes sure that the obtained simulation weights can be mappable on devices with minimal accuracy drop after realistic programming stochasticity and device relaxation effects.…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

DenRAM: neuromorphic dendritic architecture with RRAM for efficient temporal processing with delays

D’Agostino,

Moro,

Torchet

et al. 2024

Nat Commun

Self Cite

View full text Add to dashboard Cite

An increasing number of studies are highlighting the importance of spatial dendritic branching in pyramidal neurons in the neocortex for supporting non-linear computation through localized synaptic integration. In particular, dendritic branches play a key role in temporal signal processing and feature detection. This is accomplished thanks to coincidence detection (CD) mechanisms enabled by the presence of synaptic delays that align temporally disparate inputs for effective integration. Computational studies on spiking neural networks further highlight the significance of delays for achieving spatio-temporal pattern recognition with pure feed-forward neural networks, without the need of resorting to recurrent architectures. In this work, we present “DenRAM”, the first realization of a feed-forward spiking neural network with dendritic compartments, implemented using analog electronic circuits integrated into a 130 nm technology node and coupled with Resistive Random Access Memory (RRAM) technology. DenRAM’s dendritic circuits use RRAM devices to implement both delays and synaptic weights in the network. By configuring the RRAM devices to reproduce bio-realistic timescales, and by exploiting their heterogeneity we experimentally demonstrate DenRAM’s ability to replicate synaptic delay profiles, and to efficiently implement CD for spatio-temporal pattern recognition. To validate the architecture, we conduct comprehensive system-level simulations on two representative temporal benchmarks, demonstrating DenRAM’s resilience to analog hardware noise, and its superior accuracy compared to recurrent architectures with an equivalent number of parameters. DenRAM not only brings rich temporal processing capabilities to neuromorphic architectures, but also reduces the memory footprint of edge devices, warrants high accuracy on temporal benchmarks, and represents a significant step-forward in low-power real-time signal processing technologies.

show abstract

Section: Resultsmentioning

confidence: 99%

“…Smart programming of the devices can be reached to obtain more precise conductance levels and stabilize the devices with respect to the filament relaxation resulting in a conductance shift 57 , 59 .…”

Section: Methodsmentioning

confidence: 99%

DenRAM: neuromorphic dendritic architecture with RRAM for efficient temporal processing with delays

D’Agostino,

Moro,

Torchet

et al. 2024

Nat Commun

Self Cite

View full text Add to dashboard Cite

show abstract

“…(B) Confusion matrix obtained at the end of the training process. (C) Network accuracy, as a function of the time elapsed from the parameter storage, due to the drift mechanism of the assumed memristive synaptic weight physical support (model derived from Esmanhotto et al, 2022 ).…”

Section: Test Casesmentioning

confidence: 99%

“…We here illustrate how the network accuracy may change, assuming the drift of a memristive memory device serving as the physical support of the synaptic weight. We use the circuit design and experimental data presented in Esmanhotto et al (2022) to derive a drift compact model (more details on the drift model are found in Supplementary material (Section 4)). We updated the synaptic model introducing this mechanism, and we simulated the inference on the test dataset as a function varying the time elapsed from an assumed network parameter storage carried out in a write-verify schema (thus, not reliant on any weight quantization).…”

Section: Test Casesmentioning

confidence: 99%

SHIP: a computational framework for simulating and validating novel technologies in hardware spiking neural networks

Gemo,

Spiga,

Brivio

2024

Front. Neurosci.

View full text Add to dashboard Cite

Investigations in the field of spiking neural networks (SNNs) encompass diverse, yet overlapping, scientific disciplines. Examples range from purely neuroscientific investigations, researches on computational aspects of neuroscience, or applicative-oriented studies aiming to improve SNNs performance or to develop artificial hardware counterparts. However, the simulation of SNNs is a complex task that can not be adequately addressed with a single platform applicable to all scenarios. The optimization of a simulation environment to meet specific metrics often entails compromises in other aspects. This computational challenge has led to an apparent dichotomy of approaches, with model-driven algorithms dedicated to the detailed simulation of biological networks, and data-driven algorithms designed for efficient processing of large input datasets. Nevertheless, material scientists, device physicists, and neuromorphic engineers who develop new technologies for spiking neuromorphic hardware solutions would find benefit in a simulation environment that borrows aspects from both approaches, thus facilitating modeling, analysis, and training of prospective SNN systems. This manuscript explores the numerical challenges deriving from the simulation of spiking neural networks, and introduces SHIP, Spiking (neural network) Hardware In PyTorch, a numerical tool that supports the investigation and/or validation of materials, devices, small circuit blocks within SNN architectures. SHIP facilitates the algorithmic definition of the models for the components of a network, the monitoring of states and output of the modeled systems, and the training of the synaptic weights of the network, by way of user-defined unsupervised learning rules or supervised training techniques derived from conventional machine learning. SHIP offers a valuable tool for researchers and developers in the field of hardware-based spiking neural networks, enabling efficient simulation and validation of novel technologies.

show abstract

“…over time, due to the local recombination of oxygen vacancies and structural relaxation of the material, respectively 36,37 .…”

Section: Filamentary Memristor and Phase-change Memory As Normal Dist...mentioning

confidence: 99%

Bringing uncertainty quantification to the extreme-edge with memristor-based Bayesian neural networks

Bonnet

Hirtzlin

Majumdar

et al. 2023

Preprint

View full text Add to dashboard Cite

Safety-critical sensory processing applications, like medical diagnosis, require making accurate decisions based on a small amount of noisy input data. For these applications, using Bayesian neural networks, able to quantify the uncertainty of the predictions, is a superior approach to using conventional artificial neural networks. However, because of the probabilistic nature of Bayesian neural networks, they can be computationally intensive to use for inference stage and thus not well suited for extreme-edge applications. An emerging idea to solve this problem is to use the intrinsic probabilistic nature of memristors to efficiently implement Bayesian neural network inference: the variability in the resistance of memristors would represent the probability distribution of weights in Bayesian neural networks. However, when using memristors, statistical effects follow the laws of device physics, whereas in Bayesian neural networks, those effects can take on arbitrary shapes. In this work, we overcome this difficulty by adopting a dedicated synapse architecture based on two memristors, and by training Bayesian neural networks with a dedicated variational inference technique that includes a “technological loss” to take into account specificities of memristor physics. This technique allowed us to program a two-layer Bayesian neural network on 75 physical crossbar arrays of 1,024 memristors, incorporating CMOS periphery circuitry to do in-memory computing, to classify arrhythmia in electrocardiograms. Our experimental neural network classified heartbeats with high accuracy, and estimated the certainty of its predictions. In the case of uncertain predictions, it differentiated between ambivalent heartbeats (aleatoric uncertainty), and heartbeats with never-seen patterns (epistemic uncertainty). We show that our technique can also be used with phase change memories, by employing a different “technological loss” term. The great advantage of this approach is its low energy consumption: we estimate an 800 times improvement in energy efficiency compared to a GPU performing the same task.

show abstract

Experimental Demonstration of Multilevel Resistive Random Access Memory Programming for up to Two Months Stable Neural Networks Inference Accuracy

Cited by 10 publications

References 25 publications

DenRAM: neuromorphic dendritic architecture with RRAM for efficient temporal processing with delays

DenRAM: neuromorphic dendritic architecture with RRAM for efficient temporal processing with delays

SHIP: a computational framework for simulating and validating novel technologies in hardware spiking neural networks

Bringing uncertainty quantification to the extreme-edge with memristor-based Bayesian neural networks

Contact Info

Product

Resources

About