Tolerating Noise Effects in Processing‐in‐Memory Systems for Neural Networks: A Hardware–Software Codesign Perspective

Yang, Xiaoxuan; Wu, Changming; Li, Mo; Chen, Yiran

doi:10.1002/aisy.202200029

Cited by 12 publications

(2 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Apart from attempts at compressing neural network architectures [ 179 ], RRAM weight mapping algorithms [ 180 ], noise-aware training algorithm [ 181 , 182 ] and fault mitigation algorithms [ 183 ] have been reported with much success in recent literature. An alternative strategy is the hardware-software codesign paradigm, where the inherent stochasticity of these devices is incorporated into neural network training and/or inference algorithms [ 184 , 185 ]. Finally, the technological adaptation of RRAM devices for neuromorphic computing requires major innovations in terms of scaling capabilities.…”

Section: Challenges and Future Outlookmentioning

confidence: 99%

Resistive Switching Devices for Neuromorphic Computing: From Foundations to Chip Level Innovations

Udaya Mohanan

2024

Nanomaterials

View full text Add to dashboard Cite

Neuromorphic computing has emerged as an alternative computing paradigm to address the increasing computing needs for data-intensive applications. In this context, resistive random access memory (RRAM) devices have garnered immense interest among the neuromorphic research community due to their capability to emulate intricate neuronal behaviors. RRAM devices excel in terms of their compact size, fast switching capabilities, high ON/OFF ratio, and low energy consumption, among other advantages. This review focuses on the multifaceted aspects of RRAM devices and their application to brain-inspired computing. The review begins with a brief overview of the essential biological concepts that inspire the development of bio-mimetic computing architectures. It then discusses the various types of resistive switching behaviors observed in RRAM devices and the detailed physical mechanisms underlying their operation. Next, a comprehensive discussion on the diverse material choices adapted in recent literature has been carried out, with special emphasis on the benchmark results from recent research literature. Further, the review provides a holistic analysis of the emerging trends in neuromorphic applications, highlighting the state-of-the-art results utilizing RRAM devices. Commercial chip-level applications are given special emphasis in identifying some of the salient research results. Finally, the current challenges and future outlook of RRAM-based devices for neuromorphic research have been summarized. Thus, this review provides valuable understanding along with critical insights and up-to-date information on the latest findings from the field of resistive switching devices towards brain-inspired computing.

show abstract

Section: Challenges and Future Outlookmentioning

confidence: 99%

Resistive Switching Devices for Neuromorphic Computing: From Foundations to Chip Level Innovations

Udaya Mohanan

2024

Nanomaterials

View full text Add to dashboard Cite

show abstract

“…These imperfections can result in degraded accuracy for neural-network inference performed using them [9,[17][18][19]. To mitigate the impact of noise, noise-aware training schemes have been developed [20][21][22][23][24][25][26][27]. These schemes treat the noise as a relatively small perturbation to an otherwise deterministic computation, either by explicitly modeling the noise as the addition of random variables to the processor's output or by modeling the processor as having finite bit precision.…”

mentioning

confidence: 99%

Quantum-noise-limited optical neural networks operating at a few quanta per activation

Ma,

Wang,

Laydevant

et al. 2023

Preprint

View full text Add to dashboard Cite

A practical limit to energy efficiency in computation is ultimately from noise, with quantum noise [1] as the fundamental floor. Analog physical neural networks [2], which hold promise for improved energy efficiency and speed compared to digital electronic neural networks, are nevertheless typically operated in a relatively high-power regime so that the signal-to-noise ratio (SNR) is large (>10). We study optical neural networks [3] operated in the limit where all layers except the last use only a single photon to cause a neuron activation. In this regime, activations are dominated by quantum noise from the fundamentally probabilistic nature of single-photon detection. We show that it is possible to perform accurate machine-learning inference in spite of the extremely high noise (signal-to-noise ratio ~1). We experimentally demonstrated MNIST handwritten-digit classification with a test accuracy of 98% using an optical neural network with a hidden layer operating in the single-photon regime; the optical energy used to perform the classification corresponds to 0.008 photons per multiply-accumulate (MAC) operation, which is equivalent to 0.003 attojoules of optical energy per MAC. Our experiment also used >40× fewer photons per inference than previous state-of-the-art low-optical-energy demonstrations [4, 5] to achieve the same accuracy of >90%. Our training approach, which directly models the system’s stochastic behavior, might also prove useful with non-optical ultra-low-power hardware.

show abstract

Hardware-aware training for large-scale and diverse deep learning inference workloads using in-memory computing-based accelerators

Rasch,

Mackin,

Le Gallo

et al. 2023

Nat Commun

View full text Add to dashboard Cite

Analog in-memory computing—a promising approach for energy-efficient acceleration of deep learning workloads—computes matrix-vector multiplications but only approximately, due to nonidealities that often are non-deterministic or nonlinear. This can adversely impact the achievable inference accuracy. Here, we develop an hardware-aware retraining approach to systematically examine the accuracy of analog in-memory computing across multiple network topologies, and investigate sensitivity and robustness to a broad set of nonidealities. By introducing a realistic crossbar model, we improve significantly on earlier retraining approaches. We show that many larger-scale deep neural networks—including convnets, recurrent networks, and transformers—can in fact be successfully retrained to show iso-accuracy with the floating point implementation. Our results further suggest that nonidealities that add noise to the inputs or outputs, not the weights, have the largest impact on accuracy, and that recurrent networks are particularly robust to all nonidealities.

show abstract

Tolerating Noise Effects in Processing‐in‐Memory Systems for Neural Networks: A Hardware–Software Codesign Perspective

Cited by 12 publications

References 34 publications

Resistive Switching Devices for Neuromorphic Computing: From Foundations to Chip Level Innovations

Resistive Switching Devices for Neuromorphic Computing: From Foundations to Chip Level Innovations

Quantum-noise-limited optical neural networks operating at a few quanta per activation

Hardware-aware training for large-scale and diverse deep learning inference workloads using in-memory computing-based accelerators

Contact Info

Product

Resources

About