Nonideality‐Aware Training for Accurate and Robust Low‐Power Memristive Neural Networks

Abstract

Recent years have seen a rapid rise of artificial neural networks being employed in a number of cognitive tasks. The ever‐increasing computing requirements of these structures have contributed to a desire for novel technologies and paradigms, including memristor‐based hardware accelerators. Solutions based on memristive crossbars and analog data processing promise to improve the overall energy efficiency. However, memristor nonidealities can lead to the degradation of neural network accuracy, while the attempts to mitigate these negative effects often introduce design trade‐offs, such as those between power and reliability. In this work, authors design nonideality‐aware training of memristor‐based neural networks capable of dealing with the most common device nonidealities. The feasibility of using high‐resistance devices that exhibit high I‐V nonlinearity is demonstrated—by analyzing experimental data and employing nonideality‐aware training, it is estimated that the energy efficiency of memristive vector‐matrix multipliers is improved by almost three orders of magnitude (0.715 TOPs−1W−1 to 381 TOPs−1W−1) while maintaining similar accuracy. It is shown that associating the parameters of neural networks with individual memristors allows to bias these devices toward less conductive states through regularization of the corresponding optimization problem, while modifying the validation procedure leads to more reliable estimates of performance. The authors demonstrate the universality and robustness of this approach when dealing with a wide range of nonidealities.

Memristive, Spintronic, and 2D‐Materials‐Based Devices to Improve and Complement Computing Hardware

“…As an example, devices like memristors may get stuck in a certain conductance state [119] or even fail to electroform (i.e., become conductive), [120] experience random telegraph noise (RTN) [121,122] or programming variability, [123] or have their conductance state drift over time. [124] Even more difficult to tackle are nonidealities that result in deviations from the linear (with respect to conductance and/or voltage) behavior, which DPEs rely on; such nonidealities include I-V nonlinearity [125,126] and line resistance. [127][128][129] There are multiple ways of utilizing DPEs for the implementation of ANNs.…”

“…[135] Where the effects of nonidealities cannot be represented by injecting noise into the weights, their behavior can be redefined to reflect, for example, I-V nonlinearities. [126] Although ex situ training can significantly improve the performance, it is important to consider that it relies on a number of assumptions. If the modeling of nonidealities is inaccurate, that will be reflected in the training on a digital computer and may result in deviations from intended behavior when ANNs are implemented physically.…”

“…Therefore, it can improve the performance when the modeling is not perfectly accurate or even when different nonidealities manifest themselves. [126] Finally, one may employ in situ training, which can refer to either full or partial training directly on crossbar arrays. Performing ANN training on real devices can help networks adapt to specific instantiations of nonideal behavior-no two analog devices are the same, but in situ, unlike ex situ, training can take individual variations into account without the need to model the behavior.…”

“…[ 135 ] Where the effects of nonidealities cannot be represented by injecting noise into the weights, their behavior can be redefined to reflect, for example, I–V nonlinearities. [ 126 ]…”

Efficient Training of the Memristive Deep Belief Net Immune to Non‐Idealities of the Synaptic Devices

“…[25,26] Addtionally, process variation and stuck-at fault errors have been widely reported to cause performance degradation, although can be partially compensated by various methods with extra costs. [27][28][29][30][31][32][33][34][35][36][37][38][39] These issues can be generally attributed to the nonbiological conventional learning algorithm of DNN, that is, error backpropagation-based gradient descent weight update, [40,41] which needs both the VMMs and the conductance tuning in high precision. [42][43][44] Novel neural network structures and learning algorithms need to be explored to address these issues.…”