Preventing Vanishing Gradient Problem of Hardware Neuromorphic System by Implementing Imidazole‐Based Memristive ReLU Activation Neuron

Abstract

With advances in artificial intelligent services, brain‐inspired neuromorphic systems with synaptic devices are recently attracting significant interest to circumvent the von Neumann bottleneck. However, the increasing trend of deep neural network parameters causes huge power consumption and large area overhead of a nonlinear neuron electronic circuit, and it incurs a vanishing gradient problem. Here, a memristor‐based compact and energy‐efficient neuron device is presented to implement a rectifying linear unit (ReLU) activation function. To emulate the volatile and gradual switching of the ReLU function, a copolymer memristor with a hybrid structure is proposed using a copolymer/inorganic bilayer. The functional copolymer film developed by introducing imidazole functional groups enables the formation of nanocluster‐type pseudo‐conductive filaments by boosting the nucleation of Cu nanoclusters, causing gradual switching. The ReLU neuron device is successfully demonstrated by integrating the memristor with amorphous InGaZnO thin‐film transistors, and achieves 0.5 pJ of energy consumption based on sub‐10 µA operation current and high‐speed switching of 650 ns. Furthermore, device‐to‐system‐level simulation using neuron devices on the MNIST dataset demonstrates that the vanishing gradient problem is effectively resolved by five‐layer deep neural networks. The proposed neuron device will enable the implementation of high‐density and energy‐efficient hardware neuromorphic systems.

Binarized neural network of diode array with high concordance to vector–matrix multiplication

“…Moreover, as the number of hidden layers in the multilayer perceptron structure of neural networks increases, the neuron circuits that interconnect the pre- and post-layers become more complicated, which degrades the area and power efficiencies of computing systems based on neural networks 21 – 24 . The activation function of neuronal circuits is simplified or merged into a memory array to alleviate degradation 25 – 27 .…”

Cross-layer transmission realized by light-emitting memristor for constructing ultra-deep neural network with transfer learning ability

“…Meanwhile, achieving high accuracy in multifunctional DNNs using the reported devices and hardware network structures remains huge challenging. The main reason is that the majority of networks rely on backpropagation (BP) for weight updates 22 – 25 , yet the layer-by-layer structure leads to gradient vanishing (exploding), making it difficult to effectively train the network 26 – 29 . Although the most advanced neuromorphic chips have attempted to construct DNNs with cross-layer transmission, they still rely on repeated reading of digital memory and DAC/ADC to achieve parallel output of results 30 – 32 .…”

Reconfigurable Neuromorphic Computing: Materials, Devices, and Integration

“…Most of existing neuromorphic devices cannot be reconfigured to fulfill the diverse run-time requirements, and hence depend on tailored designs specific to targeted applications. [26,27] For example, neurons for specific activation functions, [28,29] artificial dendrites, [30] and physical reservoir computing [31] are difficult to be reconfigured thus far which play a vital role in neuromorphic computing. Furthermore, energy-and area-efficient neuromorphic hardware imposes stringent requirements for the integration of multiple sophisticated brain-like functions in an all-in-one manner.…”