Supporting the Momentum Training Algorithm Using a Memristor-Based Synapse

Abstract

Despite the increasing popularity of deep neural networks (DNNs), they cannot be trained efficiently on existing platforms, and efforts have thus been devoted to designing dedicated hardware for DNNs. In our recent work, we have provided direct support for the stochastic gradient descent (SGD) training algorithm by constructing the basic element of neural networks, the synapse, using emerging technologies, namely memristors. Due to the limited performance of SGD, optimization algorithms are commonly employed in DNN training. Therefore, DNN accelerators that only support SGD might not meet DNN training requirements. In this paper, we present a memristorbased synapse that supports the commonly used momentum algorithm. Momentum significantly improves the convergence of SGD and facilitates the DNN training stage. We propose two design approaches to support momentum: 1) a hardware friendly modification of the momentum algorithm using memory external to the synapse structure, and 2) updating each synapse with a built-in memory. Our simulations show that the proposed DNN training solutions are as accurate as training on a GPU platform while speeding up the performance by 886× and decreasing energy consumption by 7×, on average.

Training of quantized deep neural networks using a magnetic tunnel junction-based synapse

“…The free layer port of each MTJ is connected to two access transistors. This synapse is inspired by previous work [4,21], but we replace the RRAM by the MTJ device, and two synapse structures are added together to support the ternary weight. In contrast to [4,21] which supports full-precision analog weight values, the MTJ-based synapse supports quantized weights and stochastic weight updates.…”

“…This synapse is inspired by previous work [4,21], but we replace the RRAM by the MTJ device, and two synapse structures are added together to support the ternary weight. In contrast to [4,21] which supports full-precision analog weight values, the MTJ-based synapse supports quantized weights and stochastic weight updates. Sections 4.3 and 4.5 describe how our design is optimized to support quantized weights.…”

“…TNN training requires the circuits to support the feedforward and backpropagation stages [21]. The feedforward stage requires to compute matrix-vector or matrix-matrix multiplication; in TNNs, the multiplication is replaced with the gated XNOR (GXNOR) operation.…”

Real Processing-in-Memory with Memristive Memory Processing Unit (mMPU)

“…Due to their high switching speed, low operating power, scalability, and high endurance [35], memristors are considered as attractive candidates to replace conventional memory and storage technologies (e.g., DRAM and Flash). Memristive technologies have also been explored for additional applications such as analog and radiofrequency circuits [36][37][38][39][40][41][42], neuromorphic circuits [43][44][45][46][47][48], and logic circuits, which are the focus of this paper. Different methods for using memristors to perform logical operations have been proposed.…”

Memristive LIF Spiking Neuron Model and Its Application in Morse Code

“…To solve these deficiencies, we need to find a new device to promote the LIF neuron model. A memristor is a potential element to emulate the function and behavior of a biological synapse or neuron (Hu et al, 2016 ; Choi et al, 2018 ; Chen et al, 2019 ; Greenberg-Toledo et al, 2019 ; Xia and Yang, 2019 ; Wang et al, 2020 ; Shi and Zeng, 2021 ) gets a lot of attention. The non-volatile memristor modulates its conductance due to ion motion, similar to the phenomena in biological neurons and synapses.…”