Deep learning acceleration in 14nm CMOS compatible ReRAM array: device, material and algorithm co-optimization

Gong, N.; Rasch, Malte J.; Seo, Soon‐Cheon; Gasasira, Arthur; Solomon, P. M.; Bragaglia, Valeria; Consiglio, Steven; Higuchi, Hisashi; Park, C.; Brew, Kevin; Jamison, P.; Catano, Christopher; Saraf, Iqbal; Athena, Fabia Farlin; Silvestre, C.; Liu, X.; Khan, B.; Jain, N.; McDermott, Steven; Johnson, R.; Estrada-Raygoza, I.; Li, J.; Gokmen, Tayfun; Li, N.; Pujari, R.; Carta, Fabio; Miyazoe, Hiroyuki; Frank, Martin M.; Koty, Devi; Yang, Qingxiang; Clark, Robert D.; Tapily, Kandabara; Wajda, Cory S.; Mosden, Aelan; Shearer, Jeffrery C.; Metz, Andrew; Teehan, S.; Saulnier, Nicole; Offrein, Bert Jan; Tsunomura, T.; Leusink, Gert J.; Narayanan, V.; Ando, Takashi

doi:10.1109/iedm45625.2022.10019569

Cited by 11 publications

(14 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This enables the training of DNNs on extremely noisy devices and reduces the number of states required for the auxiliary device to barely 10 levels. Experimental demonstrations of the algorithms on ECRAM and ReRAM arrays have been reported, which verify the successful training with practical devices. These studies suggest that algorithmic remedies can overcome nonidealities in memory devices and improve software-level classification accuracy (Figure ).…”

Section: Algorithm and Architecturementioning

confidence: 59%

Recent Advances and Future Prospects for Memristive Materials, Devices, and Systems

et al. 2023

View full text Add to dashboard Cite

Memristive technology has been rapidly emerging as a potential alternative to traditional CMOS technology, which is facing fundamental limitations in its development. Since oxide-based resistive switches were demonstrated as memristors in 2008, memristive devices have garnered significant attention due to their biomimetic memory properties, which promise to significantly improve power consumption in computing applications. Here, we provide a comprehensive overview of recent advances in memristive technology, including memristive devices, theory, algorithms, architectures, and systems. In addition, we discuss research directions for various applications of memristive technology including hardware accelerators for artificial intelligence, in-sensor computing, and probabilistic computing. Finally, we provide a forward-looking perspective on the future of memristive technology, outlining the challenges and opportunities for further research and innovation in this field. By providing an up-to-date overview of the state-of-the-art in memristive technology, this review aims to inform and inspire further research in this field.

show abstract

Section: Algorithm and Architecturementioning

confidence: 59%

Recent Advances and Future Prospects for Memristive Materials, Devices, and Systems

et al. 2023

View full text Add to dashboard Cite

show abstract

“…In our recent work, we also showed that the Tiki-Taka algorithm improves the training accuracy compared to the SGD method. We could perform the training of a fully connected network (FCN) on a reduced Modified National Institute of Standards and Technology (MNIST) data set, but devices with enhanced properties are required for scaling to larger networks.…”

mentioning

confidence: 88%

“…In previous works, the weight update was performed row by row 14 or device by device. 15 In our recent work, 16 NN training with parallel crossbar updates was demonstrated. We implemented artificial synapses using resistive random access memory (RRAM) devices.…”

mentioning

confidence: 99%

“…Compared to such previous literature, in this work we present devices combining improvements in every important property for the Tiki-Taka algorithm, such as a higher number of states 16 and a more centered symmetry point. 21 Moreover, we show a stable symmetry point over cycling, and we demonstrate the robustness of the algorithm against device-to-device variability.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Analog Resistive Switching Devices for Training Deep Neural Networks with the Novel Tiki-Taka Algorithm

Stecconi,

Bragaglia,

Rasch

et al. 2024

Nano Lett.

View full text Add to dashboard Cite

A critical bottleneck for the training of large neural networks (NNs) is communication with off-chip memory. A promising mitigation effort consists of integrating crossbar arrays of analogue memories in the Back-End-Of-Line, to store the NN parameters and efficiently perform the required synaptic operations. The "Tiki-Taka" algorithm was developed to facilitate NN training in the presence of device nonidealities. However, so far, a resistive switching device exhibiting all the fundamental Tiki-Taka requirements, which are many programmable states, a centered symmetry point, and low programming noise, was not yet demonstrated. Here, a complementary metal-oxide semiconductor (CMOS)-compatible resistive random access memory (RRAM), showing more than 30 programmable states with low noise and a symmetry point with only 5% skew from the center, is presented for the first time. These results enable generalization of Tiki-Taka training from small fully connected networks to larger long-/short-term-memory types of NN.

show abstract

“…It was shown (Gokmen, 2021) that the filtering stage greatly lowers the device requirements, in particular the effective number of device states: only 15 states are enough to train an LSTM to acceptable accuracy. This relaxed device requirements now make it possible to think about current ReRAM devices for in-memory training (Gong et al, 2022). This Tiki-Taka version 2 (TTv2) algorithm is our baseline comparison 3 .…”

Section: Ttv2 Algorithmmentioning

confidence: 99%

Fast offset corrected in-memory training

Rasch¹,

Carta²,

Fagbohungbe³

et al. 2023

Preprint

View full text Add to dashboard Cite

In-memory computing with resistive crossbar arrays has been suggested to accelerate deep-learning workloads in highly efficient manner. To unleash the full potential of in-memory computing, it is desirable to accelerate the training as well as inference for large deep neural networks (DNNs). In the past, specialized in-memory training algorithms have been proposed that not only accelerate the forward and backward passes, but also establish tricks to update the weight in-memory and in parallel. However, the state-of-the-art algorithm (Tiki-Taka version 2 (TTv2)) still requires near perfect offset correction and suffers from potential biases that might occur due to programming and estimation inaccuracies, as well as longer-term instabilities of the device materials. Here we propose and describe two new and improved algorithms for in-memory computing (Chopped-TTv2 (c-TTv2) and Analog Gradient Accumulation with Dynamic reference (AGAD)), that retain the same runtime complexity but correct for any remaining offsets using choppers. These algorithms greatly relax the device requirements and thus expanding the scope of possible materials potentially employed for such fast in-memory DNN training.

show abstract

Deep learning acceleration in 14nm CMOS compatible ReRAM array: device, material and algorithm co-optimization

Cited by 11 publications

References 12 publications

Recent Advances and Future Prospects for Memristive Materials, Devices, and Systems

Recent Advances and Future Prospects for Memristive Materials, Devices, and Systems

Analog Resistive Switching Devices for Training Deep Neural Networks with the Novel Tiki-Taka Algorithm

Fast offset corrected in-memory training

Contact Info

Product

Resources

About