A Relaxed Quantization Training Method for Hardware Limitations of Resistive Random Access Memory (ReRAM)-Based Computing-in-Memory

Abstract

Nonvolatile computing-in-memory (nvCIM) exhibits high potential for neuromorphic computing involving massive parallel computations and for achieving high energy efficiency. nvCIM is especially suitable for deep neural networks, which are required to perform large amounts of matrix-vector multiplications. However, a comprehensive quantization algorithm has yet to be developed, which overcomes the hardware limitations of resistive random access memory (ReRAM)-based nvCIM, such as the number of I/O, word lines (WLs), and ADC outputs. In this article, we propose a quantization training method for compressing deep models. The method comprises three steps: input and weight quantization, ReRAM convolution (ReConv), and ADC quantization. ADC quantization optimizes the error sampling problem by using the Gumbel-softmax trick. Under a 4-bit ADC of nvCIM, the accuracy only decreases by 0.05% and 1.31% for the MNIST and CIFAR-10, respectively, compared with the corresponding accuracies obtained under an ideal ADC. The experimental results indicate that the proposed method is effective for compensating the hardware limitations of nvCIM macros. INDEX TERMS Compression, computing-in-memory (CIM), deep learning, quantization, resistive random access memory (ReRAM).

Improving the accuracy and robustness of RRAM-based in-memory computing against RRAM hardware noise and adversarial attacks

“…Although the IMC technology achieves high energy efficiency, when DNNs trained in software are deployed on IMC hardware, accuracy degradation can occur due to limited ADC precision, variations in the IMC devices, ambient conditions, and transistor non-linearity [11,13,19,25,27,28]. Several recent works have attempted to address these particular issues, and representative NVM-based works are described below.…”

“…However, additional area and energy overhead are incurred in the IMC hardware due to the addition of thermal reference cells. A quantization-aware DNN training scheme was proposed in [28] which considered input and weight quantization, RRAM-based convolution, and ADC quantization. However, only up to 36 rows are activated simultaneously for IMC to limit the accuracy degradation, and still, >2% accuracy loss is reported for the CIFAR-10 dataset.…”

A Review of the Gumbel-max Trick and its Extensions for Discrete Stochasticity in Machine Learning

“…Quantization is a discrete process, making it a suitable candidate to be learned using a Gumbel-based estimator. The authors of [61], [62] propose to learn the quantization levels, which form a quantization codebook, whereas the authors of [63] learn to select such a codebook as whole. Moreover, dataadaptive binarization has been learned as well [64], [65], [66].…”

CODEX: Stochastic Encoding Method to Relax Resistive Crossbar Accelerator Design Requirements

“…While effective in reducing ADC resolution requirements, bitslicing results in severe area overhead. The work in [14] Tony Liu, Amirali Amirsoleimani, Jianxiong Xu and Roman Genov are with the Edward S. Rogers Sr. Department of Electrical and Computer Engineering, University of Toronto, 10 King's College Road, Toronto, Ontario, Canada. Fabien Alibart, Yann Beilliard, Serge Ecoffey, and Dominique Drouin are with the Department of Electrical and Computer Engineering, University of Sherbrooke, QC, Canada.…”