Accurate Inference With Inaccurate RRAM Devices: A Joint Algorithm-Design Solution

Abstract

Resistive random access memory (RRAM) is a promising technology for energy-efficient neuromorphic accelerators. However, when a pretrained deep neural network (DNN) model is programmed to an RRAM array for inference, the model suffers from accuracy degradation due to RRAM nonidealities, such as device variations, quantization error, and stuck-at-faults. Previous solutions involving multiple readverify-write (R-V-W) to the RRAM cells require cell-by-cell compensation and, thus, an excessive amount of processing time. In this article, we propose a joint algorithm-design solution to mitigate the accuracy degradation. We first leverage knowledge distillation (KD), where the model is trained with the RRAM nonidealities to increase the robustness of the model under device variations. Furthermore, we propose random sparse adaptation (RSA), which integrates a small on-chip memory with the main RRAM array for postmapping adaptation. Only the on-chip memory is updated to recover the inference accuracy. The joint algorithm-design solution achieves the state-of-the-art accuracy of 99.41% for MNIST (LeNet-5) and 91.86% for CIFAR-10 (VGG-16) with up to 5% parameters as overhead while providing a 15-150× speedup compared with R-V-W. INDEX TERMS Convolution neural networks, device nonidealities, model robustness, neuromorphic computing, random sparse adaptation (RSA), resistive random access memory (RRAM).

A Fast Weight Transfer Method for Real-Time Online Learning in RRAM-Based Neuromorphic System

“…This would be a fundamentally necessary feature for synapse weight to learn or classify the characteristics of image patterns with analog levels. These synaptic properties related to quantization are generally consistent with those reported in the literature although it may be slightly affected by the neural network structure, the number of parameters and the complexity of the image dataset [32][33]. The weight values from software training can be converted into conductivity of synaptic devices.…”

Impact of L1 Batch Normalization on Analog Noise Resistant Property of Deep Learning Models

“…However, these large models require an order of millions of multiply-accumulate operations, which are fundamentally computational intensive operations [6]. These data-intensive operations and a large number of model parameters due to the model size mean that these models would require large memory and memory bandwidth in order to achieve reasonable performance [7]- [10]. As a result, these deep models cannot be deployed on resource-constrained edge computing devices with limited computing resources and power budget [7], [11] such as battery-powered mobile and internet of things (IoT) devices.…”

“…Hence, DNN models with L 1 /T opK BatchNorm has excellent noise-resistant property than DNN models with L 2 BatchNorm. Furthermore, the relation between the loss gradient of the weight of a model with BatchNorm and those without BatchNorm is shown in equation (10) [24] where L and L are the loss of the model with and without BatchNorm, respectively, σ j is the standard deviation of the BatchNorm, γ is the BatchNorm layer trainable parameter, where y j and ŷj are the output of the model with and without BatchNorm, respectively,∇ is the gradient, and m is the batch size.…”

Benchmarking Inference Performance of Deep Learning Models on Analog Devices

“…Despite their unprecedented level of performance and improvement in their design in recent years, DL models require high computational and energy resources during training and inference [2], [3]. The high computational resource requirement is because of the intense fundamental operations by these models, such as dot product of vector and matrix, and multiplications of matrices, during training and inference [4], [5]. This is further complicated by the large increase in the quantity of these operations with the increase in the size of the models.…”

“…This tight requirement and the desire to fix compute and memory transfer bottlenecks in current set of hardware has led to a significant interest in analog specialized hardware for DL, as they have the potential to deliver at least 2X better performance than the conventional digital hardware in both speed and energy efficiency [8], [9]. In fact, they can deliver at projected throughput of multiple tera-operations (TOPs) per seconds and also achieve femtojoule energy budgets per multiply-and-accumulate (MAC) operation [5], [10]- [12]. The improvement can be attributed to the use of non-volatile memory cross bar arrays to encode DL model weights and biases, a form of computing known as in-memory computing.…”