AccuReD: High Accuracy Training of CNNs on ReRAM/GPU Heterogeneous 3-D Architecture

Joardar, Biresh Kumar; Doppa, Janardhan Rao; Pande, Partha Pratim; Li, Hai; Chakrabarty, Krishnendu

doi:10.1109/tcad.2020.3013194

Cited by 36 publications

(14 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Joardar et al [ 69 ] implement a 32- to 16-bit fixed-point arithmetic SR unit in an in-memory computing device for neural network training, based on resistive random access memory (ReRAM), which uses an LFSR 16-bit pseudo-random number generator. They report that SR added negligible ReRAM cell area overhead with each SR circuit adding less than 1%.…”

Section: Applicationsmentioning

confidence: 99%

Stochastic rounding: implementation, error analysis and applications

et al. 2022

View full text Add to dashboard Cite

Stochastic rounding (SR) randomly maps a real number x to one of the two nearest values in a finite precision number system. The probability of choosing either of these two numbers is 1 minus their relative distance to x . This rounding mode was first proposed for use in computer arithmetic in the 1950s and it is currently experiencing a resurgence of interest. If used to compute the inner product of two vectors of length n in floating-point arithmetic, it yields an error bound with constant n u with high probability, where u is the unit round-off. This is not necessarily the case for round to nearest (RN), for which the worst-case error bound has constant nu . A particular attraction of SR is that, unlike RN, it is immune to the phenomenon of stagnation, whereby a sequence of tiny updates to a relatively large quantity is lost. We survey SR by discussing its mathematical properties and probabilistic error analysis, its implementation, and its use in applications, with a focus on machine learning and the numerical solution of differential equations.

show abstract

Section: Applicationsmentioning

confidence: 99%

Stochastic rounding: implementation, error analysis and applications

et al. 2022

View full text Add to dashboard Cite

show abstract

“…In [29], the authors present DNNs that are tolerant to ambient temperature fluctuations. However, additional area and energy overhead are incurred in the IMC hardware due to the addition of thermal reference cells.…”

Section: Hardware-aware Dnn Training For Accurate Dnn Inference With ...mentioning

confidence: 99%

Improving the accuracy and robustness of RRAM-based in-memory computing against RRAM hardware noise and adversarial attacks

Cherupally¹,

Meng²,

Rakin³

et al. 2022

Semicond. Sci. Technol.

View full text Add to dashboard Cite

We present a novel deep neural network (DNN) training scheme and RRAM in-memory computing (IMC) hardware evaluation towards achieving high robustness to the RRAM device/array variations and adversarial input attacks. We present improved IMC inference accuracy results evaluated on state-of-the-art DNNs including ResNet-18, AlexNet, and VGG with binary, 2-bit, and 4-bit activation/weight precision for the CIFAR-10 dataset. These DNNs are evaluated with measured noise data obtained from three different RRAM-based IMC prototype chips. Across these various DNNs and IMC chip measurements, we show that our proposed hardware noise-aware DNN training consistently improves DNN inference accuracy for actual IMC hardware, up to 8% accuracy improvement for the CIFAR-10 dataset. We also analyze the impact of our proposed noise injection scheme on the adversarial robustness of ResNet-18 DNNs with 1-bit, 2-bit, and 4-bit activation/weight precision. Our results show up to 6% improvement in the robustness to black-box adversarial input attacks.

show abstract

“…We start with a pre-trained model to initialize noise-aware training. Batch normalization layers are included after each Conv layer [42], [43]. The computation during the training process considers the hardware configurations.…”

Section: Resna: Hardware-aware Training Approachmentioning

confidence: 99%

Multi-Objective Optimization of ReRAM Crossbars for Robust DNN Inferencing under Stochastic Noise

Yang¹,

Belakaria²,

Joardar³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Resistive random-access memory (ReRAM) is a promising technology for designing hardware accelerators for deep neural network (DNN) inferencing. However, stochastic noise in ReRAM crossbars can degrade the DNN inferencing accuracy. We propose the design and optimization of a high-performance, area-and energy-efficient ReRAMbased hardware accelerator to achieve robust DNN inferencing in the presence of stochastic noise. We make two key technical contributions. First, we propose a stochastic-noise-aware training method, referred to as ReSNA, to improve the accuracy of DNN inferencing on ReRAM crossbars with stochastic noise. Second, we propose an informationtheoretic algorithm, referred to as CF-MESMO, to identify the Pareto set of solutions to trade-off multiple objectives, including inferencing accuracy, area overhead, execution time, and energy consumption. The main challenge in this context is that executing the ReSNA method to evaluate each candidate ReRAM design is prohibitive. To address this challenge, we utilize the continuous-fidelity evaluation of ReRAM designs associated with prohibitive high computation cost by varying the number of training epochs to trade-off accuracy and cost. CF-MESMO iteratively selects the candidate ReRAM design and fidelity pair that maximizes the information gained per unit computation cost about the optimal Pareto front. Our experiments on benchmark DNNs show that the proposed algorithms efficiently uncover high-quality Pareto fronts. On average, ReSNA achieves 2.57% inferencing accuracy improvement for ResNet20 on the CIFAR-10 dataset with respect to the baseline configuration. Moreover, CF-MESMO algorithm achieves 90.91% reduction in computation cost compared to the popular multi-objective optimization algorithm NSGA-II to reach the best solution from NSGA-II.

show abstract

AccuReD: High Accuracy Training of CNNs on ReRAM/GPU Heterogeneous 3-D Architecture

Cited by 36 publications

References 32 publications

Stochastic rounding: implementation, error analysis and applications

Stochastic rounding: implementation, error analysis and applications

Improving the accuracy and robustness of RRAM-based in-memory computing against RRAM hardware noise and adversarial attacks

Multi-Objective Optimization of ReRAM Crossbars for Robust DNN Inferencing under Stochastic Noise

Contact Info

Product

Resources

About