Uncertainty Modeling of Emerging Device based Computing-in-Memory Neural Accelerators with Application to Neural Architecture Search

Abstract

Emerging device-based Computing-in-memory (CiM) has been proved to be a promising candidate for high-energy efficiency deep neural network (DNN) computations. However, most emerging devices suffer uncertainty issues, resulting in a difference between actual data stored and the weight value it is designed to be. This leads to an accuracy drop from trained models to actually deployed platforms. In this work, we offer a thorough analysis of the effect of such uncertainties-induced changes in DNN models. To reduce the impact of device uncertainties, we propose UAE, an uncertaintyaware Neural Architecture Search scheme to identify a DNN model that is both accurate and robust against device uncertainties.

On the Reliability of Computing-in-Memory Accelerators for Deep Neural Networks

“…Thus this observation generalizes across different DNN models targeting classification tasks. With this conclusion, the authors of [49] claim that, with any independent and identically distributed Gaussian noise on weight, the output vector of the same input image follows a multi-dimensional Gaussian distribution over different samples of noise.…”

“…After finishing modeling the device variations, we can then investigate the impact of device variations on nvCiM DNN accelerators. A typical study is to evaluate such impact on an accelerator targeting image classification tasks [49]. In this section, we introduce the findings of the authors of [49].…”

“…A typical study is to evaluate such impact on an accelerator targeting image classification tasks [49]. In this section, we introduce the findings of the authors of [49].…”

“…This reliability issue and its impact on DNN performance have been studied from different levels of design, including behavioral level explorations [50,15], architecture level analysis [49], and device-level observations [56]. Cross-layer codesign efforts that simultaneously explore DNN model and hardware design pairs that can together achieve both high perception task performance and desirable hardware reliability are the current direction of this field [23].…”

SWIM: Selective Write-Verify for Computing-in-Memory Neural Accelerators

“…Different strategies have been proposed to tackle these issues. Noise-aware training [4] and uncertainty-aware neural architecture search [10][11][12] aim at fortifying DNNs so that their performance remains mostly unaffected even in the presence of device variations. However, these methods are not economical because they require re-training DNNs from scratch and cannot make use of existing pre-trained models.…”