ApproxANN: An Approximate Computing Framework for Artificial Neural Network

Zhang, Qian; Wang, Ting; Tian, Ye; Yuan, Feng; Xu, Qiang

doi:10.7873/date.2015.0618

Cited by 134 publications

(73 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On the other hand, approximate computing (e.g., [10,11]) has been advocated to trade off computation quality for energy savings and/or performance improvement. While a majority of existing approximate computing techniques were applied for image processing applications (e.g., [12,13]), there are some recent research efforts to expand it to other areas.…”

Section: Related Work and Motivationmentioning

confidence: 99%

ApproxEigen: An approximate computing technique for large-scale eigen-decomposition

Zhang¹,

Tian²,

Wang³

et al. 2015

2015 IEEE/ACM International Conference on Computer-Aided Design (ICCAD)

Self Cite

View full text Add to dashboard Cite

Recognition, Mining, and Synthesis (RMS) applications are expected to make up much of the computing workloads of the future. Many of these applications (e.g., recommender systems and search engine) are formulated as finding eigenvalues/vectors of large-scale matrices. These applications are inherently error-tolerant, and it is often unnecessary, sometimes even impossible, to calculate all the eigenpairs. Motivated by the above, in this work, we propose a novel approximate computing technique for large-scale eigen-decomposition, namely ApproxEigen, wherein we focus on the practically-used Krylov subspace methods to find finite number of eigenpairs. With ApproxEigen, we provide a set of computation kernels with different levels of approximation for data pre-processing and solution finding, and conduct accuracy tuning under given quality constraints. Experimental results demonstrate that ApproxEigen is able to achieve significant energy-efficiency improvement while keeping high accuracy.

show abstract

Section: Related Work and Motivationmentioning

confidence: 99%

ApproxEigen: An approximate computing technique for large-scale eigen-decomposition

Zhang¹,

Tian²,

Wang³

et al. 2015

2015 IEEE/ACM International Conference on Computer-Aided Design (ICCAD)

Self Cite

View full text Add to dashboard Cite

show abstract

“…The state-ofthe-art Tensor Processor Unit (TPU) accelerator accomplishes a satisfactory accuracy with 8-bit integer operations. If the precision is fixed, different approaches can be applied to make the computing more effective, e.g., pruning [4], [5], which involves removing some connections from the DNN or introducing approximate components into the CP [6], [7]. In the prior research, significant energy savings resulting from introducing various approximations in the computational path were documented.…”

Section: Introductionmentioning

confidence: 99%

ALWANN: Automatic Layer-Wise Approximation of Deep Neural Network Accelerators without Retraining

Mrázek

Vašíček

Sekanina

et al. 2019

2019 IEEE/ACM International Conference on Computer-Aided Design (ICCAD)

110

View full text Add to dashboard Cite

The state-of-the-art approaches employ approximate computing to reduce the energy consumption of DNN hardware. Approximate DNNs then require extensive retraining afterwards to recover from the accuracy loss caused by the use of approximate operations. However, retraining of complex DNNs does not scale well. In this paper, we demonstrate that efficient approximations can be introduced into the computational path of DNN accelerators while retraining can completely be avoided. ALWANN provides highly optimized implementations of DNNs for custom low-power accelerators in which the number of computing units is lower than the number of DNN layers. First, a fully trained DNN (e.g., in TensorFlow) is converted to operate with 8-bit weights and 8-bit multipliers in convolutional layers. A suitable approximate multiplier is then selected for each computing element from a library of approximate multipliers in such a way that (i) one approximate multiplier serves several layers, and (ii) the overall classification error and energy consumption are minimized. The optimizations including the multiplier selection problem are solved by means of a multiobjective optimization NSGA-II algorithm. In order to completely avoid the computationally expensive retraining of DNN, which is usually employed to improve the classification accuracy, we propose a simple weight updating scheme that compensates the inaccuracy introduced by employing approximate multipliers. The proposed approach is evaluated for two architectures of DNN accelerators with approximate multipliers from the open-source "EvoApprox" library, while executing three versions of ResNet on CIFAR-10. We report that the proposed approach saves 30% of energy needed for multiplication in convolutional layers of ResNet-50 while the accuracy is degraded by only 0.6% (0.9% for the ResNet-14). The proposed technique and approximate layers are available as an open-source extension of TensorFlow at https://github.com/ehw-fit/tf-approximate.Index Terms-approximate computing, deep neural networks, computational path, ResNet, CIFAR-10

show abstract

“…These are multi-layered extensions of the Discriminative Restricted Boltzmann Machine (DRBM) [3], which can be used for classification in both supervised and semi-supervised settings. Our proposed AX-DBN framework extends the analysis of deterministic networks [4], [5] to the domain of stochastic neural networks. AX-DBN involves training and approximating on the cloud and performing classification on embedded hardware, as depicted in Fig.…”

Section: Introductionmentioning

confidence: 99%

AX-DBN: An Approximate Computing Framework for the Design of Low-Power Discriminative Deep Belief Networks

Colbert

Kreutz-Delgado

Das

2019

2019 International Joint Conference on Neural Networks (IJCNN)

View full text Add to dashboard Cite

The power budget for embedded hardware implementations of Deep Learning algorithms can be extremely tight. To address implementation challenges in such domains, new design paradigms, like Approximate Computing, have drawn significant attention. Approximate Computing exploits the innate error-resilience of Deep Learning algorithms, a property that makes them amenable for deployment on low-power computing platforms. This paper describes an Approximate Computing design methodology, AX-DBN, for an architecture belonging to the class of stochastic Deep Learning algorithms known as Deep Belief Networks (DBNs). Specifically, we consider procedures for efficiently implementing the Discriminative Deep Belief Network (DDBN), a stochastic neural network which is used for classification tasks, extending Approximation Computing from the analysis of deterministic to stochastic neural networks. For the purpose of optimizing the DDBN for hardware implementations, we explore the use of: (a) Limited precision of neurons and functional approximations of activation functions; (b) Criticality analysis to identify the nodes in the network which can operate at reduced precision while allowing the network to maintain target accuracy levels; and (c) A greedy search methodology with incremental retraining to determine the optimal reduction in precision for all neurons to maximize power savings. Using the AX-DBN methodology proposed in this paper, we present experimental results across several network architectures that show significant power savings under a user-specified accuracy loss constraint with respect to ideal full precision implementations.

show abstract

ApproxANN: An Approximate Computing Framework for Artificial Neural Network

Cited by 134 publications

References 0 publications

ApproxEigen: An approximate computing technique for large-scale eigen-decomposition

ApproxEigen: An approximate computing technique for large-scale eigen-decomposition

ALWANN: Automatic Layer-Wise Approximation of Deep Neural Network Accelerators without Retraining

AX-DBN: An Approximate Computing Framework for the Design of Low-Power Discriminative Deep Belief Networks

Contact Info

Product

Resources

About