Training high-performance and large-scale deep neural networks with full 8-bit integers

Yang, Yukuan; Deng, Lei; Wu, Shuang; Yan, Tianyi; Xie, Yuan; Li, Guoqi

doi:10.1016/j.neunet.2019.12.027

Cited by 95 publications

(67 citation statements)

References 23 publications

(33 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Table 2 presents a comparison of the accuracies with those obtained from earlier studies [13] that used 16-bit DFP and [17] that used 8-bit DFP for quantized DNN training. The proposed method achieved a smaller accuracy degradation than the that of conventional DFP8.…”

Section: Resultsmentioning

confidence: 99%

S-DFP: shifted dynamic fixed point for quantized deep neural network training

Sakai

Tamiya

2021

Neural Comput & Applic

View full text Add to dashboard Cite

Recent advances in deep neural networks have achieved higher accuracy with more complex models. Nevertheless, they require much longer training time. To reduce the training time, training methods using quantized weight, activation, and gradient have been proposed. Neural network calculation by integer format improves the energy efficiency of hardware for deep learning models. Therefore, training methods for deep neural networks with fixed point format have been proposed. However, the narrow data representation range of the fixed point format degrades neural network accuracy. In this work, we propose a new fixed point format named shifted dynamic fixed point (S-DFP) to prevent accuracy degradation in quantized neural networks training. S-DFP can change the data representation range of dynamic fixed point format by adding bias to the exponent. We evaluated the effectiveness of S-DFP for quantized neural network training on the ImageNet task using ResNet-34, ResNet-50, ResNet-101 and ResNet-152. For example, the accuracy of quantized ResNet-152 is improved from 76.6% with conventional 8-bit DFP to 77.6% with 8-bit S-DFP.

show abstract

Section: Resultsmentioning

confidence: 99%

S-DFP: shifted dynamic fixed point for quantized deep neural network training

Sakai

Tamiya

2021

Neural Comput & Applic

View full text Add to dashboard Cite

show abstract

“…The significant reduction of DRAM access is the major source of Shift-BNN's high energy efficiency. As various lowerprecision training techniques [4,23,62] Scalability to larger sample size. In some high-risk applications, one may need a more robust BNN model to make decisions, thus requires training BNNs with a larger sample size to strictly approximate the loss function in Eq.1.…”

Section: Evaluation Resultsmentioning

confidence: 99%

Shift-BNN: Highly-Efficient Probabilistic Bayesian Neural Network Training via Memory-Friendly Pattern Retrieving

Wan

Xia

Zhang

et al. 2021

MICRO-54: 54th Annual IEEE/ACM International Symposium on Microarchitecture

View full text Add to dashboard Cite

Bayesian Neural Networks (BNNs) that possess a property of uncertainty estimation have been increasingly adopted in a wide range of safety-critical AI applications which demand reliable and robust decision making, e.g., self-driving, rescue robots, medical image diagnosis. The training procedure of a probabilistic BNN model involves training an ensemble of sampled DNN models, which induces orders of magnitude larger volume of data movement than training a single DNN model. In this paper, we reveal that the root cause for BNN training inefficiency originates from the massive off-chip data transfer by Gaussian Random Variables (GRVs). To tackle this challenge, we propose a novel design that eliminates all the off-chip data transfer by GRVs through the reversed shifting of Linear Feedback Shift Registers (LFSRs) without incurring any training accuracy loss. To efficiently support our LFSR reversion strategy at the hardware level, we explore the design space of the current DNN accelerators and identify the optimal computation mapping scheme to best accommodate our strategy. By leveraging this finding, we design and prototype the first highly efficient BNN training accelerator, named Shift-BNN, that is low-cost and scalable. Extensive evaluation on five representative BNN models demonstrates that Shift-BNN achieves an average of 4.9× (up to 10.8×) boost in energy efficiency and 1.6× (up to 2.8×) speedup over the baseline DNN training accelerator. CCS CONCEPTS• Computer systems organization → Neural networks; • Hardware → Hardware accelerators.

show abstract

“…Although researchers proposed many approaches to quantize weights and activations, very little work was able to quantize the gradients [16,18]. What is more, the derivatives of most quantization functions are almost everywhere zero.…”

Section: Problem Formulationmentioning

confidence: 99%

Training Quantized Deep Neural Networks via Cooperative Coevolution

Fu¹,

Liu²,

Lü³

et al. 2021

Preprint

View full text Add to dashboard Cite

Quantizing deep neural networks (DNNs) has been a promising solution for deploying deep neural networks on embedded devices. However, most of the existing methods do not quantize gradients, and the process of quantizing DNNs still has a lot of floating-point operations, which hinders the further applications of quantized DNNs. To solve this problem, we propose a new heuristic method based on cooperative coevolution for quantizing DNNs. Under the framework of cooperative coevolution, we use the estimation of distribution algorithm to search for the low-bits weights. Specifically, we first construct an initial quantized network from a pre-trained network instead of random initialization and then start searching from it by restricting the search space. So far, the problem is the largest discrete problem known to be solved by evolutionary algorithms. Experiments show that our method can train 4 bit ResNet-20 on the Cifar-10 dataset without sacrificing accuracy.

show abstract

Training high-performance and large-scale deep neural networks with full 8-bit integers

Cited by 95 publications

References 23 publications

S-DFP: shifted dynamic fixed point for quantized deep neural network training

S-DFP: shifted dynamic fixed point for quantized deep neural network training

Shift-BNN: Highly-Efficient Probabilistic Bayesian Neural Network Training via Memory-Friendly Pattern Retrieving

Training Quantized Deep Neural Networks via Cooperative Coevolution

Contact Info

Product

Resources

About