Enabling Binary Neural Network Training on the Edge

Wang, Erwei; Davis, James J.; Moro, Daniele; Zieliński, Piotr; Lim, Jia Jie; Coelho, Carlyle Mendes; Chatterjee, Saurav; Cheung, Peter Y. K.; Constantinides, George A.

doi:10.1145/3469116.3470015

Cited by 14 publications

(2 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another training direction for low memory BNN that is suitable for on-edge devices is proposed in [36] that introduced a low memory and low energy training. J. Laydevant et al [38] introduced training the BNN utilizing Equilibrium Propagation (EP) that provides the ability of on-chip training.…”

Section: ) Other Training Methodsmentioning

confidence: 99%

A Systematic Literature Review on Binary Neural Networks

et al. 2023

View full text Add to dashboard Cite

This paper presents an extensive literature review on Binary Neural Network (BNN). BNN utilizes binary weights and activation function parameters to substitute the full-precision values. In digital implementations, BNN replaces the complex calculations of Convolutional Neural Networks (CNNs) with simple bitwise operations. BNN optimizes large computation and memory storage requirements, which leads to less area and power consumption compared to full-precision models. Although there are many advantages of BNN, the binarization process has a significant impact on the performance and accuracy of the generated models. To reflect the state-of-the-art in BNN and explore how to develop and improve BNNbased models, we conduct a systematic literature review on BNN with data extracted from 239 research studies. Our review discusses various BNN architectures and the optimization approaches developed to improve their performance. There are three main research directions in BNN: accuracy optimization, compression optimization, and acceleration optimization. The accuracy optimization approaches include quantization error reduction, special regularization, gradient error minimization, and network structure. The compression optimization approaches combine fractional BNN and pruning. The acceleration optimization approaches comprise computing in-memory, FPGA-based implementations, and ASIC-based implementations. At the end of our review, we present a comprehensive analysis of BNN applications and their evaluation metrics. Also, we shed some light on the most common BNN challenges and the future research trends of BNN.

show abstract

Section: ) Other Training Methodsmentioning

confidence: 99%

A Systematic Literature Review on Binary Neural Networks

et al. 2023

View full text Add to dashboard Cite

show abstract

“…In addition, this work does not consider data formats other than floating-point and fixed-point formats. Other research has shown good potential for block floating-point [26], [39]- [42], posits [43]- [45], logarithmic number systems [46]- [49] and even binary formats [50]. The investigation of mixed data formats DNN training including those data types, especially for block mini-floats [41], is left for future work.…”

Section: Limitationsmentioning

confidence: 99%

Mixing Low-Precision Formats in Multiply-Accumulate Units for DNN Training

Tatsumi

Filip

White

et al. 2022

2022 International Conference on Field-Programmable Technology (ICFPT)

View full text Add to dashboard Cite

The most compute-intensive stage of deep neural network (DNN) training is matrix multiplication where the multiply-accumulate (MAC) operator is key. To reduce training costs, we consider using low-precision arithmetic for MAC operations. While low-precision training has been investigated in prior work, the focus has been on reducing the number of bits in weights or activations without compromising accuracy. In contrast, the focus in this paper is on implementation details beyond weight or activation width that affect area and accuracy. In particular, we investigate the impact of fixed-versus floating-point representations, multiplier rounding, and floatingpoint exceptional value support. Results suggest that (1) lowprecision floating-point is more area-effective than fixed-point for multiplication, (2) standard IEEE-754 rules for subnormals, NaNs, and intermediate rounding serve little to no value in terms of accuracy but contribute significantly to area, (3) lowprecision MACs require an adaptive loss-scaling step during training to compensate for limited representation range, and (4) fixed-point is more area-effective for accumulation, but the cost of format conversion and downstream logic can swamp the savings. Finally, we note that future work should investigate accumulation structures beyond the MAC level to achieve further gains. • a new, extensible, open-source DNN training framework called Archimedes-MPO which uses FPGA or GPU

show abstract

A comprehensive review of Binary Neural Network

Yuan

Agaian

2023

Artif Intell Rev

View full text Add to dashboard Cite

Enabling Binary Neural Network Training on the Edge

Cited by 14 publications

References 19 publications

A Systematic Literature Review on Binary Neural Networks

A Systematic Literature Review on Binary Neural Networks

Mixing Low-Precision Formats in Multiply-Accumulate Units for DNN Training

A comprehensive review of Binary Neural Network

Contact Info

Product

Resources

About