Bactran: A Hardware Batch Normalization Implementation for CNN Training Engine

Yang, Zhijie; Wang, Lei; Luo, Li; Li, Shiming; Guo, Shasha; Wang, Shuquan

doi:10.1109/les.2020.2975055

Cited by 14 publications

(15 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On the other hand, General Processing Units (GPUs) supply more resources in terms of memory and computational units. But, during the validation process, GPUs have a long execution time because of sequential logic design (Wang et al 1999 ; Zhijie et al 2020 ). FPGA has parallel architectures on its logic configuration.…”

Section: Introductionmentioning

confidence: 99%

“…This feature makes it better choice for hardware designers because of the validation process of CNN configurations. At the implementation stage of CNN models, normalization process ensures fewer number of hardware resources and reduced power consumption during the training and validation processes on any hardware platforms (Wang et al 1999 ; Zhijie et al 2020 ; Nurvitadhi et al 2016 ; Baptista et al 2020 ).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A novel normalization algorithm to facilitate pre-assessment of Covid-19 disease by improving accuracy of CNN and its FPGA implementation

2022

View full text Add to dashboard Cite

COVID-19 is still a fatal disease, which has threatened all people by affecting the human lungs. Chest X-Ray or computed tomography imaging is commonly used to make a fast and reliable medical investigation to detect the COVID-19 virus. These medical images are remarkably challenging because it is a full-time job and prone to human errors. In this paper, a new normalization algorithm that consists of Mean–Variance-Softmax-Rescale (MVSR) processes respectively is proposed to provide facilitation pre-assessment and diagnosis Covid-19 disease. In order to show the effect of MVSR normalization technique, the algorithm of proposed method is applied to chest X-ray and Sars-Cov-2 computed tomography images dataset. The normalized X-ray images with MVSR are used to recognize Covid-19 virus via Convolutional Neural Network (CNN) model. At the implementation stage, the MVSR algorithm is executed on MATLAB environment, then all the arithmetic operations of the MVSR normalization are coded in VHDL with the help of fixed-point fractional number representation format on FPGA platform. The experimental platform consists of Zynq-7000 Development FPGA Board and VGA monitor to display the both original and MVSR normalized chest X-ray images. The CNN model is constructed and executed using Anaconda Navigator interface with python language. Based on the results of this study, infections of Covid-19 disease can be easily diagnosed with MVSR normalization technique. The proposed MVSR normalization technique increased the classification accuracy of the CNN model from 83.01, to 96.16% for binary class of chest X-ray images.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

A novel normalization algorithm to facilitate pre-assessment of Covid-19 disease by improving accuracy of CNN and its FPGA implementation

2022

View full text Add to dashboard Cite

show abstract

“…FPGA has parallel architectures on its logic con guration and this feature makes it better choice for hardware designers during validation process of CNN con gurations. At the implementation stage of CNN models on any hardware, normalization process ensures fewer number of hardware resources and reduced power consumption during the training and validation processes [22][23][24][25].…”

Section: Introductionmentioning

confidence: 99%

“…On the other hand, General Processing Units (GPUs) supply more resources in terms of memory and computational units. But, during the validation process, GPUs have a long execution time because of sequential logic design [22,23]. FPGA has parallel architectures on its logic con guration and this feature makes it better choice for hardware designers during validation process of CNN con gurations.…”

Section: Introductionmentioning

confidence: 99%

A Novel Normalization Algorithm to Facilitate Pre-Assessment of Covid-19 Disease from Chest X-ray Image and its FPGA implementation

Yaman

Karakaya

Erol

2022

Preprint

View full text Add to dashboard Cite

COVID-19 is still a fatal disease, which has threatened all people by affecting the human lungs. Chest X-Ray or computed tomography (CT) imaging is commonly used to make a fast and reliable medical investigation to detect the COVID-19 virus from these medical images is remarkably challenging because it is a full-time job and prone to human errors. In this paper, a new normalization algorithm that consists of Mean-Variance-Softmax-Rescale (MVSR) processes respectively is proposed to provide facilitation pre-assessment and diagnosis Covid-19 disease. In order to show the effect of MVSR normalization technique on image processing, the algorithm is applied to chest X-ray images. Therefore, the normalized X-ray images with MVSR are used to recognize via one of the neural network models as known Convolutional Neural Networks (CNNs). At the implementation stage, the MVSR algorithm is executed on MATLAB environment, then it is implemented on FPGA platform. All the arithmetic operations of the MVSR normalization are coded in VHDL with the help of fixed-point fractional number representation format. The experimental platform consists of Zynq-7000 Development Board and VGA monitor to display the both original X-ray and MVSR normalized image. The CNN model is constructed and executed using Anaconda Navigator interface with python language. Based on the results of this study, infections of Covid-19 disease can be easily diagnosed for MVSR normalized image. The proposed MVSR normalization makes the accuracy of CNN model increase from 83.01%, to 96.16% for binary class of chest X-ray images.

show abstract

“…Last but not least, the backward propagation requires saving the pre-normalized activations, thus occupying roughly twice the memory as a non-BN network in the training phase [16]. For these reasons, BN becomes the most crucial part in non-convolution layers [17] because it involves about 58.5% of the execution time and 90% operations of non-convolution layers [18]. Research also shows that BN lowers the overall training speed by > 47% for deep ResNet and DenseNet models [19] and brings incredible difficulties in the special DNN inference and online training accelerator architecture design [18].…”

Section: Introductionmentioning

confidence: 99%

Towards Efficient Full 8-bit Integer DNN Online Training on Resource-limited Devices without Batch Normalization

Yang¹,

Chi²,

Deng³

et al. 2021

Preprint

View full text Add to dashboard Cite

Huge computational costs brought by convolution and batch normalization (BN) have caused great challenges for the online training and corresponding applications of deep neural networks (DNNs), especially in resource-limited devices. Existing works only focus on the convolution or BN acceleration and no solution can alleviate both problems with satisfactory performance. Online training has gradually become a trend in resource-limited devices like mobile phones while there is still no complete technical scheme with acceptable model performance, processing speed, and computational cost. In this research, an efficient online-training quantization framework termed EOQ is proposed by combining Fixup initialization and a novel quantization scheme for DNN model compression and acceleration. Based on the proposed framework, we have successfully realized full 8-bit integer network training and removed BN in largescale DNNs. Especially, weight updates are quantized to 8-bit integers for the first time. Theoretical analyses of EOQ utilizing Fixup initialization for removing BN have been further given using a novel Block Dynamical Isometry theory with weaker assumptions. Benefiting from rational quantization strategies and the absence of BN, the full 8-bit networks based on EOQ can achieve state-of-the-art accuracy and immense advantages in computational cost and processing speed. In addition to the huge advantages brought by quantization in convolution operations, 8-bit networks based on EOQ without BN can realize >70× lower in power, >18× faster in the processing speed compared with the traditional 32-bit floating-point BN inference process. What's more, the design of deep learning chips can be profoundly simplified for the absence of unfriendly square root operations in BN. Beyond this, EOQ has been evidenced to be more advantageous in small-batch online training with fewer batch samples. In summary, the EOQ framework is specially designed for reducing the high cost of convolution and BN in network training, demonstrating a broad application prospect of online training in resource-limited devices.

show abstract

Bactran: A Hardware Batch Normalization Implementation for CNN Training Engine

Cited by 14 publications

References 7 publications

A novel normalization algorithm to facilitate pre-assessment of Covid-19 disease by improving accuracy of CNN and its FPGA implementation

A novel normalization algorithm to facilitate pre-assessment of Covid-19 disease by improving accuracy of CNN and its FPGA implementation

A Novel Normalization Algorithm to Facilitate Pre-Assessment of Covid-19 Disease from Chest X-ray Image and its FPGA implementation

Towards Efficient Full 8-bit Integer DNN Online Training on Resource-limited Devices without Batch Normalization

Contact Info

Product

Resources

About