GAPCNN with HyPar: Global Average Pooling convolutional neural network with novel NNLU activation function and HYBRID parallelism

Abstract

With the increasing demand for deep learning in the last few years, CNNs have been widely used in many applications and have gained interest in classification, regression, and image recognition tasks. The training of these deep neural networks is compute-intensive and takes days or even weeks to train the model from scratch. The compute-intensive nature of these deep neural networks sometimes limits the practical implementation of CNNs in real-time applications. Therefore, the computational speedup in these networks is of utmost importance, which generates interest in CNN training acceleration. Much research is going on to meet the computational requirement and make it feasible for real-time applications. Because of its simplicity, data parallelism is used primarily, but it performs badly sometimes. In most cases, researchers prefer model parallelism to data parallelism, but it is not always the best choice. Therefore, in this study, we implement a hybrid of both data and model parallelism to improve the computational speed without compromising accuracy. There is only a 1.5% accuracy drop in our proposed study with an increased speed up of 3.62X. Also, a novel activation function Normalized Non-linear Activation Unit NNLU is proposed to introduce non-linearity in the model. The activation unit is non-saturated and helps avoid the model's over-fitting. The activation unit is free from the vanishing gradient problem. Also, the fully connected layer in the proposed CNN model is replaced by the Global Average Pooling layers (GAP) to enhance the model's accuracy and computational performance. When tested on a bio-medical image dataset, the model achieves an accuracy of 98.89% and requires a training time of only 1 s. The model categorizes medical images into different categories of glioma, meningioma, and pituitary tumor. The model is compared with existing state-of-art techniques, and it is observed that the proposed model outperforms others in classification accuracy and computational speed. Also, results are observed for different optimizers', different learning rates, and various epoch numbers.

Physical Activity Detection for Diabetes Mellitus Patients Using Recurrent Neural Networks

“…Afterwards, a final batch normalization layer was introduced to ensure the normalization of the data. Following this, a Global Average Pooling [ 41 ] layer is incorporated to generate a single vector from multiple time vectors. This is achieved by computing the average.…”

Significant wave height prediction from X-band marine radar images using deep learning with 3D convolutions

“…The features extracted by the convolutional layer are transformed into one-dimensional data by the global average pooling (GAP) layer, which is passed to the FC output layer. GAP converts the feature map into a one-dimensional vector, like the flatten layer but is known to improve the performance of the network compared to the flatten layer [ 21 ]. The FC output layer uses sigmoid as the activation function for the regression task, and the number of output nodes is one.…”

DFPT-CNN: A Dual Feature Extraction and Pretrained CNN Synergy for Minimal Computational Overhead and Enhanced Accuracy in Multi-Class Medical Image Classification

“…In [29], a streamlined sequential CNN architecture was proposed and compared with two pre-trained CNN models (VGG-16 and InceptionV3) for diagnosing diseases from chest X-ray images, achieving a peak accuracy of 89.89%. In the work conducted by [30], a hybrid parallelization strategy, incorporating both model and data parallelism, was applied to CNN to speed up its operation, with a normalized non-linearity activation function (NNLU) and replacing FC layers with Global Average Pooling (GAP) layers. These changes significantly improved accuracy and increased computation speed by 3.62 times with a batch size of (512,128), achieving a validation accuracy of approximately 98 with minimal training loss.…”