Comparative Study of Convolution Neural Network’s Relu and Leaky-Relu Activation Functions

Ak, Dutta; Jain, Vanita

doi:10.1007/978-981-13-6772-4_76

Cited by 134 publications

(58 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The input is one-dimensional feature vector X =[ x 1 , x 2 , x 3 ,…, x N ] with a length of N corresponding to EEG signals of N channels; the convolution layer is composed of K convolution kernels, the size of each convolution kernel is 1 ∗S , the coefficient of the convolution kernel is w k ∈ R s , k =1,2,…, K , and the output is h =[ h 1 , h 2 , h 3 ,…, h k ] ∈ R ( N − S +1) K , where

where b k denotes the bias of the convolution kernel and R denotes the nonlinear activation function that adopts the Leaky ReLU function [ 39 , 40 ].

…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

A Novel Time-Incremental End-to-End Shared Neural Network with Attention-Based Feature Fusion for Multiclass Motor Imagery Recognition

Lian

Zuo

et al. 2021

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

In the research of motor imagery brain-computer interface (MI-BCI), traditional electroencephalogram (EEG) signal recognition algorithms appear to be inefficient in extracting EEG signal features and improving classification accuracy. In this paper, we discuss a solution to this problem based on a novel step-by-step method of feature extraction and pattern classification for multiclass MI-EEG signals. First, the training data from all subjects is merged and enlarged through autoencoder to meet the need for massive amounts of data while reducing the bad effect on signal recognition because of randomness, instability, and individual variability of EEG data. Second, an end-to-end sharing structure with attention-based time-incremental shallow convolution neural network is proposed. Shallow convolution neural network (SCNN) and bidirectional long short-term memory (BiLSTM) network are used to extract frequency-spatial domain features and time-series features of EEG signals, respectively. Then, the attention model is introduced into the feature fusion layer to dynamically weight these extracted temporal-frequency-spatial domain features, which greatly contributes to the reduction of feature redundancy and the improvement of classification accuracy. At last, validation tests using BCI Competition IV 2a data sets show that classification accuracy and kappa coefficient have reached 82.7 ± 5.57% and 0.78 ± 0.074, which can strongly prove its advantages in improving classification accuracy and reducing individual difference among different subjects from the same network.

show abstract

where b k denotes the bias of the convolution kernel and R denotes the nonlinear activation function that adopts the Leaky ReLU function [ 39 , 40 ].

…”

Section: Methodsmentioning

confidence: 99%

“…where b k denotes the bias of the convolution kernel and R denotes the nonlinear activation function that adopts the Leaky ReLU function [39,40].…”

Section: Shallow Convolution Neural Networkmentioning

confidence: 99%

A Novel Time-Incremental End-to-End Shared Neural Network with Attention-Based Feature Fusion for Multiclass Motor Imagery Recognition

Lian

Zuo

et al. 2021

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

show abstract

“…Similarly, subtask 2 is allocated for identifying the health state of the bearing, which is composed of two convolution layers, one pooling layer, two fully connected layers, and the final output layers. For the activation of the fully connected layers of this framework, leaky rectified linear unit (Leaky ReLU) [ 65 ] is considered. To prevent the overfitting problem, L2 regularization with a value of 0.04 is attained on the layer before the output layer.…”

Section: Proposed Methodologymentioning

confidence: 99%

A Multitask-Aided Transfer Learning-Based Diagnostic Framework for Bearings under Inconsistent Working Conditions

Hasan

Sohaib

Kim

2020

Sensors

View full text Add to dashboard Cite

Rolling element bearings are a vital part of rotating machines and their sudden failure can result in huge economic losses as well as physical causalities. Popular bearing fault diagnosis techniques include statistical feature analysis of time, frequency, or time-frequency domain data. These engineered features are susceptible to variations under inconsistent machine operation due to the non-stationary, non-linear, and complex nature of the recorded vibration signals. To address these issues, numerous deep learning-based frameworks have been proposed in the literature. However, the logical reasoning behind crack severities and the longer training times needed to identify multiple health characteristics at the same time still pose challenges. Therefore, in this work, a diagnosis framework is proposed that uses higher-order spectral analysis and multitask learning (MTL), while also incorporating transfer learning (TL). The idea is to first preprocess the vibration signals recorded from a bearing to look for distinct patterns for a given fault type under inconsistent working conditions, e.g., variable motor speeds and loads, multiple crack severities, compound faults, and ample noise. Later, these bispectra are provided as an input to the proposed MTL-based convolutional neural network (CNN) to identify the speed and the health conditions, simultaneously. Finally, the TL-based approach is adopted to identify bearing faults in the presence of multiple crack severities. The proposed diagnostic framework is evaluated on several datasets and the experimental results are compared with several state-of-the-art diagnostic techniques to validate the superiority of the proposed model under inconsistent working conditions.

show abstract

“…Some changes were made to the default UNet architecture. First, we used the leaky rectified linear unit (ReLU) activation function as opposed to a traditional ReLU in the convolutional layers to avoid "dying ReLU" issues [84]. We also used the AdaMax optimizer [85] instead of Adam (Adaptive momentum estimation) or RMSProp (Root Mean Square Propagation) and included a callback to reduce the learning rate if the loss plateaued for more than 5 epochs.…”

Section: Modeling Trainingmentioning

confidence: 99%

Semantic Segmentation Deep Learning for Extracting Surface Mine Extents from Historic Topographic Maps

et al. 2020

View full text Add to dashboard Cite

Historic topographic maps, which are georeferenced and made publicly available by the United States Geological Survey (USGS) and the National Map’s Historical Topographic Map Collection (HTMC), are a valuable source of historic land cover and land use (LCLU) information that could be used to expand the historic record when combined with data from moderate spatial resolution Earth observation missions. This is especially true for landscape disturbances that have a long and complex historic record, such as surface coal mining in the Appalachian region of the eastern United States. In this study, we investigate this specific mapping problem using modified UNet semantic segmentation deep learning (DL), which is based on convolutional neural networks (CNNs), and a large example dataset of historic surface mine disturbance extents from the USGS Geology, Geophysics, and Geochemistry Science Center (GGGSC). The primary objectives of this study are to (1) evaluate model generalization to new geographic extents and topographic maps and (2) to assess the impact of training sample size, or the number of manually interpreted topographic maps, on model performance. Using data from the state of Kentucky, our findings suggest that DL semantic segmentation can detect surface mine disturbance features from topographic maps with a high level of accuracy (Dice coefficient = 0.902) and relatively balanced omission and commission error rates (Precision = 0.891, Recall = 0.917). When the model is applied to new topographic maps in Ohio and Virginia to assess generalization, model performance decreases; however, performance is still strong (Ohio Dice coefficient = 0.837 and Virginia Dice coefficient = 0.763). Further, when reducing the number of topographic maps used to derive training image chips from 84 to 15, model performance was only slightly reduced, suggesting that models that generalize well to new data and geographic extents may not require a large training set. We suggest the incorporation of DL semantic segmentation methods into applied workflows to decrease manual digitizing labor requirements and call for additional research associated with applying semantic segmentation methods to alternative cartographic representations to supplement research focused on multispectral image analysis and classification.

show abstract

Comparative Study of Convolution Neural Network’s Relu and Leaky-Relu Activation Functions

Cited by 134 publications

References 7 publications

A Novel Time-Incremental End-to-End Shared Neural Network with Attention-Based Feature Fusion for Multiclass Motor Imagery Recognition

A Novel Time-Incremental End-to-End Shared Neural Network with Attention-Based Feature Fusion for Multiclass Motor Imagery Recognition

A Multitask-Aided Transfer Learning-Based Diagnostic Framework for Bearings under Inconsistent Working Conditions

Semantic Segmentation Deep Learning for Extracting Surface Mine Extents from Historic Topographic Maps

Contact Info

Product

Resources

About