Comparison and Evaluation of Activation Functions in Term of Gradient Instability in Deep Neural Networks

Liu, Xin; Zhou, Jun; Qian, Huimin

doi:10.1109/ccdc.2019.8832578

Cited by 5 publications

(1 citation statement)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To illustrate some of important contributions, the influence of the activation function in the convolutional neural network (CNN) model is studied in [24], improving a ReLU activation by construction of a novel surrogate. Theoretical analysis about gradient instability as well as the fundamental explanation for the exploding/vanishing gradient and the performances of different activation functions are given in [13].…”

Section: Introductionmentioning

confidence: 99%

“SPOCU”: scaled polynomial constant unit activation function

Kiseľák

Lü

Švihra

et al. 2020

Neural Comput & Applic

View full text Add to dashboard Cite

We address the following problem: given a set of complex images or a large database, the numerical and computational complexity and quality of approximation for neural network may drastically differ from one activation function to another. A general novel methodology, scaled polynomial constant unit activation function ''SPOCU,'' is introduced and shown to work satisfactorily on a variety of problems. Moreover, we show that SPOCU can overcome already introduced activation functions with good properties, e.g., SELU and ReLU, on generic problems. In order to explain the good properties of SPOCU, we provide several theoretical and practical motivations, including tissue growth model and memristive cellular nonlinear networks. We also provide estimation strategy for SPOCU parameters and its relation to generation of random type of Sierpinski carpet, related to the [pppq] model. One of the attractive properties of SPOCU is its genuine normalization of the output of layers. We illustrate SPOCU methodology on cancer discrimination, including mammary and prostate cancer and data from Wisconsin Diagnostic Breast Cancer dataset. Moreover, we compared SPOCU with SELU and ReLU on large dataset MNIST, which justifies usefulness of SPOCU by its very good performance.

show abstract

Section: Introductionmentioning

confidence: 99%

“SPOCU”: scaled polynomial constant unit activation function

Kiseľák

Lü

Švihra

et al. 2020

Neural Comput & Applic

View full text Add to dashboard Cite

show abstract

Campaign participation prediction with deep learning

Ayvaz

Aydoğan

Akçura

et al. 2021

Electronic Commerce Research and Applications

View full text Add to dashboard Cite

Short-term wind power forecasting by bidirectional attention mechanism LSTM and its probability interval prediction by sliding-window KDE

Liu,

Li,

2023

AIP Advances

View full text Add to dashboard Cite

Deterministic point wind power forecasting (DP-WPF) and its probability interval prediction (PIP) are indispensable to short-term peak alleviation and frequency regulation in power systems with large-scale wind power injection. To improve short-term DP-WPF by long short-term memory (LSTM), a horizontal/vertical bidirectional feature attention (BFA) based LSTM model is proposed. More specifically, the BFA-LSTM model has three parts: first, multivariate time series are fed into LSTM to extract long-short-term temporal features; second, the LSTM outputs are processed horizontally as well as vertically for retrieving step-wise/multistep-wise temporal features, respectively, namely, in the bidirectional attention sense; third, both the horizontal and vertical attention weights are adaptively adjusted according to the feature importance in DP-WPF. Cases comparison shows that the suggested modeling is stably superior to most common counterparts. To address PIP by kernel density estimation (KDE), sliding-window KDE is leveraged for probability analysis. More precisely, probability density functions (PDF) and probability intervals are estimated with sliding-window samples, which are non-parametric operations and involve finitely many local samples. Superior performances of PIP by sliding-window KDE at different confidence levels indicate that the sliding-window PDF approach is highly effective in contrast to those with all samples.

show abstract

Comparison and Evaluation of Activation Functions in Term of Gradient Instability in Deep Neural Networks

Cited by 5 publications

References 7 publications

“SPOCU”: scaled polynomial constant unit activation function

“SPOCU”: scaled polynomial constant unit activation function

Campaign participation prediction with deep learning

Short-term wind power forecasting by bidirectional attention mechanism LSTM and its probability interval prediction by sliding-window KDE

Contact Info

Product

Resources

About