A Legendre polynomial based activation function: An aid for modeling of max pooling

Venkatappareddy, P.; Culli, Jayanth; Srivastava, Siddharth; Lall, Brejesh

doi:10.1016/j.dsp.2021.103093

Cited by 6 publications

(3 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…with starting values G 0 (z) = 1 and G 1 (z) = z [575]. The LPAF was found to outperform ELU, ReLU, LReLU, and softplus on the MNIST [182] and Fashion MNIST [314] datasets [575].…”

Section: Legendre Polynomial-based Activation Function (Lpaf)mentioning

confidence: 98%

See 1 more Smart Citation

On Transformative Adaptive Activation Functions in Neural Networks for Gene Expression Inference

Kunc

Kléma²

2019

Preprint

View full text Add to dashboard Cite

Motivation: Gene expression profiling was made cheaper by the NIH LINCS program that profiles only ∼1, 000 selected landmark genes and uses them to reconstruct the whole profile. The D-GEX method employs neural networks to infer the whole profile. However, the original D-GEX can be further significantly improved. Results: We have analyzed the D-GEX method and determined that the inference can be improved using a logistic sigmoid activation function instead of the hyperbolic tangent. Moreover, we propose a novel transformative adaptive activation function that improves the gene expression inference even further and which generalizes several existing adaptive activation functions. Our improved neural network achieves average mean absolute error of 0.1340 which is a significant improvement over our reimplementation of the original D-GEX which achieves average mean absolute error 0.1637

show abstract

“…with starting values G 0 (z) = 1 and G 1 (z) = z [575]. The LPAF was found to outperform ELU, ReLU, LReLU, and softplus on the MNIST [182] and Fashion MNIST [314] datasets [575].…”

Section: Legendre Polynomial-based Activation Function (Lpaf)mentioning

confidence: 98%

“…A Legendre polynomial-based activation function (LPAF) was used for the study of approximations of several nonlinearities in [575]. The activation is a linear combination of Legendre polynomials and is defined as…”

Section: Legendre Polynomial-based Activation Function (Lpaf)mentioning

confidence: 99%

On Transformative Adaptive Activation Functions in Neural Networks for Gene Expression Inference

Kunc

Kléma²

2019

Preprint

View full text Add to dashboard Cite

show abstract

“…The PFLU and FPFLU [37] are nonmonotonic, without exponential terms, and have a square root in the denominator, increasing the computational complexity. Venkatappareddy et al [38] proposed a polynomial-based activation function; Hegui et al [39] proposed a nonmonotonic activation function called Logish; both show high accuracy on the CIFAR-10 dataset. TanhSoft-1, TanhSoft-2, and TanhSoft-3 [40] are combinations of Tanh; the exponential terms also increase the computational complexity.…”

Section: Introductionmentioning

confidence: 99%

PolyLU: A Simple and Robust Polynomial-Based Linear Unit Activation Function for Deep Learning

Feng,

Yang

2023

IEEE Access

View full text Add to dashboard Cite

The activation function has a critical influence on whether a convolutional neural network in deep learning can converge or not; a proper activation function not only makes the convolutional neural network converge faster but also can reduce the complexity of convolutional neural network architecture and gets the same or better performance. Many activation functions have been proposed; however, various activation functions have advantages, defects, and applicable network architectures. A new activation function called Polynomial Linear Unit (PolyLU) is proposed in this paper to improve some of the shortcomings of the existing activation functions. The PolyLU meets the following basic properties: continuously differentiable, approximate identity near the origin, unbounded for positive inputs, bounded for negative inputs, smooth, monotonic, and zero-centered. There is a polynomial term for the negative inputs and no exponential terms in the PolyLU that reduces the computational complexity of the network. Compared to those common activation functions like Sigmoid, Tanh, ReLU, LeakyReLU, ELU, Mish, and Swish, the experiments show that the PolyLU has improved some network complexity and has better accuracy over MNIST, Kaggle Cats and Dogs, CIFAR-10 and CIFAR-100 datasets. Test by the CIFAR-100 dataset with batch normalization, PolyLU improves by 0.

show abstract

Development of a novel activation function based on Chebyshev polynomials: an aid for classification and denoising of images

2023

View full text Add to dashboard Cite

A Legendre polynomial based activation function: An aid for modeling of max pooling

Cited by 6 publications

References 11 publications

On Transformative Adaptive Activation Functions in Neural Networks for Gene Expression Inference

On Transformative Adaptive Activation Functions in Neural Networks for Gene Expression Inference

PolyLU: A Simple and Robust Polynomial-Based Linear Unit Activation Function for Deep Learning

Development of a novel activation function based on Chebyshev polynomials: an aid for classification and denoising of images

Contact Info

Product

Resources

About