P-Swish: Activation Function with Learnable Parameters Based on Swish Activation Function in Deep Learning

Mercioni, Marina Adriana; Holban, Ştefan

doi:10.1109/isetc50328.2020.9301059

Cited by 27 publications

(10 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Many studies have tried to improve the performance of CNNs models from network architecture [22]- [25], different variants of optimization [26]- [28], activations [29]- [32], regularization methods [33], [34] and so no. However, little attention has been paid to investigating the padding schemes during the convolution operation.…”

Section: Literature Review and Related Workmentioning

confidence: 99%

“…The CIFAR-10 dataset includes a training dataset of 50,000 images and a test dataset of 10,000 images. The images are of shape (32,32,3), distributed equally to ten classes of airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck. The Padding Module was applied to different networks namely: VGG16 [8] and ResNet50V2 [39]; to make the deeper layers in these networks carry out a valid convolution, the images were resized to (64, 64, 3) and (224, 224, 3) for the VGG16 and the ResNet50V2, respectively.…”

Section: A Experiments Setupmentioning

confidence: 99%

See 1 more Smart Citation

Padding Module: Learning the Padding in Deep Neural Networks

2023

View full text Add to dashboard Cite

During the last decades, many studies have been dedicated to improving the performance of neural networks, for example, the network architectures, initialization, and activation. However, investigating the importance and effects of learnable padding methods in deep learning remains relatively open. To mitigate the gap, this paper proposes a novel trainable Padding Module that can be placed in a deep learning model. The Padding Module can optimize itself without requiring or influencing the model's entire loss function. To train itself, the Padding Module constructs a ground truth and a predictor from the inputs by leveraging the underlying structure in the input data for supervision. As a result, the Padding Module can learn automatically to pad pixels to the border of its input images or feature maps. The padding contents are realistic extensions to its input data and simultaneously facilitate the deep learning model's downstream task. Experiments have shown that the proposed Padding Module outperforms the state-of-the-art competitors and the baseline methods. For example, the Padding Module has 1.23% and 0.44% more classification accuracy than the zero padding when tested on the VGG16 and ResNet50.

show abstract

Section: Literature Review and Related Workmentioning

confidence: 99%

Section: A Experiments Setupmentioning

confidence: 99%

Padding Module: Learning the Padding in Deep Neural Networks

2023

View full text Add to dashboard Cite

show abstract

“…to make use of the general observations that it outperforms, or matches, the widely used ReLU activation function as a result of its improved smoothness and differentiability properties [53,54].…”

Section: Training and Adaptive Sampling Of Training Datamentioning

confidence: 99%

Efficient data acquisition and training of collisional-radiative model artificial neural network surrogates through adaptive parameter space sampling

Garland

Maulik

Tang

et al. 2022

Mach. Learn.: Sci. Technol.

View full text Add to dashboard Cite

Effective plasma transport modeling of magnetically confined fusion devices relies on having an accurate understanding of the ion composition and radiative power losses of the plasma. Generally, these quantities can be obtained from solutions of a collisional-radiative (CR) model at each time step within a plasma transport simulation. However, even compact, approximate CR models can be computationally onerous to evaluate, and in-situ evaluation of these models within a larger plasma transport code can lead to a rigid bottleneck. As a way to bypass this bottleneck, we propose deploying artificial neural network surrogates to allow rapid evaluation of the necessary plasma quantities. However, one issue with training an accurate artificial neural network surrogate is the reliance on a sufficiently large and representative training and validation data set, which can be time-consuming to generate. In this work we explore a data-driven active learning and training routine to allow autonomous adaptive sampling of the problem parameter space to ensure a sufficiently large and meaningful set of training data is assembled for the network training. As a result, we can demonstrate approximately order-of-magnitude savings in required training data samples to produce an accurate surrogate.

show abstract

“…The parametric swish (p-swish) [429] is another AF proposed by Mercioni and Holban. It is defined as…”

Section: Parametric Swish (P-swish)mentioning

confidence: 99%

On Transformative Adaptive Activation Functions in Neural Networks for Gene Expression Inference

Kunc

Kléma²

2019

Preprint

View full text Add to dashboard Cite

Motivation: Gene expression profiling was made cheaper by the NIH LINCS program that profiles only ∼1, 000 selected landmark genes and uses them to reconstruct the whole profile. The D-GEX method employs neural networks to infer the whole profile. However, the original D-GEX can be further significantly improved. Results: We have analyzed the D-GEX method and determined that the inference can be improved using a logistic sigmoid activation function instead of the hyperbolic tangent. Moreover, we propose a novel transformative adaptive activation function that improves the gene expression inference even further and which generalizes several existing adaptive activation functions. Our improved neural network achieves average mean absolute error of 0.1340 which is a significant improvement over our reimplementation of the original D-GEX which achieves average mean absolute error 0.1637

show abstract

P-Swish: Activation Function with Learnable Parameters Based on Swish Activation Function in Deep Learning

Cited by 27 publications

References 2 publications

Padding Module: Learning the Padding in Deep Neural Networks

Padding Module: Learning the Padding in Deep Neural Networks

Efficient data acquisition and training of collisional-radiative model artificial neural network surrogates through adaptive parameter space sampling

On Transformative Adaptive Activation Functions in Neural Networks for Gene Expression Inference

Contact Info

Product

Resources

About