Activation functions facilitate deep neural networks by introducing non-linearity to the learning process. The non-linearity feature gives the neural network the ability to learn complex patterns. Recently, the most widely used activation function is the Rectified Linear Unit (ReLU). Though, other various existing activation including hand-designed alternatives to ReLU have been proposed. However, none has succeeded in replacing ReLU due to their existing inconsistencies. In this work, activation function called ReLU-Memristor-like Activation Function (RMAF) is proposed to leverage benefits of negative values in neural networks. RMAF introduces a constant parameter (α) and a threshold parameter (p) making the function smooth, non-monotonous, and introduces non-linearity in the network. Our experiments show that, the RMAF works better than ReLU and other activation functions on deeper models and across number of challenging datasets. Firstly, experiments are performed by training and classifying on multi-layer perceptron (MLP) over benchmark data such as the Wisconsin breast cancer, MNIST, Iris and Car evaluation. RMAF achieves high performance of 98.74%, 99.67%, 98.81% and 99.42% respectively, compared to Sigmoid, Tanh and ReLU. Secondly, experiments were performed on convolution neural network (ResNet) over MNIST, CIFAR-10 and CIFAR-100 data and observed the proposed activation function achieves higher performance accuracy of 99.73%, 98.77% and 79.82% respectively than Tanh, ReLU and Swish. Additionally, we experimented our work on deep networks i.e. squeeze network (SqueezeNet), Dense connected neural network (DenseNet121) and ImageNet dataset, which RMAF produced the best performance. We note that, the RMAF converges faster than the other functions and can replace ReLU in any neural network due to the efficiency, scalability and its similarity to both ReLU and Swish.

Capsule network's hierarchical framework (CapsNets) consists of an initial standard convolution layer that uses an activation function at its core. The rectified linear unit (ReLU) activation function is widely used in CapsNet and brain tumor classification tasks among several existing activation functions. However, ReLU has some shortcomings where the zero derivatives of the function cause failure of neuron activation. Furthermore, the performance accuracy obtained by the ReLU with CapsNet on brain tumor classification is unsatisfactory. We proposed a new activation function called parametric scaled hyperbolic tangent (PSTanh), which enhances the conventional hyperbolic tangent by avoiding vanishing gradient, provides a small gradient with the introduction of λ and β parameters, and enables faster optimization. Eight standard activation functions (i.e., tanh, Memrister-Like Activation Function (ReLU), Leaky-ReLU, PReLU, ELU, SELU, Swish, ReLU-Memrister-Like Activation Function (RMAF), and the proposed activation) are analyzed and compared in brain tumor classification tasks. Furthermore, extensive experiments are conducted using MNIST, fashion-MNIST, CIFAR-10, CIFAR-100, and ImageNet datasets trained on CapsNets models and deep CNN models (i.e., AlexNet, SqueezeNet, ResNet50, and DenseNet121). The brain tumor's experimental results based on CapsNet and CNN model show that the proposed PSTanh activation achieves better performance than other functions.

This paper proposes a new dual horizontal squash capsule network (DHS-CapsNet) to classify the lung and colon cancers on histopathological images. DHS-CapsNet is made up of encoder feature fusion (EFF) and a novel horizontal squash (HSquash) function. The EFF aggregates the extracted feature from the 2-lane convolutional layers, which provides rich information for better accuracy. HSquash is proposed as a squash function to ensure that vectors are effectively squashed and produces sparsity for a high discriminative capsule to extract important information from images with varied backgrounds. To present the effectiveness of DHS-CapsNet empirically, we applied this method on histopathological images (LC25000 dataset). We achieved better results of 99.23% compared to traditional CapsNet (85.55%). The DHS-CapsNet provides the top-1 classification error of 0.77% compared to 14.45% of the traditional CapsNet. Our results illustrate that our method improves CapsNet and can be adopted as a computer-aided diagnostic method to support doctors in lung and colon cancer diagnostics.

No abstract

No abstract