A Large-Scale Study of Activation Functions in Modern Deep Neural Network Architectures for Efficient Convergence

Abstract

Activation functions play an important role in the convergence of learning algorithms based on neural networks. Theyprovide neural networks with nonlinear ability and the possibility to fit in any complex data. However, no deep study exists in theliterature on the comportment of activation functions in modern architecture. Therefore, in this research, we compare the 18 most used activation functions on multiple datasets (CIFAR-10, CIFAR-100, CALTECH-256) using 4 different models (EfficientNet,ResNet, a variation of ResNet using the bag of tricks, and MobileNet V3). Furthermore, we explore the shape of the losslandscape of those different architectures with various activation functions. Lastly, based on the result of our experimentation,we introduce a new locally quadratic activation function namely Hytana alongside one variation Parametric Hytana whichoutperforms common activation functions and address the dying ReLU problem.

Construction of electric emergency materials storage system based on BP neural network

“…Other researchers believe that the purpose of emergency inventory is to keep it within a reasonable range, not to cause inventory shortage due to supply bottlenecks, provided that emergency care is guaranteed [3][4]. Other researchers are using mobile digital terminals and QR code scanning technology to overcome the shortcomings of traditional manual and non-mobile warehouse management [5][6]. The power emergency material storage system is a comprehensive multi-level control system, which requires the cooperation of multiple departments.…”