Design Space Exploration of Neural Network Activation Function Circuits

Yang, Tao; Wei, Yadong; Tu, Zhijun; Zeng, Haolun; Kinsy, Michel A.; Zheng, Nanning; Ren, Pengju

doi:10.1109/tcad.2018.2871198

Cited by 51 publications

(18 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…After the characterization and modeling of a single column cell, we have simulated the multiply-accumulate operation in a single tile. The design space of a single tile has been considered according to [24]. The BL, WL and SL resistance has been calculated according to [25].…”

Section: Experiments and Resultsmentioning

confidence: 99%

Ferroelectric FET analog synapse for acceleration of deep neural network training

Jerry

Chen

Zhang

et al. 2017

2017 IEEE International Electron Devices Meeting (IEDM)

449

393

View full text Add to dashboard Cite

In this paper we propose and evaluate the performance of a 3D-embedded neuromorphic computation block based on indium gallium zinc oxide ( -IGZO) based nanosheet transistor and bi-layer resistive memory devices. We have fabricated bi-layer resistive random-access memory (RRAM) devices with Ta2O5 and Al2O3 layers. The device has been characterized and modeled. The compact models of RRAM and -IGZO based embedded nanosheet structures have been used to evaluate the system level performance of 8 vertically stacked -IGZO based nanosheet layers with RRAM for neuromorphic applications. The model considers the design space with uniform bit line (BL), select line (SL) and word line (WL) resistance. Finally, we have simulated the weighted sum operation with our proposed 8-layer stacked nanosheet based embedded memory and evaluated the performance for VGG-16 convolutional neural network (CNN) for Fashion-MNIST and CIFAR-10 data recognition, which yielded 92% and 75% accuracy respectively with drop out layers amid device variation.

show abstract

Section: Experiments and Resultsmentioning

confidence: 99%

Ferroelectric FET analog synapse for acceleration of deep neural network training

Jerry

Chen

Zhang

et al. 2017

2017 IEEE International Electron Devices Meeting (IEDM)

449

393

View full text Add to dashboard Cite

show abstract

“…When the equation is examined, when 𝑥 > 0 value, ReLU behaves like an activation function. SELU can perform the learning process robustly due to its selfnormalization feature, analytically zero mean and unit variance convergence, and allows it to be trained over many layers [22,23].…”

Section: Selu Activation Functionmentioning

confidence: 99%

An overview of the activation functions used in deep learning algorithms

Kılıçarslan

Adem

Çelik

2021

Journal of New Results in Science

View full text Add to dashboard Cite

In deep learning models, the inputs to the network are processed using activation functions to generate the output corresponding to these inputs. Deep learning models are of particular importance in analyzing big data with numerous parameters and forecasting and are useful for image processing, natural language processing, object recognition, and financial forecasting. Also, in deep learning algorithms, activation functions have been developed by taking into account features such as performing the learning process in a healthy way, preventing excessive learning, increasing the accuracy performance, and reducing the computational cost. In this study, we present an overview of common and current activation functions used in deep learning algorithms. In the study, fixed and trainable activation functions are introduced. As fixed activation functions, sigmoid, hyperbolic tangent, ReLU, softplus and swish, and as trainable activation functions, LReLU, ELU, SELU and RSigELU are introduced.

show abstract

“…x depicts the input vector and b shows the bias vector. There are multiple activation functions that are being used in literature, such as Rectified Linear Unit (ReLU), Leaky ReLU, Parametric ReLU (PReLU) [44], Exponential Linear Unit (ELU) and Scaled ELU (SELU) [45]. These are explained below.…”

Section: ) Convolution Layermentioning

confidence: 99%

“…The only difference is that it uses two parameters in order to reduce vanishing gradient issue, which improves the learning speed of the model. It is given in Equation 5, [45].…”

Section: ) Convolution Layermentioning

confidence: 99%

Electricity Consumption Forecasting Using Gated-FCN With Ensemble Strategy

Naz

Javaid²,

Asif

et al. 2021

IEEE Access

View full text Add to dashboard Cite

Accurate electricity consumption forecasting in the power grids ensures efficient generation and distribution of electricity. Keeping this in mind, the paper introduces a novel deep learning model, termed Gated-FCN, for short-term load forecasting. The key idea is to introduce an automated feature selection and deep learning model for forecasting. The model includes an eight-layered Fully Convolutional Network (FCN-8) in which the hand-crafted feature selection that requires expert domain knowledge is avoided. Furthermore, the model also reduces noise as it learns internal dependencies and the correlation of the time series. Enhanced Bidirectional Gated Recurrent Unit (EBiGRU) is used in combination with FCN-8 to learn long-term temporal dependencies of the time series. Moreover, a weighted averaging mechanism of multiple snapshot models is adopted in the proposed model to assign optimized weights to BiGRU. At the end of FCN-8 and BiGRU, a fully connected dense layer is used that gives final prediction results. Gated-FCN is an end-to-end forecasting model that does not require any other model for enhancing its forecasting efficiency. Different activation functions are initially analyzed to determine how the proposed model learns complex patterns from the time series data. Later, the activation function having the best accuracy is used for forecasting. The proposed model extracts both spatial and temporal features from the data. Furthermore, this paper also provides predictive and exploratory data analyses to assist policymakers in making optimal decisions regarding power production and dispatch. In order to demonstrate the applicability of the proposed technique, the simulations are performed using nine years' load consumption data taken from Independent System Operators New England (ISO-NE). The comparison with five state-of-the-art techniques is also provided to prove the fact that Gated-FCN gives the best forecasting accuracy as compared to other benchmark techniques in terms of two performance metrics: Mean Absolute Percentage Error (MAPE) and Root Mean Square Error (RMSE).

show abstract

Design Space Exploration of Neural Network Activation Function Circuits

Cited by 51 publications

References 14 publications

Ferroelectric FET analog synapse for acceleration of deep neural network training

Ferroelectric FET analog synapse for acceleration of deep neural network training

An overview of the activation functions used in deep learning algorithms

Electricity Consumption Forecasting Using Gated-FCN With Ensemble Strategy

Contact Info

Product

Resources

About