An Optimized Back Propagation Learning Algorithm with Adaptive Learning Rate

Nawi, Nazri Mohd; Hamzah, Faridah; Hamid, Norhamreeza Abdul; Rehman, Muhammad Zubair; Aamir, Mohammad; Ramli, Azizul Azhar

doi:10.18517/ijaseit.7.5.2972

Cited by 12 publications

(6 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, there are also the researchers who use architectures with the number of neurons in the hidden layer equal to the number of neurons in the input layer [4], [5]. Some studies on the prediction of time series data using NNBP architecture with one hidden layer has been widely done namely classification of Australian credit card [6], diabetic detection [7], identification for a single-shaft gas turbine [8], particle swarm optimization [9], measuring the severity of osteoarthritis [10] [9] with architecture 40-20-1, and obtained a coefficient of correlation value of 0.990. The average architecture they used is that the number of neurons in the hidden layer is smaller than the number of neurons in the input layer.…”

Section: Introductionmentioning

confidence: 99%

The Formula Study in Determining the Best Number of Neurons in Neural Network Backpropagation Architecture with Three Hidden Layers

Syaharuddin

Fatmawati

Suprajitno

2022

J. RESTI (Rekayasa Sist. Teknol. Inf.)

View full text Add to dashboard Cite

The researchers conducted data simulation experiments, but they did so unstructured in determining the number of neurons in the hidden layer in the Artificial Neural Network Back-Propagation architecture. The researchers also used a general architecture consisting of one hidden layer. Researchers are still producing minimal research that discusses how to determine the number of neurons when using hidden layers. This article examines the results of experiments by conducting training and testing data using seven recommended formulas including the Hecht-Nelson, Marchandani-Cao, Lawrence & Fredrickson, Berry-Linoff, Boger-Guterman, JingTao-Chew, and Lawrence & Fredrickson modifications. We use rainfall data and temperature data with a 10-day type for the last 10 years (2012-2021) sourced from Lombok International Airport Station, Indonesia. The training and testing data used showed the results that in determining the number of neurons on the hidden-1 screen, it was more appropriate to use the Hecht-Nelson formula and the Lawrence & Fredricson formula which is more suitable for use in the 2nd & 3rd hidden layer. The resulting research was able to provide an accuracy rate of up to 97.79% (temperature data) and 99.94% (rainfall data) with an architecture of 36-73-37-19-1.

show abstract

Section: Introductionmentioning

confidence: 99%

The Formula Study in Determining the Best Number of Neurons in Neural Network Backpropagation Architecture with Three Hidden Layers

Syaharuddin

Fatmawati

Suprajitno

2022

J. RESTI (Rekayasa Sist. Teknol. Inf.)

View full text Add to dashboard Cite

show abstract

“…This is because even if the output unit saturates the corresponding decent gradient takes a small value, even if the output error is large, which will result in no significant progress in the weight adjustment. The second disadvantage of this method is the difficulty in choosing a proper learning rate ƞ to achieve fast learning while maintaining the learning procedure stable [ 35 ]. These problems contribute to the lack of an inability to apply conventional BP to a wide number of applications.…”

Section: The Proposed Approachmentioning

confidence: 99%

A novel adaptive momentum method for medical image classification using convolutional neural network

2022

View full text Add to dashboard Cite

Background AI for medical diagnosis has made a tremendous impact by applying convolutional neural networks (CNNs) to medical image classification and momentum plays an essential role in stochastic gradient optimization algorithms for accelerating or improving training convolutional neural networks. In traditional optimizers in CNNs, the momentum is usually weighted by a constant. However, tuning hyperparameters for momentum can be computationally complex. In this paper, we propose a novel adaptive momentum for fast and stable convergence. Method Applying adaptive momentum rate proposes increasing or decreasing based on every epoch's error changes, and it eliminates the need for momentum hyperparameter optimization. We tested the proposed method with 3 different datasets: REMBRANDT Brain Cancer, NIH Chest X-ray, COVID-19 CT scan. We compared the performance of a novel adaptive momentum optimizer with Stochastic gradient descent (SGD) and other adaptive optimizers such as Adam and RMSprop. Results Proposed method improves SGD performance by reducing classification error from 6.12 to 5.44%, and it achieved the lowest error and highest accuracy compared with other optimizers. To strengthen the outcomes of this study, we investigated the performance comparison for the state-of-the-art CNN architectures with adaptive momentum. The results shows that the proposed method achieved the highest with 95% compared to state-of-the-art CNN architectures while using the same dataset. The proposed method improves convergence performance by reducing classification error and achieves high accuracy compared with other optimizers.

show abstract

“…The Mini-batch gradient descent is a compromise between the two approaches. This gradient descent method performs an update for each mini-batch of training data; therefore, the entire training data set is not used, but only a certain part of it, see (3) [16], [17]. Unlike the stochastic gradient descent, this method is not as sensitive to changes in hyper-parameters, especially the learning rate used during optimization, which leads to a reduction in fluctuations and thus to a more stable convergence [14].…”

Section: Deep Feedforward Networkmentioning

confidence: 99%

“…Back-propagation (BP) is a commonly used algorithm for training the multilayer feed forward NN [16]. There are various learning parameters, such as the learning rate, momentum or activation function which can improve the BP learning algorithm.…”

Section: A Back-propagationmentioning

confidence: 99%

Estimating Harvestable Solar Energy from Atmospheric Pressure Using Deep Learning

Paterova

Prauzek

2021

ELEKTRON ELEKTROTECH

View full text Add to dashboard Cite

This article focuses on applying a deep learning approach to predict daily total solar energy for the next day by a neural network. Predicting future solar irradiance is an important topic in the renewable energy generation field to improve the performance and stability of the system. The forecast is used as a support parameter to control the operation duty-cycle, data collection or communication activities at energy-independent energy harvesting embedded devices. The prediction is based on previous hourly-measured atmospheric pressure values. For prediction, a back-propagation algorithm in combination with deep learning methods is used for multilayer network training. The ability of the proposed system to estimate the daily solar energy is compared to the support vector regression model and to the evolutionary-fuzzy prediction scheme presented in previous research studies. It is concluded that the presented neural network approach gave satisfying predictions in early spring, autumn, and winter. In a particular setting, the proposed solution provides better results than a model using the support vector regression method (e.g., the MAPE value of the proposed algorithm is 0.032 less than the MAPE value of support vector regression method). The time and computational complexity for neural network training is considerable, and therefore it was assumed to train the network on an external computer or a cloud, where only the network parameters have been obtained and transferred to the embedded devices.

show abstract

An Optimized Back Propagation Learning Algorithm with Adaptive Learning Rate

Cited by 12 publications

References 14 publications

The Formula Study in Determining the Best Number of Neurons in Neural Network Backpropagation Architecture with Three Hidden Layers

The Formula Study in Determining the Best Number of Neurons in Neural Network Backpropagation Architecture with Three Hidden Layers

A novel adaptive momentum method for medical image classification using convolutional neural network

Estimating Harvestable Solar Energy from Atmospheric Pressure Using Deep Learning

Contact Info

Product

Resources

About