Survey of Optimization Algorithms in Modern Neural Networks

Abstract

The main goal of machine learning is the creation of self-learning algorithms in many areas of human activity. It allows a replacement of a person with artificial intelligence in seeking to expand production. The theory of artificial neural networks, which have already replaced humans in many problems, remains the most well-utilized branch of machine learning. Thus, one must select appropriate neural network architectures, data processing, and advanced applied mathematics tools. A common challenge for these networks is achieving the highest accuracy in a short time. This problem is solved by modifying networks and improving data pre-processing, where accuracy increases along with training time. Bt using optimization methods, one can improve the accuracy without increasing the time. In this review, we consider all existing optimization algorithms that meet in neural networks. We present modifications of optimization algorithms of the first, second, and information-geometric order, which are related to information geometry for Fisher–Rao and Bregman metrics. These optimizers have significantly influenced the development of neural networks through geometric and probabilistic tools. We present applications of all the given optimization algorithms, considering the types of neural networks. After that, we show ways to develop optimization algorithms in further research using modern neural networks. Fractional order, bilevel, and gradient-free optimizers can replace classical gradient-based optimizers. Such approaches are induced in graph, spiking, complex-valued, quantum, and wavelet neural networks. Besides pattern recognition, time series prediction, and object detection, there are many other applications in machine learning: quantum computations, partial differential, and integrodifferential equations, and stochastic processes.

The Classification of Aflatoxin Contamination Level in Cocoa Beans using Fluorescence Imaging and Deep learning

“…Furthermore, this improvement enhances the accuracy of object categorization and pattern recognition. [78]. The performance graphs for training and validation showed a rapid progression in the initial epochs, followed by convergence in the subsequent epochs, resulting in an accuracy value that approached an impressive 96.42%.…”

A simple theory for training response of deep neural networks

“…As a training algorithm, we use an optimizer, known as SGD [10]. Ideally, this can be regarded as a potential dynamics for each parameter, w,…”

“…We can update the network parameters along the gradient of the loss step by step. The training dynamics are generated through the update steps and are often stochastic depending on the update procedures [10].…”

Classification of plant leaf diseases using deep neural networks in color and grayscale images

“…Moreover, we also analyzed the performance of various DNNs for the classification of diseases in leaf images using different solver algorithms. The application of powerful solver algorithms is an efficient way to improve the classification accuracy of DNNs (Abdulkadirov et al, 2023;Iqbal et al, 2021).…”