The article presents new methods for searching critical points of a function of several variables, including saddle points. Such problems are found in various fields of theoretical and practical science, for example, saddle-point construction lens design, machine and deep learning, problems of convex optimization and nonlinear programming (necessary and sufficient conditions for the solution are formulated using saddle points of the Lagrange function and proved in the Kuhn-Tucker theorem. When training neural networks, it is necessary to repeat the training process on large clusters and check the network's trainability at different loss functions and different network depth. Which means that thousands of new calculations are run, where each time the loss function is optimized on large amounts of data. So any acceleration in the process of finding critical points is a major advantage and saves computing resources. Many modern methods of searching saddle points are based on calculating the Hessian matrix, inverting this matrix, the scalar product of the gradient vector and the current vector, finding the full Lagrangian, etc. However, all these operations are computationally “expensive” and it would make sense to bypass such complex calculations. The idea of modifying the standard gradient methods used in the article is to apply fixed-point search schemes for nonlinear discrete dynamical systems for gradient descent problems. It is assumed that these fixed points correspond to unstable equilibrium positions, and there are large units among the multipliers of each equilibrium position. The averaged predictive control methods are used. Results of numerical modeling and visualization are presented in the form of two tables, which indicate basins of attraction for each critical point in each scheme, and statistical data by the convergence rates.
The article investigated a modification of stochastic gradient descent (SGD), based on the previously developed stabilization theory of discrete dynamical system cycles. Relation between stabilization of cycles in discrete dynamical systems and finding extremum points allowed us to apply new control methods to accelerate gradient descent when approaching local minima. Gradient descent is often used in training deep neural networks on a par with other iterative methods. Two gradient SGD and Adam were experimented, and we conducted comparative experiments. All experiments were conducted during solving a practical problem of teeth recognition on 2-D panoramic images. Network training showed that the new method outperforms the SGD in its capabilities and as for parameters chosen it approaches the capabilities of Adam, which is a “state of the art” method. Thus, practical utility of using control theory in the training of deep neural networks and possibility of expanding its applicability in the process of creating new algorithms in this important field are shown.
The paper shows the importance of reducing the neural networks’ training time at present stage and the role of new optimization methods in neural networks’ training. The paper researches a modification of stochastic gradient descent, which is based on the idea of gradient descent representation as a discrete dynamical system. The connection between the extreme points, to which the gradient descent iterations tend, and the stationary points of the corresponding discrete dynamicalsystem is a consequence of this representation. The further applied stabilizing scheme with predictive control, for which a theoretical apparatus was developed bymeans of geometric complex analysis together with solving optimization tasks in a set of polynomials with real coefficients, was able to train a multilevel perceptron for recognizing handwritten numbers many times faster. The new algorithm software implementation used the PyTorch library, created for researches in the field of neural networks. All experiments were run on NVidia graphical processing unit to check the processing unit’s resource consumption. The numerical experiments did not reveal any deviation in training time. There was a slight increase in the used video memory, which was expected as the new algorithm retains one additional copy of perceptron internal parameters. The importance of this result is associated with the growth in the useof deep neural network technology, which has grown three hundred thousand times from 2012 till 2018, and the associated resource consumption. This situation forces the industry to consider training optimization issues as well as their accuracy. Therefore, any training process acceleration that reduces the time or resources of the clusters is a desirableand important result, which was achieved in this article. The results obtained discover a new area of theoretical and practical research, since the stabilization usedis only one ofthe methods of stabilization and search for cycles in control theory. Such good practical results confirm the need to add the lagging control and the additional experiments with both predictive and lagging control elements
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
hi@scite.ai
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.