Newton Methods for Convolutional Neural Networks

Abstract

Deep learning involves a difficult non-convex optimization problem, which is often solved by stochastic gradient (SG) methods. While SG is usually effective, it may not be robust in some situations. Recently, Newton methods have been investigated as an alternative optimization technique, but nearly all existing studies consider only fully-connected feedforward neural networks. They do not investigate other types of networks such as Convolutional Neural Networks (CNN), which are more commonly used in deep-learning applications. One reason is that Newton methods for CNN involve complicated operations, and so far no works have conducted a thorough investigation. In this work, we give details of all building blocks including function, gradient, and Jacobian evaluation, and Gauss-Newton matrix-vector products. These basic components are very important because with them further developments of Newton methods for CNN become possible. We show that an efficient MATLAB implementation can be done in just several hundred lines of code and demonstrate that the Newton method gives competitive test accuracy.

A literature survey of matrix methods for data science

“…These typically require the use of direct or iterative solvers to compute the step towards optimality, for example, solving with the Hessian matrix in a Newton method. As a result the applicability of Newton-type schemes for the computation of W (j) and b (j) has received more attention with a strong focus on exploiting the structure of the Hessian matrix [35,51,65,239,283,287]. The computational complexity of neural networks is challenging on many levels.…”

Newton methods based convolution neural networks using parallel processing

“…Hessian-vector product have use case in training deep neural network, also known as Hessian-free optimization. Recently, Newton methods have been investigated as an alternative optimization technique, but nearly all existing studies consider only fully-connected feed-forward neural networks [19]. Newton methods for CNN involve complicated operations due to this limited researchers have conducted a thorough investigation.…”

“…Newton methods for CNN involve complicated operations due to this limited researchers have conducted a thorough investigation. One of the major work in this direction is introduction of Newton methods in CNN for the optimization [19]. There are many reasons to work further in this direction.…”

“…At first, it is generally more substantial to apply weight updates derived from second-order methods in terms of optimization aspect. Meanwhile, it takes roughly the same time to obtain curvature-vector products [8] and computing the gradient which make it possible to use second-order method on large scale model [6].This second order method have shown better results to traditional stochastic gradient (SG) methods [19]. But it needs to find Hessian matrix.…”

“…But it needs to find Hessian matrix. Finding Hessian matrix becomes very complex as we might resort to finite differences method while dealing with the Image data [19]. Another issue is that if we have n dimensions, the Hessian will need O(n 2 ) space and computational complexity for n × n matrix.…”