2020
DOI: 10.1145/3368271
|View full text |Cite
|
Sign up to set email alerts
|

Newton Methods for Convolutional Neural Networks

Abstract: Deep learning involves a difficult non-convex optimization problem, which is often solved by stochastic gradient (SG) methods. While SG is usually effective, it may not be robust in some situations. Recently, Newton methods have been investigated as an alternative optimization technique, but nearly all existing studies consider only fully-connected feedforward neural networks. They do not investigate other types of networks such as Convolutional Neural Networks (CNN), which are more commonly used in deep-learn… Show more

Help me understand this report

Search citation statements

Order By: Relevance

Paper Sections

Select...
4
1

Citation Types

0
16
0

Year Published

2020
2020
2024
2024

Publication Types

Select...
3
2
1

Relationship

0
6

Authors

Journals

citations
Cited by 9 publications
(16 citation statements)
references
References 24 publications
(26 reference statements)
0
16
0
Order By: Relevance
“…These typically require the use of direct or iterative solvers to compute the step towards optimality, for example, solving with the Hessian matrix in a Newton method. As a result the applicability of Newton-type schemes for the computation of W (j) and b (j) has received more attention with a strong focus on exploiting the structure of the Hessian matrix [35,51,65,239,283,287]. The computational complexity of neural networks is challenging on many levels.…”
Section: Numerical Linear Algebra In Deep Learningmentioning
confidence: 99%
“…These typically require the use of direct or iterative solvers to compute the step towards optimality, for example, solving with the Hessian matrix in a Newton method. As a result the applicability of Newton-type schemes for the computation of W (j) and b (j) has received more attention with a strong focus on exploiting the structure of the Hessian matrix [35,51,65,239,283,287]. The computational complexity of neural networks is challenging on many levels.…”
Section: Numerical Linear Algebra In Deep Learningmentioning
confidence: 99%
“…Hessian-vector product have use case in training deep neural network, also known as Hessian-free optimization. Recently, Newton methods have been investigated as an alternative optimization technique, but nearly all existing studies consider only fully-connected feed-forward neural networks [19]. Newton methods for CNN involve complicated operations due to this limited researchers have conducted a thorough investigation.…”
Section: Introductionmentioning
confidence: 99%
“…Newton methods for CNN involve complicated operations due to this limited researchers have conducted a thorough investigation. One of the major work in this direction is introduction of Newton methods in CNN for the optimization [19]. There are many reasons to work further in this direction.…”
Section: Introductionmentioning
confidence: 99%
See 2 more Smart Citations