On the Use of Stochastic Hessian Information in Optimization Methods for Machine Learning

Abstract

This paper describes how to incorporate sampled curvature information in a Newton-CG method and in a limited memory quasi-Newton method for statistical learning. The motivation for this work stems from supervised machine learning applications involving a very large number of training points. We follow a batch approach, also known in the stochastic optimization literature as a sample average approximation (SAA) approach. Curvature information is incorporated in two sub-sampled Hessian algorithms, one based on a matrix-free inexact Newton iteration and one on a preconditioned limited memory BFGS iteration. A crucial feature of our technique is that Hessian-vector multiplications are carried out with a significantly smaller sample size than is used for the function and gradient. The efficiency of the proposed methods is illustrated using a machine learning application involving speech recognition.

Training Deep and Recurrent Networks with Hessian-Free Optimization

“…Martens (2010) recommended using this technique (as does Byrd et al (2011)), and in our experience it can often improve optimization speed if used carefully and in the right contexts. But despite this, there are several good theoretical reasons why it might be better, at least in some situations, to use the same mini-batch to compute gradient and curvature matrix-vector products.…”

“…And in practice, we have found that such an approach does not seem to work very well, and results in CG itself diverging in some cases. The solution advocated by Martens (2010) and independently by Byrd et al (2011) is to fix the mini-batch used to define B for the entire run of CG. Mini-batches and the practical issues which arise when using them will be discussed in more depth in Section 12.…”

“…This is because when we compute estimates of the curvature by averaging, we must divide by the number of cases in the mini-batch, and since this number will be smaller for the curvature estimate than for the gradient, the gradient contributions from these cases will be smaller in proportion to the corresponding curvature terms. Byrd et al (2011) showed that if the eigenvalues of the curvature matrices (estimated using any method) are uniformly bounded from below in the sense that there exists µ > 0 s.t.B − µI is PSD for all possibleB's which we might produce, then assuming the use of a basic line-search and other mild conditions, a truncated Newton algorithm like HF which estimates the gradient on the full training set will converge in the sense that the gradients will go to zero. But this result makes no use of the particular form of B and says nothing about the rate of convergence, or how small the updates will have to be.…”

Sample size selection in optimization methods for machine learning

“…Sample selection also plays a crucial role in the incorporation of curvature information in a Hessian-free Newton method for machine learning [7,21]. In this so-called subsampled Hessian Newton-CG method, the step computation is obtained by applying the conjugate gradient (CG) method, which only requires Hessian-vector products and not the Hessian matrix itself.…”

“…This paper can be seen as a continuation of the work in [7], which dealt with Hessian sampling techniques for a Newton-CG method. Here we take this work further, first by considering also sample selection techniques for the evaluation of function and gradients, and second, by studying the extension of Hessian sampling techniques to nonsmooth L 1 regularized problems.…”