Sample size selection in optimization methods for machine learning

Abstract

This paper presents a methodology for using varying sample sizes in batch-type optimization methods for large scale machine learning problems. The first part of the paper deals with the delicate issue of dynamic sample selection in the evaluation of the function and gradient. We propose a criterion for increasing the sample size based on variance estimates obtained during the computation of a batch gradient. We establish an O(1/ ) complexity bound on the total cost of a gradient method. The second part of the paper describes a practical Newton method that uses a smaller sample to compute Hessian vector-products than to evaluate the function and the gradient, and that also employs a dynamic sampling technique. The focus of the paper shifts in the third part of the paper to L 1 regularized problems designed to produce sparse solutions. We propose a Newton-like method that consists of two phases: a (minimalistic) gradient projection phase that identifies zero variables, and subspace phase that applies a subsampled Hessian Newton iteration in the free variables. Numerical tests on speech recognition problems illustrate the performance of the algorithms.

Training Deep and Recurrent Networks with Hessian-Free Optimization

“…Finally, Byrd et al (2012) has developed methods to deal with the mini-batch overfitting problem, which are based on heuristics that increase the mini-batch size and also terminate CG early, according to estimates of the variance of the gradient and curvature-matrix vector products. While this is a potentially effective approach (which we don't have experience with), there are several problems with it, in theory.…”

Convergence of Trust-Region Methods Based on Probabilistic Models

“…If {p i } is finite, then this result trivially follows. Otherwise, since δ k → 0, we have that for sufficiently large p i , δ p i < b, with b defined by (5). Since ∇f (x p i ) > , and m p i is fully linear on B(x p i , ∆ p i ), then by the derivations in Theorem 4.2, we have g p i ≥ 2 , and by Lemma 3.2 ρ p i ≥ η 1 .…”

“…The resulting methods may be very simple and enjoy low per-iteration complexity, but the practical performance of these approaches can be very poor. On the other hand, it was noted in [5] that the performance of stochastic gradient methods for large-scale machine learning improves substantially if the sample size is increased during the optimization process. Within direct search, the use of random positive spanning sets has also been recently investigated [1,34] with gains in performance and convergence theory for nonsmooth problems.…”

Global convergence rate analysis of unconstrained optimization methods based on probabilistic models

“…In [4] an adaptive sample size strategy was proposed in the setting where ∇f (x) = N i=1 ∇f i (x), for large values of N . In this case computing ∇f (x) accurately can be prohibitive, hence, instead an estimate ∇f S (x) = i∈S ∇f i (x) is often computed in hopes that it provides a good estimate of the gradient and a descent direction.…”