On the Insufficiency of Existing Momentum Schemes for Stochastic Optimization

Abstract

Momentum based stochastic gradient methods such as heavy ball (HB) and Nesterov's accelerated gradient descent (NAG) method are widely used in practice for training deep networks and other supervised learning models, as they often provide significant improvements over stochastic gradient descent (SGD). Rigorously speaking, "fast gradient" methods have provable improvements over gradient descent only for the deterministic case, where the gradients are exact. In the stochastic case, the popular explanations for their wide applicability is that when these fast gradient methods are applied in the stochastic case, they partially mimic their exact gradient counterparts, resulting in some practical gain. This work provides a counterpoint to this belief by proving that there exist simple problem instances where these methods cannot outperform SGD despite the best setting of its parameters. These negative problem instances are, in an informal sense, generic; they do not look like carefully constructed pathological instances. These results suggest (along with empirical evidence) that HB or NAG's practical performance gains are a by-product of mini-batching.Furthermore, this work provides a viable (and provable) alternative, which, on the same set of problem instances, significantly improves over HB, NAG, and SGD's performance. This algorithm, referred to as Accelerated Stochastic Gradient Descent (ASGD), is a simple to implement stochastic algorithm, based on a relatively less popular variant of Nesterov's Acceleration. Extensive empirical results in this paper show that ASGD has performance gains over HB, NAG, and SGD. The code implementing the ASGD Algorithm can be found here 1 .

Dynamics of Stochastic Momentum Methods on Large-scale, Quadratic Models

“…As we show in this work, both theoretically and empirically, (small batch size) SGD with heavyball momentum (sHB) for any fixed momentum parameter does not provide any acceleration over plain SGD on large-scale least square problems. We conclude under an identification of the parameters that f (x shb k ) = f (x sgd k ) for all k up to errors that vanish as n grows large (upper bounds of this nature have been observed before: see Kidambi et al [2018], Sebbouh et al [2020], Zhang et al [2019]). Thus while sHB may provide a speed-up over SGD, it is only due to an effective increase in the learning rate, and this speed-up could be matched by appropriately adjusting the learning rate of SGD.…”

Learning-Rate Annealing Methods for Deep Neural Networks

“…The variance of stochastic gradients is detrimental to SGD, motivating variance reduction techniques [22][23][24][25][26][27][28] that aim to reduce the variance incurred due to their stochastic process of estimation, and improve the convergence rate mainly for convex optimization, while some are extended to non-convex problems [29][30][31]. One of the most practical algorithms for better convergence rates includes momentum [32], modified momentum for accelerated gradient [33], and stochastic estimation of accelerated gradient descent [34]. These algorithms are more focused on the efficiency in convergence than the generalization of models for accuracy.…”

Stochastic Momentum Method With Double Acceleration for Regularized Empirical Risk Minimization

“…The inclusion of Nesterov's momentum in deterministic step provides a potential of stabilization for stochastic variance-reduced algorithms. However, the acceleration gained with Nesterov's momentum in stochastic steps is mainly the by-product of mini-batching [28]. This double acceleration in DASVRDA [24] is possibly limited to the setting of mini-batching, which prevents us to develop a stochastic algorithm in the non-mini-batch setting.…”

“…The popularity and wide applicability of momentum methods in stochastic setting are that they mimic their deterministic gradient counterparts, which result in some practical gains. However, these gains in stochastic cases are particularly the by-products of mini-batching [28]. For instance, Nesterov's momentum is applied in the pure stochastic setting with a batch size of 1 in Acc-Prox-SVRG [18], wherein no acceleration can be guaranteed in theory.…”