A variational perspective on accelerated methods in optimization

Abstract

Accelerated gradient methods play a central role in optimization, achieving optimal rates in many settings. Although many generalizations and extensions of Nesterov's original acceleration method have been proposed, it is not yet clear what is the natural scope of the acceleration concept. In this paper, we study accelerated methods from a continuous-time perspective. We show that there is a Lagrangian functional that we call the Bregman Lagrangian, which generates a large class of accelerated methods in continuous time, including (but not limited to) accelerated gradient descent, its nonEuclidean extension, and accelerated higher-order gradient methods. We show that the continuous-time limit of all of these methods corresponds to traveling the same curve in spacetime at different speeds. From this perspective, Nesterov's technique and many of its generalizations can be viewed as a systematic way to go from the continuous-time curves generated by the Bregman Lagrangian to a family of discrete-time accelerated algorithms.O ptimization lies at the core of many fields concerned with data analysis. It provides a mathematical language in which both computational and statistical concepts can be expressed and it delivers practical data analysis algorithms that can scale to the enormous datasets that are increasingly the norm in science and technology. The recent literature on data analysis and optimization has focused on gradient-based optimization methods, given their low per-iteration cost and the relative ease with which they can be deployed on parallel and distributed processing architectures. Establishing that such methods do indeed address the scalability problems inherent in large-scale data analysis raises fundamental questions concerning the convergence rate of gradient-based methods, the extent to which those rates can be increased systematically, and whether there are upper bounds on achievable rates.In the body of theory and practice built up to answer such questions, the phenomenon of acceleration plays a key role. In 1983, Nesterov introduced acceleration in the context of gradient descent for convex functions (1), showing that it achieves an improved convergence rate with respect to gradient descent and moreover that it achieves an optimal convergence rate under an oracle model of optimization complexity (2). The acceleration idea has since been extended to a wide range of other settings, including composite optimization (3-5), stochastic optimization (6, 7), nonconvex optimization (8, 9), and conic programming (10). There have been generalizations to non-Euclidean optimization (11, 12) and higher-order algorithms (13,14), and there have been numerous applications that further extend the reach of the idea (15-18).Despite this compelling evidence of the value of the idea of acceleration, it remains something of a conceptual mystery. Derivations of accelerated methods do not flow from a single underlying principle, but tend to rely on case-specific algebra (19). The basic Nesterov technique is often explai...

Optimal deterministic algorithm generation

“…Another interesting class are algorithms that include an optimization step within them, such as selection of an optimal step, e.g., [44,46]. We can consider a bilevel formulation of the form…”

“…Note that although we have several arg min, they are not correlated, i.e., we do not obtain a n-level optimization problem but rather a bilevel problem with multiple lower level problems. With such a formulation we might be able to prove optimality of one of the three gradient methods described in [44] or discover alternatives. Additionally it would be possible to discover well-known methods such as the nonlinear conjugate gradient method [26], where the line search is described by the arg min operator, or even stochastic methods which often depend on the computation of the maximum expected value of some iterative random variable X (it) .…”

“…See also the recent embedding of discrete algorithms like Nesterov's scheme in continuous implementations [39,43]. It seems possible to consider the optimization of these continuous variants of the algorithms using similar formulations.…”

The Frontier of SGD and its Variants in Machine Learning

“…Recently, Jordan and Al. [19] Show that the Nesterov's momentum has strong relation with the partial differential equations. They show that the trajectory of NAG is corresponding to brachistochrone according to the variational method.…”

Efficient parameterization of large-scale dynamic models based on relative measurements

“…Indeed, the evaluation of the ratio (E[|∇ s J|]/E[|∇ θ J|]) revealed that the objective function is usually 10 times more sensitive with respect to scaling parameters than dynamic parameters ( Figure 2B). This indicates the presence of two separate timescales in the continuous representation of the optimization problem (Wibisono et al, 2016), which suggests that the optimization problem is stiff. As standard optimization methods correspond to explicit solving schemes of the continuous optimization problem, the stiffness could explain the problem encountered for the standard approach.…”