Local Convergence of the Heavy-Ball Method and iPiano for Non-convex Optimization

Ochs, Peter

doi:10.1007/s10957-018-1272-y

Cited by 30 publications

(20 citation statements)

References 52 publications

(115 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…By incorporating the idea of proximal mapping, the inertial proximal gradient algorithm (iPiano) was proposed in (Ochs et al, 2014), whose convergence in nonconvex case was thoroughly discussed. Locally linear convergence of iPiano and Heavy-ball method was later proved in (Ochs, 2016). In the strongly convex case, the linear convergence was proved for iPiano with fixed β k (Ochs, Brox, and Pock, 2015).…”

Section: Introductionmentioning

confidence: 97%

Non-Ergodic Convergence Analysis of Heavy-Ball Algorithms

Sun

Yin

et al. 2019

AAAI

View full text Add to dashboard Cite

In this paper, we revisit the convergence of the Heavyball method, and present improved convergence complexity results in the convex setting. We provide the first non-ergodic O(1/k) rate result of the Heavy-ball algorithm with constant step size for coercive objective functions. For objective functions satisfying a relaxed strongly convex condition, the linear convergence is established under weaker assumptions on the step size and inertial parameter than made in the existing literature. We extend our results to multi-block version of the algorithm with both the cyclic and stochastic update rules. In addition, our results can also be extended to decentralized optimization, where the ergodic analysis is not applicable.

show abstract

Section: Introductionmentioning

confidence: 97%

Non-Ergodic Convergence Analysis of Heavy-Ball Algorithms

Sun

Yin

et al. 2019

AAAI

View full text Add to dashboard Cite

show abstract

“…In the literature, several accelerations of the Sinkhorn-Knopp algorithm have been proposed, using for instance greedy coordinate descent [13] or screening strategies [14]. In another line of research, the introduction of relaxation variables through heavy ball approaches [15] has recently gained popularity to speed up the convergence of algorithms optimizing convex [16] or non-convex [17,18] problems. In this context, the use of regularized nonlinear accelerations (RNAs) [19][20][21] based on Anderson mixing has led to important numerical improvements, although the global convergence is not guaranteed with such approaches, as is shown further.…”

Section: Accelerations Of the Sinkhorn-knopp Algorithmmentioning

confidence: 99%

Overrelaxed Sinkhorn–Knopp Algorithm for Regularized Optimal Transport

Thibault¹,

Chizat²,

Dossal³

et al. 2021

Algorithms

View full text Add to dashboard Cite

This article describes a set of methods for quickly computing the solution to the regularized optimal transport problem. It generalizes and improves upon the widely used iterative Bregman projections algorithm (or Sinkhorn–Knopp algorithm). We first proposed to rely on regularized nonlinear acceleration schemes. In practice, such approaches lead to fast algorithms, but their global convergence is not ensured. Hence, we next proposed a new algorithm with convergence guarantees. The idea is to overrelax the Bregman projection operators, allowing for faster convergence. We proposed a simple method for establishing global convergence by ensuring the decrease of a Lyapunov function at each step. An adaptive choice of the overrelaxation parameter based on the Lyapunov function was constructed. We also suggested a heuristic to choose a suitable asymptotic overrelaxation parameter, based on a local convergence analysis. Our numerical experiments showed a gain in convergence speed by an order of magnitude in certain regimes.

show abstract

“…Such a local property was then widely applied to study the asymptotic convergence behavior of various gradient-based algorithms in nonconvex optimization [Attouch and Bolte, 2009;Bolte et al, 2014;Zhou et al, 2016;2018b]. The KŁ property has also been applied to study convergence properties of accelerated gradient algorithms [Li et al, 2017;Li and Lin, 2015] and heavy-ball algorithms [Ochs, 2018; in nonconvex optimization. Some other works exploited the KŁ property to study the convergence of second-order algorithms in nonconvex optimization, e.g., [Zhou et al, 2018a].…”

Section: Related Workmentioning

confidence: 99%

Proximal Gradient Algorithm with Momentum and Flexible Parameter Restart for Nonconvex Optimization

Zhou

Wang

et al. 2020

Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence

View full text Add to dashboard Cite

Various types of parameter restart schemes have been proposed for proximal gradient algorithm with momentum to facilitate their convergence in convex optimization. However, under parameter restart, the convergence of proximal gradient algorithm with momentum remains obscure in nonconvex optimization. In this paper, we propose a novel proximal gradient algorithm with momentum and parameter restart for solving nonconvex and nonsmooth problems. Our algorithm is designed to 1) allow for adopting flexible parameter restart schemes that cover many existing ones; 2) have a global sub-linear convergence rate in nonconvex and nonsmooth optimization; and 3) have guaranteed convergence to a critical point and have various types of asymptotic convergence rates depending on the parameterization of local geometry in nonconvex and nonsmooth optimization. Numerical experiments demonstrate the convergence and effectiveness of our proposed algorithm.

show abstract

Local Convergence of the Heavy-Ball Method and iPiano for Non-convex Optimization

Cited by 30 publications

References 52 publications

Non-Ergodic Convergence Analysis of Heavy-Ball Algorithms

Non-Ergodic Convergence Analysis of Heavy-Ball Algorithms

Overrelaxed Sinkhorn–Knopp Algorithm for Regularized Optimal Transport

Proximal Gradient Algorithm with Momentum and Flexible Parameter Restart for Nonconvex Optimization

Contact Info

Product

Resources

About