iPiano: Inertial Proximal Algorithm for Nonconvex Optimization

Abstract

Abstract. In this paper we study an algorithm for solving a minimization problem composed of a differentiable (possibly non-convex) and a convex (possibly non-differentiable) function. The algorithm iPiano combines forward-backward splitting with an inertial force. It can be seen as a non-smooth split version of the Heavy-ball method from Polyak. A rigorous analysis of the algorithm for the proposed class of problems yields global convergence of the function values and the arguments. This makes the algorithm robust for usage on non-convex problems. The convergence result is obtained based on the Kurdyka-Lojasiewicz inequality. This is a very weak restriction, which was used to prove convergence for several other gradient methods. First, an abstract convergence theorem for a generic algorithm is proved, and, then iPiano is shown to satisfy the requirements of this theorem. Furthermore, a convergence rate is established for the general problem class. We demonstrate iPiano on computer vision problems: image denoising with learned priors and diffusion based image compression.Key words. non-convex optimization, Heavy-ball method, inertial forward-backward splitting, Kurdyka-Lojasiewicz inequality, proof of convergence 1. Introduction. The gradient method is certainly one of the most fundamental but also one of the most simple algorithms to solve smooth convex optimization problems. In the last decades, the gradient method has been modified in many ways. One of those improvements is to consider so-called multi-step schemes [38,35]. It has been shown that such schemes significantly boost the performance of the plain gradient method. Triggered by practical problems in signal processing, image processing and machine learning, there has been an increased interest in so-called composite objective functions, where the objective function is given by the sum of a smooth function and a non-smooth function with an easy to compute proximal map. This initiated the development of the so-called proximal gradient or forward-backward method [28], that combines explicit (forward) gradient steps w.r.t. the smooth part with proximal (backward) steps w.r.t. the non-smooth part.In this paper, we combine the concepts of multi-step schemes and the proximal gradient method to efficiently solve a certain class of non-convex, non-smooth optimization problems. Although, the transfer of knowledge from convex optimization to non-convex problems is very challenging, it aspires to find efficient algorithms for certain non-convex problems. Therefore, we consider the subclass of non-convex problems

A Logarithmic Image Prior for Blind Deconvolution

“…They also argue that a variational Bayes approach should be preferred because it is more robust when minimizing a highly non-convex function. Their conclusions are however in contrast with several MAP approaches that have demonstrated effective results in various non-convex problems (Strekalovskiy and Cremers 2014;Möllenhoff et al 2014a;Ochs et al 2014;Möllenhoff et al 2014b). The conclusions given in (Wipf and Zhang 2013) suggest that minimizing a cost functional as in (2) is not limited per se, as long as one finds a minimization strategy that carefully avoids its local minima.…”

“…However, we use the logarithm of TV at each pixel, which yields a simple energy term while providing a good approximation to the number of nonzero gradients. Indeed, this prior has already demonstrated promising results in blind deconvolution (Babacan et al 2012;Wipf and Zhang 2013) and denoising (Ochs et al 2014). The MAP estimators are usually discredited to be theoretically less convenient than the conditional mean (CM) estimator (Burger and Lucka 2014).…”

Techniques for Gradient-Based Bilevel Optimization with Non-smooth Lower Level Problems

“…We focus on gradient based methods, such as gradient descent, L-BFGS [25], non-linear conjugate gradient [19,1], Heavy-ball method [40], iPiano [29], and others, for solving the bilevel optimization problem. In particular, this paper focuses on the estimation of descent directions.…”

“…In order to solve the optimization problem in (3), we can apply iPiano [29], a gradient-based algorithm that can handle the non-smooth part (ϑ). The extension of iPiano in [28,Chapter 6] allows for a proxbounded (non-convex, non-smooth) function (ϑ).…”

Optimised photometric stereo via non-convex variational minimisation

“…Our numerical strategy uses recent findings of Ochs et al [2]. They proposed a novel method to handle such tasks, called iPiano.…”