Choose Your Path Wisely: Gradient Descent in a Bregman Distance Framework

Abstract

We propose an extension of a special form of gradient descent-in the literature known as linearized Bregman iteration-to a larger class of nonconvex functions. We replace the classical (squared) two norm metric in the gradient descent setting with a generalized Bregman distance, based on a proper, convex, and lower semicontinuous function. The algorithm's global convergence is proven for functions that satisfy the Kurdyka-Lojasiewicz property. Examples illustrate that features of different scale are being introduced throughout the iteration, transitioning from coarse to fine. This coarse-to-fine approach with respect to scale allows us to recover solutions of nonconvex optimization problems that are superior to those obtained with conventional gradient descent, or even projected and proximal gradient descent. The effectiveness of the linearized Bregman iteration in combination with early stopping is illustrated for the applications of parallel magnetic resonance imaging, blind deconvolution, as well as image classification with neural networks.

Revisiting Linearized Bregman Iterations under Lipschitz-like Convexity Condition

“…Although a group of numerical tests were reported in [4] to demonstrate that the LBreI in nonconvex optimization still leads to superior performance than that of the regularized problems (1.2), the current theory is far from satisfying. On one hand, as partially mentioned in section 4.2 in [4], the required gradient Lipschitz continuity assumption precludes the application of LBreI to many practical problems such as blind deconvolution problems, Poisson inverse problems, and quadratic inverse problems. On the other hand, it is unclear whether similar results to Theorem 1.1 can be established for general convex energy function E. These two aspects contribute the main motivation of this study.…”

“…Very recently, some nonconvex extension of the LBreI, allowing E(x) = E(Ax, b) to be in a general form which has a Lipschitz continuous gradient, was made in [4]. Although a group of numerical tests were reported in [4] to demonstrate that the LBreI in nonconvex optimization still leads to superior performance than that of the regularized problems (1.2), the current theory is far from satisfying.…”

“…Very recently, some nonconvex extension of the LBreI, allowing E(x) = E(Ax, b) to be in a general form which has a Lipschitz continuous gradient, was made in [4]. Although a group of numerical tests were reported in [4] to demonstrate that the LBreI in nonconvex optimization still leads to superior performance than that of the regularized problems (1.2), the current theory is far from satisfying. On one hand, as partially mentioned in section 4.2 in [4], the required gradient Lipschitz continuity assumption precludes the application of LBreI to many practical problems such as blind deconvolution problems, Poisson inverse problems, and quadratic inverse problems.…”

“…For nonconvex convergence analysis, we require the following assumptions. The first one was used in [4] and the second one in [7]. Again, if R itself is real-valued convex, then the firs assumption below holds trivially.…”

“…This part is about the convergence analysis of LBreIF for minimizing a nonconvex objective function E(x). We start with a sufficient descent lemma, which generalizes the central result-Lemma 4.2 in [4].…”