Choose your path wisely: gradient descent in a Bregman distance framework

Abstract

We propose an extension of a special form of gradient descent -in the literature known as linearised Bregman iteration -to a larger class of non-convex functions. We replace the classical (squared) two norm metric in the gradient descent setting with a generalised Bregman distance, based on a proper, convex and lower semi-continuous function. The algorithm's global convergence is proven for functions that satisfy the Kurdyka-Lojasiewicz property. Examples illustrate that features of different scale are being introduced throughout the iteration, transitioning from coarse to fine. This coarse-to-fine approach with respect to scale allows to recover solutions of non-convex optimisation problems that are superior to those obtained with conventional gradient descent, or even projected and proximal gradient descent. The effectiveness of the linearised Bregman iteration in combination with early stopping is illustrated for the applications of parallel magnetic resonance imaging, blind deconvolution as well as image classification with neural networks.

Neural Architecture Search via Bregman Iterations

“…Their main characteristic is that they start with a sparse solution and successively add features to minimize a loss function, this way forming an inverse scale space. As we demonstrate in this article this special feature makes them a well-suited algorithm for NAS, additionally being supported with a rich mathematical theory [23,24,25,26,27].…”

A Bregman Learning Framework for Sparse Neural Networks

“…Bregman Iterations Bregman iterations and in particular linearized Bregman iterations have been introduced and thoroughly analyzed for sparse regularization approaches in imaging and compressed sensing (see, e.g., [37,43,44,45,46,48,49,51]). Linearized Bregman iterations for non-convex problems, which appear in machine learning and imaging applications like blind deblurring, have first been analyzed in [15,16,44]. In [16] it was also shown that linearized Bregman iterations for convex problems can be formulated as forward pass of a neural network.…”

“…In [16] it was also shown that linearized Bregman iterations for convex problems can be formulated as forward pass of a neural network. In [15] they were applied for training neural networks with low-rank weight matrices, using nuclear norm regularization. In [27] a split Bregman approach for training sparse neural networks was suggested and a deterministic convergence result along the lines of [15] is provided in [13].…”

“…In [15] they were applied for training neural networks with low-rank weight matrices, using nuclear norm regularization. In [27] a split Bregman approach for training sparse neural networks was suggested and a deterministic convergence result along the lines of [15] is provided in [13]. A recent analysis of a stochastic Bregman gradient method, however using strong regularity assumptions on the involved functions, is provided in [1].…”

“…Hence, the inverse scale space is a proper generalization of the gradient flow and allows for regularizing the path along which the loss is minimized using J (see [15]). For strictly convex loss functions this might seem pointless since they have a unique minimum anyways, however, for merely convex or even non-convex losses the inverse scale space allows to 'select' (local) minima with desirable properties.…”