On Quasi-Newton Forward-Backward Splitting: Proximal Calculus and Convergence

Abstract

We introduce a framework for quasi-Newton forward-backward splitting algorithms (proximal quasi-Newton methods) with a metric induced by diagonal ± rank-r symmetric positive definite matrices. This special type of metric allows for a highly efficient evaluation of the proximal mapping. The key to this efficiency is a general proximal calculus in the new metric. By using duality, formulas are derived that relate the proximal mapping in a rank-r modified metric to the original metric. We also describe efficient implementations of the proximity calculation for a large class of functions; the implementations exploit the piece-wise linear nature of the dual problem. Then, we apply these results to acceleration of composite convex minimization problems, which leads to elegant quasi-Newton methods for which we prove convergence. The algorithm is tested on several numerical examples and compared to a comprehensive list of alternatives in the literature. Our quasi-Newton splitting algorithm with the prescribed metric compares favorably against state-of-the-art. The algorithm has extensive applications including signal processing, sparse recovery, machine learning and classification to name a few. Contributions.We introduce an general proximal calculus in a metric V = P ± Q ∈ S ++ (N ) given by P ∈ S ++ (N ) and a positive semi-definite rank-r matrix Q. This significantly extends the result in the preliminary version of this paper [7], where only V = P + Q with a rank-1 matrix Q is addressed. The general calculus is accompanied by several more concrete examples (see Section 3.3.4 for a non-exhaustive list), where, for example, the piecewise linear nature of certain dual problems is rigorously exploited.

Adaptive FISTA for Nonconvex Optimization

“…The modified metric is of type "identity minus rank R", where R is the column rank of D. Proximal quasi-Newton methods have recently drawn attention, as for R = 1, the associated proximal mapping can be evaluated efficiently [4,34]. See [5], for the rank-R case.…”

“…A special case of an "identity minus rank 1" metric was studied in [34], which also leads to an efficiently solvable proximal mapping. Recently, proximal mappings with respect to metrics of type "identity plus/minus rank R" have been studied in [5]. In [27], interior point methods are used to solve the proximal mapping efficiently for so-called quadratic support functions.…”

“…The considered proximal mapping is also of type "identity minus rank 1", however the relation to the SR1 metric [46,Section 8.2] is unclear and the analysis is completely different to ours. The case of an "identity plus or minus a rank-R matrix", which generalizes [4,34], was recently studied in [5].…”

“…• If f is a convex quadratic function or Algorithm 2 is used, and T k is a diagonal matrix, the update step is a proximal step of type "identity minus rank R". According to [5], such proximal mappings can be computed efficiently. The computation can be reduced to proximal mappings with respect to the diagonal part of the metric and R-dimensional root finding problems, which can be solved efficiently thanks to their small dimension.…”

“…The convergence that we establish for our method in a general non-smooth non-convex setting translates directly to a class of proximal quasi-Newton methods. Unlike the general class of proximal quasi-Newton methods which usually suffer from the problem that simple proximal mappings become hard to solve, efficient solutions to the "rank-1 proximal mappings" are known [4,34,5].…”