The Proximal Alternating Direction Method of Multipliers in the Nonconvex Setting: Convergence Analysis and Rates

Abstract

We propose two numerical algorithms in the fully nonconvex setting for the minimization of the sum of a smooth function and the composition of a nonsmooth function with a linear operator. The iterative schemes are formulated in the spirit of the proximal alternating direction method of multipliers and its linearized variant, respectively. The proximal terms are introduced via variable metrics, a fact that allows us to derive new proximal splitting algorithms for nonconvex structured optimization problems, as particular instances of the general schemes. Under mild conditions on the sequence of variable metrics and by assuming that a regularization of the associated augmented Lagrangian has the Kurdyka–Łojasiewicz property, we prove that the iterates converge to a Karush–Kuhn–Tucker point of the objective function. By assuming that the augmented Lagrangian has the Łojasiewicz property, we also derive convergence rates for both the augmented Lagrangian and the iterates.

Factorization of completely positive matrices using iterative projected gradient steps

“…An essential tool for deriving the rates of convergence is the following lemma, the proof of which can be found in Reference 50, Lemma 15.…”

A Proximal Minimization Algorithm for Structured Nonconvex and Nonsmooth Problems

“…Splitting algorithms for solving problems of the form (1.2) have been considered in [19], under the assumption that H is twice continuously differentiable with bounded Hessian, in [25], under the assumption that one of the summands is convex and continuous on its effective domain, and in [13], as a particular case of a general nonconvex proximal ADMM algorithm. We would like to mention in this context also [10] for the case when A is nonlinear.…”

“…Remark 1. piq In case Gpyq " 0 and Hpx, yq " Hpxq for any px, yq P R mˆRq , where H : R m Ñ R is a Fréchet differentiable function with Lipschitz continuous gradient, Algorithm 1 gives rise to an iterative scheme which has been proposed in [13] for solving the optimization problem (1.2). This reads for any n ě 0 z n`1 P prox β´1F`A x n`β´1 u nx n`1 :" x n´τ´1`∇ H px n q`A T u n`β A T pAx n´zn`1 qȗ n`1 :" u n`σ β pAx n`1´zn`1 q .…”

“…Usually, for nonconvex algorithms, the fact that the sequences of differences of consecutive iterates converge to zero is shown by assuming that the generated sequences are bounded (see [13,19,25]). In our analysis the only ingredients for obtaining statement (ii) in Theorem 6 are the descent property and Lemma 2.…”

“…The following lemma will provide convergence rates for a particular class of monotonically decreasing real sequences converging to 0. Its proof can be found in [13,Lemma 15].…”