Douglas--Rachford Splitting and ADMM for Nonconvex Optimization: Tight Convergence Results

Abstract

Although originally designed and analyzed for convex problems, the alternating direction method of multipliers (ADMM) and its close relatives, Douglas-Rachford splitting (DRS) and Peaceman-Rachford splitting (PRS), have been observed to perform remarkably well when applied to certain classes of structured nonconvex optimization problems. However, partial global convergence results in the nonconvex setting have only recently emerged. In this paper we show how the Douglas-Rachford envelope (DRE), introduced in 2014, can be employed to unify and considerably simplify the theory for devising global convergence guarantees for ADMM, DRS and PRS applied to nonconvex problems under less restrictive conditions, larger prox-stepsizes and over-relaxation parameters than previously known. In fact, our bounds are tight whenever the over-relaxation parameter ranges in (0, 2]. The analysis of ADMM uses a universal primal equivalence with DRS that generalizes the known duality of the algorithms. (DRS)The case λ = 1 corresponds to the classical DRS, whereas for λ = 2 the scheme is also known as Peaceman-Rachford splitting (PRS). If s is a fixed point for the DR-iterationthat is, such that s + = s -then it can be easily seen that u satisfies the first-order necessary condition for optimality in problem (1.1). When both ϕ 1 and ϕ 2 are convex functions, the

Joint Communication, Computation, Caching, and Control in Big Data Multi-Access Edge Computing

“…We solve the proximal upper-bound problem through use of distributed optimization control algorithm for 4C (Algorithm 3) and CVXPY [55] (a Python-embedded modeling language for solving convex optimization problems). Furthermore, we compare the solution of our distributed optimization control algorithm with the solution computed via Douglas-Rachford splitting [56] without applying a rounding technique. Thus, our formulated problem in (37) is decomposable.…”

Survey: Sixty Years of Douglas–rachford

“…Li and Pong also provided detailed results on the convergence rates. Andreas Themelis and Panos Patrinos have since published a follow up article [137] in which they relax some of the restrictions on the step size η, as well as providing a discussion of the connections with ADMM.…”

An Envelope for Davis–Yin Splitting and Strict Saddle-Point Avoidance

“…Recently, FBS and DRS are found to numerically converge for certain nonconvex problems, for example, FBS for image restoration [22], dictionary learning, and matrix decomposition [25], and DRS for nonconvex feasibility problem [14], matrix completion [1], and phase retrieval [7]. Theoretically, their iterates have been shown to converge to stationary points in some nonconvex settings [2,14,26,11]. In particular, any bounded sequence produced by FBS converges to a stationary point when the objective satisfies the KL property [2]; By using the Douglas-Rachford Envelope (DRE), DRS iterates are shown to converge to a stationary point when one of the two functions is Lipschitz differentiable, both of them are semi-algebraic and bounded below, and one of them is coercive [14,26]; In [11], when one function is strongly convex and the other is weakly convex, and their sum is strongly convex, DRS iterates are shown to be Fejer monotone with respect to the set of fixed points of DRS operator, thus convergent.…”

“…Theoretically, their iterates have been shown to converge to stationary points in some nonconvex settings [2,14,26,11]. In particular, any bounded sequence produced by FBS converges to a stationary point when the objective satisfies the KL property [2]; By using the Douglas-Rachford Envelope (DRE), DRS iterates are shown to converge to a stationary point when one of the two functions is Lipschitz differentiable, both of them are semi-algebraic and bounded below, and one of them is coercive [14,26]; In [11], when one function is strongly convex and the other is weakly convex, and their sum is strongly convex, DRS iterates are shown to be Fejer monotone with respect to the set of fixed points of DRS operator, thus convergent. Though unlikely, it is still possible that the limit of a convergent sequence is a saddle point instead of a local minimum (except when all stationary points are local minima, which is the case studied in [11]).…”