Anderson Accelerated Douglas-Rachford Splitting

Abstract

We consider the problem of non-smooth convex optimization with general linear constraints, where the objective function is only accessible through its proximal operator. This problem arises in many different fields such as statistical learning, computational imaging, telecommunications, and optimal control. To solve it, we propose an Anderson accelerated Douglas-Rachford splitting (A2DR) algorithm, which we show either globally converges or provides a certificate of infeasibility/unboundedness under very mild conditions. Applied to a block separable objective, A2DR partially decouples so that its steps may be carried out in parallel, yielding an algorithm that is fast and scalable to multiple processors. We describe an open-source implementation and demonstrate its performance on a wide range of examples.

Anderson Acceleration of Proximal Gradient Methods

“…However, AA and optimization algorithms have been developed quite independently and only limited connections were discovered and studied [16,18]. Very recently, the technique has started to gain a significant interest in the optimization community (see, e.g., [47,46,5,53,19,39]). Specifically, a series of papers [47,46,5] adapt AA to accelerate several classical algorithms for unconstrained optimization; [53] studies a variant of AA for non-expansive operators; [19] proposes an application of AA to Douglas-Rachford splitting; and [39] uses AA to improve the performance of the ADMM method.…”

“…Very recently, the technique has started to gain a significant interest in the optimization community (see, e.g., [47,46,5,53,19,39]). Specifically, a series of papers [47,46,5] adapt AA to accelerate several classical algorithms for unconstrained optimization; [53] studies a variant of AA for non-expansive operators; [19] proposes an application of AA to Douglas-Rachford splitting; and [39] uses AA to improve the performance of the ADMM method. There is also an emerging literature on applications of AA in machine learning [23,28,20,36].…”

“…None of the proposed techniques are able to guarantee acceleration, or even convergence, for problems with a general convex constraint set. We are only aware of the very recent work [19], which considers linearly constrained minimization. There, using an additional stabilization strategy similar to the one in [53], the authors show global and asymptotic convergence, albeit no convergence rate is given.…”

“…Extrapolation techniques have a long history in numerical analysis (see, e.g., [48,7]). Recently, its idea has resurfaced in the first-order optimization literature [47,53,27,19,39]. Unlike momentum acceleration methods such as Polyak's heavy ball [38] and Nesterov's fast gradient [32], which require knowledge of problem parameters, classical extrapolation techniques for vector sequences such as minimal polynomial extrapolation [49], reduced rank extrapolation [15], vector epsilon algorithm [52], and Anderson acceleration [1] estimate the solution directly from the available iterate sequence.…”

“…However, whether native AA converges globally still remains largely unknown (cf. [19]). Here, we show a negative answer to this question.…”