A Three-Operator Splitting Scheme and its Optimization Applications

Abstract

Operator splitting schemes have been successfully used in computational sciences to reduce complex problems into a series of simpler subproblems. Since 1950s, these schemes have been widely used to solve problems in PDE and control. Recently, large-scale optimization problems in machine learning, signal processing, and imaging have created a resurgence of interest in operator-splitting based algorithms because they often have simple descriptions, are easy to code, and have (nearly) state-of-the-art performance for large-scale optimization problems. Although operator splitting techniques were introduced over 60 years ago, their importance has significantly increased in the past decade. This paper introduces a new operator-splitting scheme for solving a variety of problems that are reduced to a monotone inclusion of three operators, one of which is cocoercive. Our scheme is very simple, and it does not reduce to any existing splitting schemes. Our scheme recovers the existing forward-backward, Douglas-Rachford, and forward-Douglas-Rachford splitting schemes as special cases.Our new splitting scheme leads to a set of new and simple algorithms for a variety of other problems, including the 3-set split feasibility problems, 3-objective minimization problems, and doubly and multiple regularization problems, as well as the simplest extension of the classic ADMM from 2 to 3 blocks of variables. In addition to the basic scheme, we introduce several modifications and enhancements that can improve the convergence rate in practice, including an acceleration that achieves the optimal rate of convergence for strongly monotone inclusions. Finally, we evaluate the algorithm on several applications.

Robust quaternion matrix completion with applications to image inpainting

“…For color images, the image inpainting methods are applied to red, green, and blue channels separately, and the resulting color image is combined with the inpainting results from the three color channels . In the low‐rank tensor completion algorithm proposed in the work of Liu et al, the operator splitting toolbox is employed. The stopping criteria of all these iterative methods are that the norm of the successive iterates is less than tol and the maximum number of iterations is 1,000.…”

Robust Convergence Analysis of Three-Operator Splitting

“…In Algorithm 1, prox is the proximal operator (see Definition 1), α is the proximal stepsize and λ is the relaxation parameter. [5] proves that a proper selection of λ and α ensures that the sequence {x k B } converges asymptotically to a minimizer of (1). The rate of convergence towards optimality depends on the regularity assumptions about f, g and h. In this paper, our goal is to develop a principled and systematic way to analyze the convergence of TOS under various assumptions about f , g and h.…”

“…[12,13] and [5] prove the O(1/k) ergodic convergence rate on the saddle point suboptimality and function value suboptimality, respectively. When both f (x) and h(x) are Lipschitz differentiable, [5,12] give an O(1/k) convergence proof in terms of the objective function value suboptimality. Furthermore, they derive linear convergence under stronger assumptions.…”

“…A differentiable function f belongs to the class F (m, L) on S if and only if the gradient function ∇f satisfies the incremental quadratic constraint in (5) where Q = Q(m, L) is given by [23,14] Q(m, L) = − mL…”

“…Problems of the form (1) encompass a variety of problems in signal processing, control, and machine learning, such as group LASSO [1], support vector machines [2], matrix completion [3] and optimal control [4]. To solve (1), [5] proposed the three-operator splitting (TOS) method outlined below.…”