An almost cyclic 2-coordinate descent method for singly linearly constrained problems

Abstract

A block decomposition method is proposed for minimizing a (possibly nonconvex) continuously differentiable function subject to one linear equality constraint and simple bounds on the variables. The proposed method iteratively selects a pair of coordinates according to an almost cyclic strategy that does not use first-order information, allowing us not to compute the whole gradient of the objective function during the algorithm. Using firstorder search directions to update each pair of coordinates, global convergence to stationary points is established for different choices of the stepsize under an appropriate assumption on the level set. In particular, both inexact and exact line search strategies are analyzed. Further, linear convergence rate is proved under standard additional assumptions. Numerical results are finally provided to show the effectiveness of the proposed method.Keywords. Block coordinate descent methods. Block decomposition methods. Linear convergence rate. SVM.MSC2000 subject classifications. 90C06. 90C30. 65K05.

Active-set identification with complexity guarantees of an almost cyclic 2-coordinate descent method with Armijo line search

“…, n. In such a case, problem (1) can be obtained by applying the variable transformation x i ← a i x i and setting the lower and the upper bound accordingly. (Examples of relevant applications where problem (1) arises can be found, e.g., in [12] and the references therein. )…”

“…Let us briefly review the algorithm proposed in [12], named Almost Cyclic 2-Coordinate Descent (AC2CD) method, to solve problem (1). The main feature of AC2CD is an almost cyclic rule to choose the working set.…”

“…Other interesting examples, including the Support Vector Machine problem and the Chebyshev center problems, are those where the objective function is quadratic of the form f (x) = x T Q T Qx − q T x, with Q being a given m × n matrix and q being a given vector. In this case, a partial derivative of f (x) can be computed with a cost O(m), while computing the whole gradient has a cost O(mn) (see [12] for details). Now, let us explain in more detail how the index j(k) is chosen at the beginning of an outer iteration k and how the two variables in the working set are moved in the inner iterations (k, 1), .…”

“…Under a technical assumption (see Assumption 1 in the next section), global convergence of AC2CD to stationary points was established in [12] for different choices of the stepsize α k,i (to be used at line 8 of Algorithm 1), including the Armijo stepsize, overestimates of the local Lipschitz constants of ∇f and the exact stepsize for strictly convex objective functions 1 .…”

“…The scope of the present paper is establishing finite active-set identification of a 2-coordinate descent method, proposed by the author in [12], for smooth minimization problems with one linear equality constraint and simple bounds on the variables. The main contributions of this paper can be summarized in the following points:…”