Stochastic Coordinate Descent Methods for Regularized Smooth and Nonsmooth Losses

Abstract

Stochastic Coordinate Descent (SCD) methods are among the first optimization schemes suggested for efficiently solving large scale problems. However, until now, there exists a gap between the convergence rate analysis and practical SCD algorithms for general smooth losses and there is no primal SCD algorithm for nonsmooth losses. In this paper, we discuss these issues using the recently developed structural optimization techniques. In particular, we first present a principled and practical SCD algorithm for regularized smooth losses, in which the one-variable subproblem is solved using the proximal gradient method and the adaptive componentwise Lipschitz constant is obtained employing the line search strategy. When the loss is nonsmooth, we present a novel SCD algorithm, in which the one-variable subproblem is solved using the dual averaging method. We show that our algorithms exploit the regularization structure and achieve several optimal convergence rates that are standard in the literature. The experiments demonstrate the expected efficiency of our SCD algorithms in both smooth and nonsmooth cases.

Stochastic Dual Ascent for Solving Linear Systems

“…Our work belongs to a growing literature on randomized methods for various problems appearing in linear algebra, optimization and computer science. In particular, relevant methods include sketching algorithms, randomized Kaczmarz, stochastic gradient descent and their variants [55,31,9,38,66,33,34,45,50,56,17,47,19,62,8,18,65,35,7,32,26,14,40,23] and randomized coordinate and subspace type methods and their variants [21,16,51,36,60,3,48,37,49,57,28,58,30,46,11,53,10,12,20,41,13,42,43,64,44,…”

Smooth Minimization of Nonsmooth Functions with Parallel Coordinate Descent Methods

“…When the function is not smooth neither composite, it is still possible to define coordinate descent methods with subgradients. An algorithm based on the averaging of past subgradient coordinates is presented in [34] and a successful subgradient-based coordinate descent method for problems with sparse subgradients is proposed by Nesterov [20]. Tappenden et al [36] analyzed an inexact randomized coordinate descent method in which proximal subproblems at each iteration are solved only approximately.…”

A generic coordinate descent solver for non-smooth convex optimisation

“…Paper Rate Const PD-CD Notable feature Platt '99 [43] × ✓ P for SVM Tseng & Yun '09 [61] ✓ ✓ P adapts Gauss-Southwell rule Tao et al '12 [57] ✓ × P uses averages of subgradients Necoara et al '12 [38] ✓ ✓ P 2-coordinate descent Nesterov '12 [40] ✓ × P uses subgradients Necoara & Clipici '13 [37] ✓ ✓ P coupled constraints Combettes & Pesquet '14 [11] × ✓ ✓ 1st PD-CD, short step sizes Bianchi et al '14 [5] × ✓ ✓ distributed optimization Hong et al '14 [24] × ✓ × updates all dual variables Fercoq & Richtárik '17 [19] ✓ × P uses smoothing Alacaoglu et al '17 [1] ✓ ✓ ✓ 1st PD-CD w rate for constraints Xu & Zhang '18 [66] ✓ ✓ × better rate than [21] Chambolle et al '18 [9] ✓ ✓ × updates all primal variables Fercoq & Bianchi '19 [15] ✓ ✓ ✓ 1st PD-CD w long step sizes Gao et al '19 [21] ✓ ✓ × 1st primal-dual w rate for constraints Latafat et al '19 [26] ✓ ✓ ✓ linear conv w growth condition Table 3. Selected papers for the minimization of non-smooth non-separable functions.…”