Coordinate-friendly structures, algorithms and applications

Abstract

This paper focuses on coordinate update methods, which are useful for solving problems involving large or high-dimensional datasets. They decompose a problem into simple subproblems, where each updates one, or a small block of, variables while fixing others. These methods can deal with linear and nonlinear mappings, smooth and nonsmooth functions, as well as convex and nonconvex problems. In addition, they are easy to parallelize.The great performance of coordinate update methods depends on solving simple subproblems. To derive simple subproblems for several new classes of applications, this paper systematically studies coordinate friendly operators that perform low-cost coordinate updates.Based on the discovered coordinate friendly operators, as well as operator splitting techniques, we obtain new coordinate update algorithms for a variety of problems in machine learning, image processing, as well as sub-areas of optimization. Several problems are treated with coordinate update for the first time in history. The obtained algorithms are scalable to large instances through parallel and even asynchronous computing. We present numerical examples to illustrate how effective these algorithms are.

Markov chain block coordinate descent

“…Third, it is interesting to develop a primal-dual type MC-BCD, which would apply to a model-free DMDP along a single trajectory. Yet another line of work applies block coordinate update to linear and nonlinear fixed-point problems [18,17,5] because it can solve optimization problems in imaging and conic programming, which are equipped with nonsmooth, nonseparable objectives, and constraints.…”

A systematic review of structured sparse learning

“…Generic QP methods after reformulation (Bach et al, 2012a), alternating direction methods (Boyd et al, 2011), proximal methods (Parikh and Boyd, 2014), block coordinate descent methods (Tseng and Yun, 2009;Wen et al, 2012;Peng et al, 2016), iteratively reweighted methods (Chartrand and Yin, 2008;Lai et al, 2013), working-set and homotopy methods (Bach et al, 2012a) CVX (Grant and Boyd, 2013), SDPT3 (Toh et al, 2006), YALL1 (Zhang Y et al, 2011), SPGL1 (van den Berg andFriedlander, 2007), SLEP , SPAMs , SparseLab (Donoho et al, 2007) Grouped Lasso (Yuan and Lin, 2006)…”

“…Generic SDP methods after reformulation (Bach, 2008a), alternating direction methods (Boyd et al, 2011), proximal methods (Parikh and Boyd, 2014), block coordinate descent methods (Tseng and Yun, 2009;Wen et al, 2012;Peng et al, 2016), iteratively reweighted methods (Chartrand and Yin, 2008;Lai et al, 2013), working-set and homotopy methods (Bach et al, 2012a) CVX (Grant and Boyd, 2013), SDPT3 (Toh et al, 2006), SLEP , SPAMs (Suzuki, 2013) Cannot be improved upon without further assumptions Gradient descent (Nesterov, 2004) O(log(1/ε)) (Hazan et al, 2007) Applicable when f (x) is a strongly convex function (Su et al, 2014) O(log(1/ε)) (Tseng, 2008) Applicable when f (x) is differentiable. Cannot be improved upon without further assumptions.…”

Asynchronous variance-reduced block schemes for composite non-convex stochastic optimization: block-specific steplengths and adapted batch-sizes

“…BCD methods [10] are widely used in machine learning and optimization, where variables are partitioned into manageable blocks and in each iteration, a single block is chosen to update while the remaining blocks remain fixed. Recently, in [25], coordinate-friendly operators were investigated that perform low-cost coordinate updates and it is shown that a variety of problems in machine learning can be efficiently resolved by such an update. The convergence properties of cyclic BCD methods has been extensively analyzed in [27,38,40].…”