On the Efficiency of Random Permutation for ADMM and Coordinate Descent

Sun, Ruoyu; Luo, Ze; Ye, Yinyu

doi:10.1287/moor.2019.0990

Cited by 16 publications

(8 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, concerning the choice of the ADMM parameter β, we observed that for larger problems an increasing value of β is recommended: we chose β = 10 In Table 2 we report the results obtained using LIBSVM Version 3.25 [8], which implements specialized algorithms to address the SVM problem (LIBSVM uses a Sequential Minimal Optimization type decomposition method [4,14,31]). In Table 3 we report the results obtained using RACQP [29] (where a multi-block generalization of ADMM is employed, see also [10,38] for related theoretical analysis).…”

Section: Numerical Resultsmentioning

confidence: 99%

Training very large scale nonlinear SVMs using Alternating Direction Method of Multipliers coupled with the Hierarchically Semi-Separable kernel approximations

Cipolla¹,

Gondzio²

2021

Preprint

View full text Add to dashboard Cite

Typically, nonlinear Support Vector Machines (SVMs) produce significantly higher classification quality when compared to linear ones but, at the same time, their computational complexity is prohibitive for large-scale datasets: this drawback is essentially related to the necessity to store and manipulate large, dense and unstructured kernel matrices. Despite the fact that at the core of training a SVM there is a simple convex optimization problem, the presence of kernel matrices is responsible for dramatic performance reduction, making SVMs unworkably slow for large problems. Aiming to an efficient solution of large-scale nonlinear SVM problems, we propose the use of the Alternating Direction Method of Multipliers coupled with Hierarchically Semi-Separable (HSS) kernel approximations. As shown in this work, the detailed analysis of the interaction among their algorithmic components unveils a particularly efficient framework and indeed, the presented experimental results demonstrate a significant speed-up when compared to the state-of-the-art nonlinear SVM libraries (without significantly affecting the classification accuracy).

show abstract

Section: Numerical Resultsmentioning

confidence: 99%

Training very large scale nonlinear SVMs using Alternating Direction Method of Multipliers coupled with the Hierarchically Semi-Separable kernel approximations

Cipolla¹,

Gondzio²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In particular, we focus mostly on the evaluation of the performance of mRrDR. We also compare mRrDR with some of the state-of-the-art methods, namely, RK [61], REK [66], RGS [31,39], and RP-ADMM [62].…”

Section: Numerical Experimentsmentioning

confidence: 99%

Randomized Douglas-Rachford method for linear systems: Improved accuracy and efficiency

Han¹,

Su²,

Xie³

2022

Preprint

View full text Add to dashboard Cite

The Douglas-Rachford (DR) method is a widely used method for finding a point in the intersection of two closed convex sets (feasibility problem). However, the method converges weakly and the associated rate of convergence is hard to analyze in general. In addition, the direct extension of the DR method for solving more-than-two-sets feasibility problems, called the r-sets-DR method, is not necessarily convergent. Recently, the introduction of randomization and momentum techniques in optimization algorithms has attracted increasing attention as it makes the algorithms more efficient. In this paper, we show that randomization is powerful for the DR method and can make the divergent r-sets-DR method converge. Specifically, we propose the randomized r-sets-DR (RrDR) method for solving the linear system Ax = b and prove its linear convergence in expectation, and the convergence rate does not depend on the dimension of the matrix A. We also study RrDR with heavy ball momentum and show that it converges at an accelerated linear rate. Numerical experiments are provided to confirm our results and demonstrate the significant improvement in accuracy and efficiency of the DR method that randomization and momentum bring.C i where C i := {x : a i , x = b i }.

show abstract

“…For convenience, we follow the notation in [17] and [68,69] to describe the iterative scheme of RAC-ADMM in a matrix form. Let L σ ∈ R n×n be s × s block matrix defined with respect to σ i rows and σ j columns as…”

Section: Rac-admm As a Linear Transformationmentioning

confidence: 99%

Managing randomization in the multi-block alternating direction method of multipliers for quadratic optimization

Mihić

Zhu

2020

Math. Prog. Comp.

Self Cite

View full text Add to dashboard Cite

The Alternating Direction Method of Multipliers (ADMM) has gained a lot of attention for solving large-scale and objective-separable constrained optimization. However, the two-block variable structure of the ADMM still limits the practical computational efficiency of the method, because one big matrix factorization is needed at least once even for linear and convex quadratic programming. This drawback may be overcome by enforcing a multi-block structure of the decision variables in the original optimization problem. Unfortunately, the multi-block ADMM, with more than two blocks, is not guaranteed to be convergent. On the other hand, two positive developments have been made: first, if in each cyclic loop one randomly permutes the updating order of the multiple blocks, then the method converges in expectation for solving any system of linear equations with any number of blocks. Secondly, such a randomly permuted ADMM also works for equality-constrained convex quadratic programming even when the objective function is not separable. The goal of this paper is twofold. First, we add more randomness into the ADMM by developing a randomly assembled cyclic ADMM (RAC-ADMM) where the decision variables in each block are randomly assembled. We discuss the theoretical properties of RAC-ADMM and show when random assembling helps and when it hurts, and develop a criterion to guarantee that it converges almost surely. Secondly, using the theoretical guidance on RAC-ADMM, we conduct multiple numerical tests on solving both randomly generated and large-scale benchmark quadratic optimization problems, which include continuous, and binary graph-partition and quadratic assignment, and selected machine learning problems. Our numerical tests show that the RAC-ADMM, with a variable-grouping strategy, could significantly improve the computation efficiency on solving most quadratic optimization problems.

show abstract

On the Efficiency of Random Permutation for ADMM and Coordinate Descent

Cited by 16 publications

References 50 publications

Training very large scale nonlinear SVMs using Alternating Direction Method of Multipliers coupled with the Hierarchically Semi-Separable kernel approximations

Training very large scale nonlinear SVMs using Alternating Direction Method of Multipliers coupled with the Hierarchically Semi-Separable kernel approximations

Randomized Douglas-Rachford method for linear systems: Improved accuracy and efficiency

Managing randomization in the multi-block alternating direction method of multipliers for quadratic optimization

Contact Info

Product

Resources

About