On the Information-Adaptive Variants of the ADMM: An Iteration Complexity Perspective

Gao, Xiang; Jiang, Bo; Zhang, Shuzhong

doi:10.1007/s10915-017-0621-6

Cited by 69 publications

(95 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Duchi et al [10] proposed a stochastic zeroth-order mirror descent based method for convex and strongly convex functions and proved an O(1/ √ R) rate for a zeroth-order stochastic gradient method on convex objectives. Gao et al [11] proposed a stochastic gradient ADMM method that allows only noisy estimations of function values to be accessible and they proved an O(1/R) rate, under some requirements for smoothing parameter and batch size. Recently, Lian et al [12] proposed an asynchronous stochastic optimization algorithm with zeroth-order methods and proved a convergence rate of O(1/ √ R).…”

Section: A Related Workmentioning

confidence: 99%

See 1 more Smart Citation

A Proximal Zeroth-Order Algorithm for Nonconvex Nonsmooth Problems

Kazemi

Wang

2018

2018 56th Annual Allerton Conference on Communication, Control, and Computing (Allerton)

View full text Add to dashboard Cite

In this paper, we focus on solving an important class of nonconvex optimization problems which includes many problems for example signal processing over a networked multi-agent system and distributed learning over networks. Motivated by many applications in which the local objective function is the sum of smooth but possibly nonconvex part, and non-smooth but convex part subject to a linear equality constraint, this paper proposes a proximal zeroth-order primal dual algorithm (PZO-PDA) that accounts for the information structure of the problem. This algorithm only utilize the zeroth-order information (i.e., the functional values) of smooth functions, yet the flexibility is achieved for applications that only noisy information of the objective function is accessible, where classical methods cannot be applied. We prove convergence and rate of convergence for PZO-PDA. Numerical experiments are provided to validate the theoretical results.

show abstract

Section: A Related Workmentioning

confidence: 99%

“…Lemma II.1. [11] Suppose that G µ (x r , v r , ξ r ) is defined as in (7), and assumptions A.1 and A.2 hold. Then…”

Section: Algorithm Developmentmentioning

confidence: 99%

A Proximal Zeroth-Order Algorithm for Nonconvex Nonsmooth Problems

Kazemi

Wang

2018

2018 56th Annual Allerton Conference on Communication, Control, and Computing (Allerton)

View full text Add to dashboard Cite

show abstract

“…Another way to handle the non-diagonal F and the expected objective function E [l(x, ξ)] is stochastic ADMM-like methods [16,25,8,19,22,2,26,20] which aim for solving the following problem after introducing an additional variable z: min x∈X ,z=F x l(x) + r(z),…”

Section: Introductionmentioning

confidence: 99%

“…Ouyang et al [16], Suzuki [19], Azadi and Sra [2], Gao et al [8], and recently Zhao et al [25] developed several stochastic variants of ADMM, which linearize l by using its noisy subgradient or gradient and add a varying proximal term. Furthermore, Zhong and Kwok [26] and Suzuki [20] respectively proposed a stochastic averaged gradient-based ADMM and a stochastic dual coordinate ascent ADMM, which can both obtain improved iteration complexities.…”

Section: Introductionmentioning

confidence: 99%

On the iteration complexity analysis of Stochastic Primal-Dual Hybrid Gradient approach with high probability

et al. 2018

View full text Add to dashboard Cite

In this paper, we propose a stochastic Primal-Dual Hybrid Gradient (PDHG) approach for solving a wide spectrum of regularized stochastic minimization problems, where the regularization term is composite with a linear function. It has been recognized that solving this kind of problem is challenging since the closed-form solution of the proximal mapping associated with the regularization term is not available due to the imposed linear composition, and the per-iteration cost of computing the full gradient of the expected objective function is extremely high when the number of input data samples is considerably large.Our new approach overcomes these issues by exploring the special structure of the regularization term and sampling a few data points at each iteration. Rather than analyzing the convergence in expectation, we provide the detailed iteration complexity analysis for the cases of both uniformly and non-uniformly averaged iterates with high probability. This strongly supports the good practical performance of the proposed approach. Numerical experiments demonstrate that the efficiency of stochastic PDHG, which outperforms other competing algorithms, as expected by the high-probability convergence analysis.

show abstract

“…The authors of monographs [14], [15] summarized several classes of derivativefree methods, including both deterministic and stochastic cases. In addition, there were some other algorithms for solving convex optimization using derivative-free or zeroth-order information, including the zeroth-order mirror decent algorithm [18], the zeroth-order ADMM algorithm [19], the Kiefer-Wolfowitz (KW) algorithm [20], and the Nelder-Mead algorithm [22]. For the Nelder-Mead algorithm, introduced in [22], to the best of our knowledge, theoretical properties are still under investigation.…”

mentioning

confidence: 99%

Distributed Subgradient-Free Stochastic Optimization Algorithm for Nonsmooth Convex Functions over Time-Varying Networks

Wang¹,

Zhao²,

Ye³

et al. 2019

SIAM J. Control Optim.

View full text Add to dashboard Cite

In this paper we consider a distributed stochastic optimization problem without the gradient/subgradient information for the local objective functions, subject to local convex constraints. The objective functions may be non-smooth and observed with stochastic noises, and the network for the distributed design is time-varying.By adding the stochastic dithers into the local objective functions and constructing the randomized differences motivated by the Kiefer-Wolfowitz algorithm, we propose a distributed subgradient-free algorithm to find the global minimizer with local observations. Moreover, we prove that the consensus of estimates and global minimization can be achieved with probability one over the time-varying network, and then obtain the convergence rate of the mean average of estimates as well. Finally, we give a numerical example to illustrate the effectiveness of the proposed algorithm.

show abstract

On the Information-Adaptive Variants of the ADMM: An Iteration Complexity Perspective

Cited by 69 publications

References 34 publications

A Proximal Zeroth-Order Algorithm for Nonconvex Nonsmooth Problems

A Proximal Zeroth-Order Algorithm for Nonconvex Nonsmooth Problems

On the iteration complexity analysis of Stochastic Primal-Dual Hybrid Gradient approach with high probability

Distributed Subgradient-Free Stochastic Optimization Algorithm for Nonsmooth Convex Functions over Time-Varying Networks

Contact Info

Product

Resources

About