Zeroth-Order Stochastic Alternating Direction Method of Multipliers for Nonconvex Nonsmooth Optimization

Abstract

Alternating direction method of multipliers (ADMM) is a popular optimization tool for the composite and constrained problems in machine learning. However, in many machine learning problems such as black-box learning and bandit feedback, ADMM could fail because the explicit gradients of these problems are difficult or even infeasible to obtain. Zeroth-order (gradient-free) methods can effectively solve these problems due to that the objective function values are only required in the optimization. Recently, though there exist a few zeroth-order ADMM methods, they build on the convexity of objective function. Clearly, these existing zeroth-order methods are limited in many applications. In the paper, thus, we propose a class of fast zeroth-order stochastic ADMM methods (i.e., ZO-SVRG-ADMM and ZO-SAGA-ADMM) for solving nonconvex problems with multiple nonsmooth penalties, based on the coordinate smoothing gradient estimator. Moreover, we prove that both the ZO-SVRG-ADMM and ZO-SAGA-ADMM have convergence rate of O(1/T ), where T denotes the number of iterations. In particular, our methods not only reach the best convergence rate of O(1/T ) for the nonconvex optimization, but also are able to effectively solve many complex machine learning problems with multiple regularized penalties and constraints. Finally, we conduct the experiments of black-box binary classification and structured adversarial attack on black-box deep neural network to validate the efficiency of our algorithms. 1 3 ,

Scalable Plug-and-Play ADMM With Convergence Guarantees

“…where each g i is evaluated only on the subset y i ∈ R mi of the full data y. PnP-ADMM is often impractical when b is very large due to the complexity of computing prox γg . As shown in Algorithm 1, the proposed IPA algorithm extends stochastic variants of traditional ADMM [31]- [35] by integrating denoisers D σ that are not associated with any h. Its per-iteration complexity is independent of the number of data blocks b, since it processes only a single component function g i at every iteration.…”

“…Scalable optimization algorithms have become increasingly important in the context of large-scale problems arising in machine learning and data science [30]. Stochastic and online optimization techniques have been investigated for traditional ADMM [31]- [35], where prox γg is approximated using a subset of observations (with or without subsequent linearization). Our work contributes to this area by investigating the scalability of PnP-ADMM that is not minimizing any explicit objective function.…”

Adaptive Stochastic ADMM for Decentralized Reinforcement Learning in Edge Industrial IoT

“…In particular, stochastic gradient-descent (SGD) [37] and its variants have drawn a lot of research interests. Researchers have also applied variance reduction techniques for ADMM, such as [38], [39]. However, they all require large storage to keep past gradients, which can be problematic in large multitask learning.…”

A Flexible Stochastic Multi-Agent ADMM Method for Large-Scale Distributed Optimization

“…[44] have investigated the well-known SVRG strategy applied in stochastic ADMM. [45] has extensively studied stochastic ADMM combined with various variance reduction strategies for nonconvex problems. Specifically, SVRG, stochastic average gradient (SAG) and the extension of SAG (SAGA) have been incorporated into ADMM method, which has resulted in SVRG-ADMM, SAG-ADMM and SAGA-ADMM respectively.…”

“…In [44], SVRG-ADMM has been further investigated for solving convex optimization problems with composite objective functions, which has achieved the convergence rates of 𝑂(log𝑆/𝑆) for strongly convex and Lipschitz smooth objective functions, and 𝑂(1/√𝑆 ) for convex objective functions without Lipschitz smoothness. In [46], a novel stochastic ADMM has been proposed, its main ingredient is to combine classical stochastic ADMM with gradient free and variance reduction strategy.…”