A hybrid stochastic optimization framework for composite nonconvex optimization

Tran-Dinh, Quoc; Pham, Nhan H.; Phan, Dzung T.; Nguyen, Lam M.

doi:10.1007/s10107-020-01583-1

Cited by 27 publications

(39 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…where (i) follows from a similar line of arguments in (25). Then (13) follows from using (44) in (43).…”

Section: C2 Proof Of Lemma 5(b)mentioning

confidence: 99%

“…In this paper, we propose GT-HSGD, a novel online variance-reduced method for decentralized non-convex optimization with stochastic first-order oracles (SFO). To achieve fast and robust performance, the GT-HSGD algorithm is built upon global gradient tracking [19,20] and a local hybrid stochastic gradient estimator [42][43][44] that can be considered as a convex combination of the vanilla stochastic gradient returned by the SFO and a SARAH-type variance-reduced stochastic gradient [45]. In the following, we emphasize the key advantages of GT-HSGD compared with the existing decentralized online (variance-reduced) approaches, from both theoretical and practical aspects.…”

Section: Our Contributionsmentioning

confidence: 99%

“…A comparison of the oracle complexity of GT-HSGD with related algorithms is provided in Table 1, from which we have the following important observations. First of all, the oracle complexity of GT-HSGD is lower than that of DSGD, D2, GT-DSGD and D-PD-SGD, which are decentralized online algorithms without variance reduction; however, GT-HSGD imposes on the SFO an additional mean-squared smoothness (MSS) assumption that is required by all online variance-reduced techniques [29][30][31][32][33][42][43][44]46,47] in the literature. Secondly, GT-HSGD further achieves a lower oracle complexity than the existing decentralized online variance-reduced methods D-GET [29] and D-SPIDER-SFO [30], especially in a regime where the required error tolerance and the network spectral gap 1 − λ are relatively small.…”

Section: Our Contributionsmentioning

confidence: 99%

“…Motivated by the strong performance of recently introduced decentralized methods that combine gradient tracking and various variance reduction schemes for finite-sum problems [29,37,39,40], we seek similar variance reduction for decentralized online problems with SFO. In particular, we focus on the following local hybrid stochastic gradient estimator v i t introduced in [42][43][44] for centralized online problems: ∀t ≥ 1,…”

Section: Algorithm Development and Descriptionmentioning

confidence: 99%

See 3 more Smart Citations

A Hybrid Variance-Reduced Method for Decentralized Stochastic Non-Convex Optimization

Ran¹,

Khan²,

Kar³

2021

Preprint

View full text Add to dashboard Cite

This paper considers decentralized stochastic optimization over a network of n nodes, where each node possesses a smooth non-convex local cost function and the goal of the networked nodes is to find an -accurate first-order stationary point of the sum of the local costs. We focus on an online setting, where each node accesses its local cost only by means of a stochastic first-order oracle that returns a noisy version of the exact gradient. In this context, we propose a novel single-loop decentralized hybrid variance-reduced stochastic gradient method, called GT-HSGD, that outperforms the existing approaches in terms of both the oracle complexity and practical implementation. The GT-HSGD algorithm implements specialized local hybrid stochastic gradient estimators that are fused over the network to track the global gradient. Remarkably, GT-HSGD achieves a network-independent oracle complexity of O(n −1 −3 ) when the required error tolerance is small enough, leading to a linear speedup with respect to the centralized optimal online variance-reduced approaches that operate on a single node. Numerical experiments are provided to illustrate our main technical results.

show abstract

“…where (i) follows from a similar line of arguments in (25). Then (13) follows from using (44) in (43).…”

Section: C2 Proof Of Lemma 5(b)mentioning

confidence: 99%

Section: Our Contributionsmentioning

confidence: 99%

Section: Our Contributionsmentioning

confidence: 99%

Section: Algorithm Development and Descriptionmentioning

confidence: 99%

See 2 more Smart Citations

A Hybrid Variance-Reduced Method for Decentralized Stochastic Non-Convex Optimization

Ran¹,

Khan²,

Kar³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…A standard (or vanilla) SGM often converges slowly. Several acceleration techniques have been used to improve its theoretical and/or empirical convergence speed (e.g., [3,15,24,62,65]) for solving convex or smooth nonconvex problems. However, for nonsmooth nonconvex problems, it appears that it is still unknown whether a proximal SGM or a stochastic subgradient method (SsGM) can still have guaranteed convergence if a certain acceleration technique is applied.…”

mentioning

confidence: 99%

Distributed stochastic inertial-accelerated methods with delayed derivatives for nonconvex problems

Xu¹,

Xu²,

Yan³

et al. 2021

Preprint

View full text Add to dashboard Cite

Stochastic gradient methods (SGMs) are predominant approaches for solving stochastic optimization. On smooth nonconvex problems, a few acceleration techniques have been applied to improve the convergence rate of SGMs. However, little exploration has been made on applying a certain acceleration technique to a stochastic subgradient method (SsGM) for nonsmooth nonconvex problems. In addition, few efforts have been made to analyze an (accelerated) SsGM with delayed derivatives. The information delay naturally happens in a distributed system, where computing workers do not coordinate with each other.In this paper, we propose an inertial proximal SsGM for solving nonsmooth nonconvex stochastic optimization problems. The proposed method can have guaranteed convergence even with delayed derivative information in a distributed environment. Convergence rate results are established to three classes of nonconvex problems: weaklyconvex nonsmooth problems with a convex regularizer, composite nonconvex problems with a nonsmooth convex regularizer, and smooth nonconvex problems. For each problem class, the convergence rate is O(1/K 1 2 ) in the expected value of the gradient norm square, for K iterations. In a distributed environment, the convergence rate of the proposed method will be slowed down by the information delay. Nevertheless, the slow-down effect will decay with the number of iterations for the latter two problem classes. We test the proposed method on three applications. The numerical results clearly demonstrate the advantages of using the inertial-based acceleration. Furthermore, we observe higher parallelization speed-up in asynchronous updates over the synchronous counterpart, though the former uses delayed derivatives.

show abstract