Accelerated Randomized Mirror Descent Algorithms For Composite Non-strongly Convex Optimization

Hien, Le Thi Khanh; Nguyen, Cuong V.; Xu, Huan; Lu, Canyi; Feng, Jiashi

doi:10.48550/arxiv.1605.06892

Cited by 11 publications

(14 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…2. Our algorithm also achieves the optimal convergence rate O(1/T 2 ) for non-strongly convex functions as in [48], [49]. Fig.…”

Section: Equivalent To Its Momentum Accelerated Variantmentioning

confidence: 75%

“…More recently, many acceleration techniques were proposed to further speed up the stochastic variance reduced methods mentioned above. These techniques mainly include the Nesterov's acceleration techniques in [25], [39], [40], [41], [42], reducing the number of gradient calculations in early iterations [36], [43], [44], the projection-free property of the conditional gradient method (also known as the Frank-Wolfe algorithm [45]) as in [46], the stochastic sufficient decrease technique [47], and the momentum acceleration tricks in [36], [48], [49]. [40] proposed an accelerating Catalyst framework and achieved the oracle complexity of O((n + nL/µ) log(L/µ) log(1/ )) for strongly convex problems.…”

Section: Accelerated Sgdmentioning

confidence: 99%

See 1 more Smart Citation

VR-SGD: A Simple Stochastic Variance Reduction Method for Machine Learning

Shang

Zhou

Liu

et al. 2020

IEEE Trans. Knowl. Data Eng.

View full text Add to dashboard Cite

In this paper, we propose a simple variant of the original SVRG, called variance reduced stochastic gradient descent (VR-SGD). Unlike the choices of snapshot and starting points in SVRG and its proximal variant, Prox-SVRG, the two vectors of VR-SGD are set to the average and last iterate of the previous epoch, respectively. The settings allow us to use much larger learning rates, and also make our convergence analysis more challenging. We also design two different update rules for smooth and non-smooth objective functions, respectively, which means that VR-SGD can tackle non-smooth and/or non-strongly convex problems directly without any reduction techniques. Moreover, we analyze the convergence properties of VR-SGD for strongly convex problems, which show that VR-SGD attains linear convergence. Different from most algorithms that have no convergence guarantees for non-strongly convex problems, we also provide the convergence guarantees of VR-SGD for this case, and empirically verify that VR-SGD with varying learning rates achieves similar performance to its momentum accelerated variant that has the optimal convergence rate O(1/T 2 ). Finally, we apply VR-SGD to solve various machine learning problems, such as convex and non-convex empirical risk minimization, and leading eigenvalue computation. Experimental results show that VR-SGD converges significantly faster than SVRG and Prox-SVRG, and usually outperforms state-of-the-art accelerated methods, e.g., Katyusha.

show abstract

“…2. Our algorithm also achieves the optimal convergence rate O(1/T 2 ) for non-strongly convex functions as in [48], [49]. Fig.…”

Section: Equivalent To Its Momentum Accelerated Variantmentioning

confidence: 75%

Section: Accelerated Sgdmentioning

confidence: 99%

VR-SGD: A Simple Stochastic Variance Reduction Method for Machine Learning

Shang

Zhou

Liu

et al. 2020

IEEE Trans. Knowl. Data Eng.

View full text Add to dashboard Cite

show abstract

“…Recently momentum or Nesterov [23] technique has been successfully combined with SVRG to achieve faster convergence in real-world applications [4,12,20]. In this section, we propose an accelerated method named ALPC-SVRG, based on Katyusha momentum [4], to obtain both faster running speed and high communication-efficiency.…”

Section: Accelerated Low-precision Algorithmmentioning

confidence: 99%

Exploring Fast and Communication-Efficient Algorithms in Large-scale Distributed Networks

Yu¹,

Wu²,

Huang³

2019

Preprint

View full text Add to dashboard Cite

The communication overhead has become a significant bottleneck in data-parallel network with the increasing of model size and data samples. In this work, we propose a new algorithm LPC-SVRG with quantized gradients and its acceleration ALPC-SVRG to effectively reduce the communication complexity while maintaining the same convergence as the unquantized algorithms. Specifically, we formulate the heuristic gradient clipping technique within the quantization scheme and show that unbiased quantization methods in related works [3,33,38] are special cases of ours. We introduce double sampling in the accelerated algorithm ALPC-SVRG to fully combine the gradients of full-precision and low-precision, and then achieve acceleration with fewer communication overhead. Our analysis focuses on the nonsmooth composite problem, which makes our algorithms more general. The experiments on linear models and deep neural networks validate the effectiveness of our algorithms.

show abstract

“…When the objective function is smooth, these methods can achieve optimal O(1/K 2 ) convergence rate. Recently, Allen-Zhu (2017) and Hien et al (2016) propose optimal O(1/K 2 ) algorithms for general convex problems, named Katyusha and ASMD, respectively. For σ-strongly convex problems, Katyusha also meets the optimal O((n + nL/σ) log 1 )) rate.…”

Section: Accelerated Stochastic Gradient Algorithmsmentioning

confidence: 99%

Faster and Non-ergodic O(1/K) Stochastic Alternating Direction Method of Multipliers

Fang,

Cheng,

Lin

2017

Preprint

View full text Add to dashboard Cite

We study stochastic convex optimization subjected to linear equality constraints. Traditional Stochastic Alternating Direction Method of Multipliers Ouyang et al. (2013) and its Nesterov's acceleration scheme AzadiSra & Sra (2014) can only achieve ergodic O(1/ √ K) convergence rates, where K is the number of iteration. By introducing Variance Reduction (VR) techniques, the convergence rates improve to ergodic O(1/K) Zhong & Kwok (2014); Zheng & Kwok (2016).In this paper, we propose a new stochastic ADMM which elaborately integrates Nesterov's extrapolation and VR techniques. We prove that our algorithm can achieve a non-ergodic O(1/K) convergence rate which is optimal for separable linearly constrained non-smooth convex problems, while the convergence rates of VR based ADMM methods are actually tight O(1/ √ K) in non-ergodic sense. To the best of our knowledge, this is the first work that achieves a truly accelerated, stochastic convergence rate for constrained convex problems. The experimental results demonstrate that our algorithm is faster than the existing state-of-the-art stochastic ADMM methods.

show abstract

Accelerated Randomized Mirror Descent Algorithms For Composite Non-strongly Convex Optimization

Cited by 11 publications

References 20 publications

VR-SGD: A Simple Stochastic Variance Reduction Method for Machine Learning

VR-SGD: A Simple Stochastic Variance Reduction Method for Machine Learning

Exploring Fast and Communication-Efficient Algorithms in Large-scale Distributed Networks

Faster and Non-ergodic O(1/K) Stochastic Alternating Direction Method of Multipliers

Contact Info

Product

Resources

About