An optimal randomized incremental gradient method

Lan, Guanghui; Zhou, Yi

doi:10.48550/arxiv.1507.02000

Cited by 31 publications

(61 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To the best of our knowledge, ANITA is the first accelerated algorithm which can exactly achieve this optimal result O n + nL for general convex finite-sum problems. In the strongly convex setting, we also show that ANITA can achieve the optimal convergence result O n + nL µ log 1 matching the lower bound Ω n + nL µ log 1 provided by Lan and Zhou (2015). Moreover, ANITA enjoys a simpler loopless algorithmic structure unlike previous accelerated algorithms such as Katyusha (Allen-Zhu, 2017) and Varag (Lan et al, 2019) where they use an inconvenient double-loop structure.…”

supporting

confidence: 59%

“…• Finally, for strongly convex finite-sum problems (i.e., Assumption 2 holds), we also prove that ANITA achieves the optimal convergence result O n+ nL µ log 1 matching the lower bound Ω n+ nL µ log 1 provided by Lan and Zhou (2015) (see Table 1). b ANITA can achieve this optimal result for a very wide range of , i.e., ∈ (0, 2 for more details).…”

Section: Our Contributionsmentioning

confidence: 63%

“…Therefore, much recent research effort has been devoted to the design of accelerated gradient methods (e.g., Nesterov, 2004;Lan, 2012;Allen-Zhu and Orecchia, 2014;Su et al, 2014;Lin et al, 2015;Allen-Zhu, 2017;Lan and Zhou, 2018;Lan et al, 2019;Li et al, 2020b). As can be seen from Table 1, for strongly convex finite-sum problems, existing accelerated methods such as RPDG (Lan and Zhou, 2015), Katyusha (Allen-Zhu, 2017) and Varag (Lan et al, 2019) are optimal since their convergence results are O n + nL µ log 1 matching the lower bound Ω n + nL µ log 1 given by Lan and Zhou (2015). However, for general (non-strongly) convex finite-sum problems, all previous accelerated methods do not achieve the optimal convergence result.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

ANITA: An Optimal Loopless Accelerated Variance-Reduced Gradient Method

Li¹

2021

Preprint

View full text Add to dashboard Cite

We propose a novel accelerated variance-reduced gradient method called ANITA for finite-sum optimization. In this paper, we consider both general convex and strongly convex settings. In the general convex setting, ANITA achieves the convergence result O n min 1 + log 1 √ n , log √ n + nL , which improves the previous best result O n min{log 1 , log n} + nL given by Varag (Lan et al., 2019). In particular, for a very wide range of , i.e., ∈ (0,, where is the error tolerance f (xT ) − f * ≤ and n is the number of data samples, ANITA can achieve the optimal convergence result O n + nL matching the lower bound Ω n + nL provided by Woodworth and Srebro (2016). To the best of our knowledge, ANITA is the first accelerated algorithm which can exactly achieve this optimal result O n + nL for general convex finite-sum problems. In the strongly convex setting, we also show that ANITA can achieve the optimal convergence result O n + nL µ log 1 matching the lower bound Ω n + nL µ log 1 provided by Lan and Zhou (2015). Moreover, ANITA enjoys a simpler loopless algorithmic structure unlike previous accelerated algorithms such as Katyusha (Allen-Zhu, 2017) and Varag (Lan et al., 2019) where they use an inconvenient double-loop structure. Finally, the experimental results also show that ANITA converges faster than previous state-of-the-art Varag (Lan et al., 2019), validating our theoretical results and confirming the practical superiority of ANITA.

show abstract

supporting

confidence: 59%

Section: Our Contributionsmentioning

confidence: 63%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

ANITA: An Optimal Loopless Accelerated Variance-Reduced Gradient Method

Li¹

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Acceleration of gradient-type methods is widely-studied for standard optimization problems (Lan and Zhou, 2015;Lin et al, 2015;Allen-Zhu, 2017;Lan et al, 2019;Li, 2021a). Deep learning practitioners typically rely on Adam (Kingma and Ba, 2014), or one of its many variants, which besides other tricks also adopts momentum.…”

Section: Methods With Accelerationmentioning

confidence: 99%

CANITA: Faster Rates for Distributed Convex Optimization with Communication Compression

Li¹,

Richtárik²

2021

Preprint

View full text Add to dashboard Cite

Due to the high communication cost in distributed and federated learning, methods relying on compressed communication are becoming increasingly popular. Besides, the best theoretically and practically performing gradient-type methods invariably rely on some form of acceleration/momentum to reduce the number of communications (faster convergence), e.g., Nesterov's accelerated gradient descent (Nesterov, 2004) and Adam (Kingma and Ba, 2014).In order to combine the benefits of communication compression and convergence acceleration, we propose a compressed and accelerated gradient method for distributed optimization, which we call CANITA. Our CANITA achieves the first accelerated rate O, which improves upon the state-of-the-art non-accelerated rate (Khaled et al., 2020b) for distributed general convex problems, where ǫ is the target error, L is the smooth parameter of the objective, n is the number of machines/devices, and ω is the compression parameter (larger ω means more compression can be applied, and no compression implies ω = 0). Our results show that as long as the number of devices n is large (often true in distributed/federated learning), or the compression ω is not very high, CANITA achieves the faster convergence rate O L ǫ , i.e., the number of communication rounds is O L ǫ (vs. O L ǫ achieved by previous works). As a result, CANITA enjoys the advantages of both compression (compressed communication in each round) and acceleration (much fewer communication rounds).

show abstract

“…◇ Momentum. A very successful and popular technique for enhancing both optimization and generalization is momentum/acceleration (Polyak, 1964;Nesterov, 1983;Lan & Zhou, 2015;Allen-Zhu, 2017;Lan et al, 2019;Li, 2021a). For instance, momentum is a key building block behind the widely-used Adam method (Kingma & Ba, 2014).…”

Section: Our Contributionsmentioning

confidence: 99%

EF21 with Bells & Whistles: Practical Algorithmic Extensions of Modern Error Feedback

Fatkhullin,

Sokolov,

Gorbunov

et al. 2021

Preprint

View full text Add to dashboard Cite

First proposed by Seide et al. (2014) as a heuristic, error feedback (EF) is a very popular mechanism for enforcing convergence of distributed gradient-based optimization methods enhanced with communication compression strategies based on the application of contractive compression operators. However, existing theory of EF relies on very strong assumptions (e.g., bounded gradients), and provides pessimistic convergence rates (e.g., while the best known rate for EF in the smooth nonconvex regime, and when full gradients are compressed, is 𝑂(1/𝑇 2/3 ), the rate of gradient descent in the same regime is 𝑂(1/𝑇 )). Recently, Richtárik et al. ( 2021) (2021) proposed a new error feedback mechanism, EF21, based on the construction of a Markov compressor induced by a contractive compressor. EF21 removes the aforementioned theoretical deficiencies of EF and at the same time works better in practice. In this work we propose six practical extensions of EF21, all supported by strong convergence theory: partial participation, stochastic approximation, variance reduction, proximal setting, momentum and bidirectional compression. Several of these techniques were never analyzed in conjunction with EF before, and in cases where they were (e.g., bidirectional compression), our rates are vastly superior.* The work of Ilyas Fatkhullin was performed during a Summer research internship conducted in the Optimization and Machine Learning Lab led by Peter Richtárik. At the time when this paper was first released, Ilyas Fatkhullin was a master's student at the

show abstract

An optimal randomized incremental gradient method

Cited by 31 publications

References 17 publications

ANITA: An Optimal Loopless Accelerated Variance-Reduced Gradient Method

ANITA: An Optimal Loopless Accelerated Variance-Reduced Gradient Method

CANITA: Faster Rates for Distributed Convex Optimization with Communication Compression

EF21 with Bells & Whistles: Practical Algorithmic Extensions of Modern Error Feedback

Contact Info

Product

Resources

About