Towards Statistical and Computational Complexities of Polyak Step Size Gradient Descent

Ren, Tongzheng; Cui, Fuheng; Atsidakou, Alexia; Sanghavi, Sujay; Ho, Nhat

doi:10.48550/arxiv.2110.07810

Cited by 2 publications

(8 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The proof idea of Proposition 1 follows the proof argument from [28] and subsequently Lemma 4 in Appendix E of [33]; therefore, the proof is omitted.…”

Section: Optimization Rate For the Egd Algorithmmentioning

confidence: 99%

“…Under the high SNR regime, the population least-square function L is locally strongly convex and smooth (see Section 3.1 in [33]). Furthermore, there exist universal constants C 1 and C 2 such that with probability 1 − δ…”

Section: Generalized Linear Modelsmentioning

confidence: 99%

“…where X ∼ 1 2 N (X| − θ * , σ 2 I d ) + 1 2 N (X| θ * , σ 2 I d ) and θ * = 0. Indeed, from Appendix A.1 in [33], the homogeneous Assumption (W.1) is satisfied when α = 2 and ρ = σ/2, namely, we have…”

Section: Gaussian Mixture Modelsmentioning

confidence: 99%

See 2 more Smart Citations

An Exponentially Increasing Step-size for Parameter Estimation in Statistical Models

Ho¹,

Ren²,

Sanghavi³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Using gradient descent (GD) with fixed or decaying step-size is standard practice in unconstrained optimization problems. However, when the loss function is only locally convex, such a step-size schedule artificially slows GD down as it cannot explore the flat curvature of the loss function. To overcome that issue, we propose to exponentially increase the step-size of the GD algorithm. Under homogeneous assumptions on the loss function, we demonstrate that the iterates of the proposed exponential step size gradient descent (EGD) algorithm converge linearly to the optimal solution. Leveraging that optimization insight, we then consider using the EGD algorithm for solving parameter estimation under non-regular statistical models whose the loss function becomes locally convex when the sample size goes to infinity. We demonstrate that the EGD iterates reach the final statistical radius within the true parameter after a logarithmic number of iterations, which is in stark contrast to a polynomial number of iterations of the GD algorithm. Therefore, the total computational complexity of the EGD algorithm is optimal and exponentially cheaper than that of the GD for solving parameter estimation in nonregular statistical models. To the best of our knowledge, it resolves a long-standing gap between statistical and algorithmic computational complexities of parameter estimation in non-regular statistical models. Finally, we provide targeted applications of the general theory to several classes of statistical models, including generalized linear models with polynomial link functions and location Gaussian mixture models.

show abstract

“…The proof idea of Proposition 1 follows the proof argument from [28] and subsequently Lemma 4 in Appendix E of [33]; therefore, the proof is omitted.…”

Section: Optimization Rate For the Egd Algorithmmentioning

confidence: 99%

Section: Generalized Linear Modelsmentioning

confidence: 99%

See 1 more Smart Citation

An Exponentially Increasing Step-size for Parameter Estimation in Statistical Models

Ho¹,

Ren²,

Sanghavi³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…In addition, our methodology focuses on the situation when the loss function is globally or locally strong convex. How to generalize our theoretical results to more general locally convex settings (Ho et al, 2020;Ren et al, 2022) remains to be a challenging but also an interesting topic for future study.…”

Section: Discussionmentioning

confidence: 99%

“…It is remarkable that Corollary 1 assumed locally strong convexity. To deal with more general locally convex settings, a novel method has been developed by Ho et al (2020) and Ren et al (2022) for studying the estimation error and the computational complexity of algorithm-based estimators. The key idea is to construct an interesting reference estimator sequence for the actual algorithm-based estimator sequence.…”

Section: General Loss Functionsmentioning

confidence: 99%

Network Gradient Descent Algorithm for Decentralized Federated Learning

Wu¹,

Huang²,

Wang³

2022

Preprint

View full text Add to dashboard Cite

We study a fully decentralized federated learning algorithm, which is a novel gradient descent algorithm executed on a communication-based network. For convenience, we refer to it as a network gradient descent (NGD) method. In the NGD method, only statistics (e.g., parameter estimates) need to be communicated, minimizing the risk of privacy. Meanwhile, different clients communicate with each other directly according to a carefully designed network structure without a central master. This greatly enhances the reliability of the entire algorithm. Those nice properties inspire us to carefully study the NGD method both theoretically and numerically. Theoretically, we start with a classical linear regression model. We find that both the learning rate and the network structure play significant roles in determining the NGD estimator's statistical efficiency. The resulting NGD estimator can be statistically as efficient as the global estimator, if the learning rate is sufficiently small and the network structure is well balanced, even if the data are distributed heterogeneously. Those interesting findings are then extended to general models and loss functions. Extensive numerical studies are presented to corroborate our theoretical findings. Classical deep learning models are also presented for illustration purpose.

show abstract

Towards Statistical and Computational Complexities of Polyak Step Size Gradient Descent

Cited by 2 publications

References 26 publications

An Exponentially Increasing Step-size for Parameter Estimation in Statistical Models

An Exponentially Increasing Step-size for Parameter Estimation in Statistical Models

Network Gradient Descent Algorithm for Decentralized Federated Learning

Contact Info

Product

Resources

About