Optimal Complexity in Non-Convex Decentralized Learning over Time-Varying Networks

Huang, Xinmeng; Yuan, Kun

doi:10.48550/arxiv.2211.00533

Cited by 1 publication

(1 citation statement)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…3, by enhancing the vanilla Push-DIGing. Inspired by the algorithm development in [36,37], we add two additional components to Push-DIGing: gradient accumulation and multiple-gossip communication. We call the new algorithm as MG-Push-DIGing where "MG" indicates "multiple gossips".…”

Section: Mg-push-diging Algorithmmentioning

confidence: 99%

Chinese public opinions on death penalty: measurement, analysis and communication

Liang

Chen

2020

Peking University Law Journal

View full text Add to dashboard Cite

This paper investigates the influence of directed networks on the convergence of smooth non-convex stochastic decentralized optimization associated with column-stochastic mixing matrices. We find that the canonical spectral gap, a widely-used metric in undirected networks, alone fails to adequately characterize the impact of directed networks. Through a new analysis of the Push-Sum strategy, a fundamental building block for decentralized algorithms over directed graphs, we identify another novel metric called the equilibrium skewness. Next, we establish the first convergence lower bound for non-convex stochastic decentralized algorithms over directed networks, which explicitly manifests the impact of both the spectral gap and equilibrium skewness and justifies the imperative need for both metrics in analysis.Moreover, by jointly considering the spectral gap and equilibrium skewness, we present the state-of-theart convergence rate for the Push-DIGing algorithm. Our findings reveal that these two metrics exert a more pronounced negative impact on Push-DIGing than suggested by our lower bound. We further integrate the technique of multi-round gossip to Push-DIGing to obtain MG-Push-DIGing, which nearly achieves the established lower bound, demonstrating its convergence optimality, best-possible resilience to directed networks, and the tightness of our lower bound. Experiments verify our theoretical findings.

show abstract

Section: Mg-push-diging Algorithmmentioning

confidence: 99%