Demystifying Why Local Aggregation Helps: Convergence Analysis of Hierarchical SGD

Abstract

Hierarchical SGD (H-SGD) has emerged as a new distributed SGD algorithm for multi-level communication networks. In H-SGD, before each global aggregation, workers send their updated local models to local servers for aggregations. Despite recent research efforts, the effect of local aggregation on global convergence still lacks theoretical understanding. In this work, we first introduce a new notion of "upward" and "downward" divergences. We then use it to conduct a novel analysis to obtain a worst-case convergence upper bound for two-level H-SGD with non-IID data, non-convex objective function, and stochastic gradient. By extending this result to the case with random grouping, we observe that this convergence upper bound of H-SGD is between the upper bounds of two single-level local SGD settings, with the number of local iterations equal to the local and global update periods in H-SGD, respectively. We refer to this as the "sandwich behavior". Furthermore, we extend our analytical approach based on "upward" and "downward" divergences to study the convergence for the general case of H-SGD with more than two levels, where the "sandwich behavior" still holds. Our theoretical results provide key insights of why local aggregation can be beneficial in improving the convergence of H-SGD.

Incentive Mechanism Design for Joint Resource Allocation in Blockchain-Based Federated Learning

“…In FL, the participants work together to solve a finite-sum optimization problem with SGD, while in hierarchical FL (HFL), the hierarchical SGD (H-SGD) is adopted [7]. The main difference between SGD and H-SGD is that H-SGD requires several rounds of intermediate aggregation before global aggregation.…”

“…Here we introduce two assumptions that are important for the proof of convergence. The first one indicates the property of the loss function employed in our proposed BHFL framework, which has also been widely included in the existing studies [7], [17], [29]. The second ensures that the model updating process will not lead to a significant bias.…”

“…Liu et al [8] propose a client-edge-cloud HFL framework running with the Hier-FAVG aggregation algorithm and demonstrate that commu- nication efficiency can be improved by introducing the hierarchical architecture in FL. Wang et al [7] provide theoretical analysis about the convergence of HFL based on Stochastic Gradient Decent (SGD) and emphasize the importance of local aggregation before global aggregation. In [6], the focus is on protecting participants' privacy in HFL with flexible and decentralized control.…”

“…Hierarchical federated learning (HFL) provides a promising solution to the above challenge [4]- [7]. The basic idea is to conduct multiple intermediate aggregations at proxy servers (e.g., edge servers) before global aggregation on the central server.…”

A Hessian-Based Federated Learning Approach to Tackle Statistical Heterogeneity

“…The authors are with the School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology, Stockholm, Sweden (emails: seyaa,vjfodor @kth.se). arXiv:2403.01540v1 [cs.LG] 3 Mar 2024 aggregated at both the edge servers and the cloud server for example in [6], [21], [23], while gradient aggregation is applied on both levels in [7], [8]. The mix of gradient and model parameter aggregation is proposed in [19], [20], where gradient aggregation is performed at the intra-set iterations and model aggregation at the inter-set iterations.…”