Semi-Decentralized Federated Learning With Cooperative D2D Local Model Aggregations

Lin, Frank Po-Chen; Hosseinalipour, Seyyedali; Azam, Sheikh Shams; Brinton, Christopher G.; Michelusi, Nicolò

doi:10.1109/jsac.2021.3118344

Cited by 77 publications

(31 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This migrates away from the star topology of FedL and paves the road to more decentralized distributed ML architectures. This approach is complementary to recent works that utilize D2D in distributed ML to conduct model consensus [17]- [20]. II.…”

Section: Summary Of Contributionsmentioning

confidence: 97%

“…Additionally, there is a literature on fully decentralized FedL over mesh network architectures without a centralized server, where device-server communications are replaced with device-to-device (D2D) communications [17], [18]. Building upon this, semi-decentralized architectures for FedL have also been proposed, where D2D communications are exploited in conjunction with device-server interactions to improve the model training performance [19], [20]. In this literature, D2D communications are solely used for distributed model parameter aggregation across the nodes.…”

Section: A Related Workmentioning

confidence: 99%

See 1 more Smart Citation

From Federated to Fog Learning: Distributed Machine Learning over Heterogeneous Wireless Networks

et al. 2020

Self Cite

View full text Add to dashboard Cite

Federated learning (FedL) has emerged as a popular technique for distributing model training over a set of wireless devices, via iterative local updates (at devices) and global aggregations (at the server). In this paper, we develop parallel successive learning (PSL), which expands the FedL architecture along three dimensions: (i) Network, allowing decentralized cooperation among the devices via device-to-device (D2D) communications. (ii) Heterogeneity, interpreted at three levels: (ii-a) Learning: PSL considers heterogeneous number of stochastic gradient descent iterations with different mini-batch sizes at the devices; (ii-b) Data: PSL presumes a dynamic environment with data arrival and departure, where the distributions of local datasets evolve over time, captured via a new metric for model/concept drift. (ii-c) Device: PSL considers devices with different computation and communication capabilities. (iii) Proximity, where devices have different distances to each other and the access point. PSL considers the realistic scenario where global aggregations are conducted with idle times in-between them for resource efficiency improvements, and incorporates data dispersion and model dispersion with local model condensation into FedL. Our analysis sheds light on the notion of cold vs. warmed up models, and model inertia in distributed machine learning. We then propose networkaware dynamic model tracking to optimize the model learning vs. resource efficiency tradeoff, which we show is an NP-hard signomial programming problem. We finally solve this problem through proposing a general optimization solver. Our numerical results reveal new findings on the interdependencies between the idle times in-between the global aggregations, model/concept drift, and D2D cooperation configuration.

show abstract

Section: Summary Of Contributionsmentioning

confidence: 97%

Section: A Related Workmentioning

confidence: 99%

From Federated to Fog Learning: Distributed Machine Learning over Heterogeneous Wireless Networks

et al. 2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…Although HFL has certain advantages, this framework requires employing multiple PSs that may not be practical in certain scenarios. Instead, the idea of hierarchical collaborative learning can be redesigned to combine hierarchical and decentralized learning concepts, which is referred to as semi-decentralized FL, where the local consensus follows decentralized learning with D2D communications, whereas the global consensus is orchestrated by the PS [43,44]. One of the major challenges in FL that is not considered in the aforementioned works on semi-decentralized FL is the partial client connectivity [45,46].…”

Section: Related Workmentioning

confidence: 99%

Robust Federated Learning with Connectivity Failures: A Semi-Decentralized Framework with Collaborative Relaying

Yemini¹,

Saha²,

Ozfatura³

et al. 2022

Preprint

View full text Add to dashboard Cite

Intermittent client connectivity is one of the major challenges in centralized federated edge learning frameworks. Intermittently failing uplinks to the central parameter server (PS) can induce a large generalization gap in performance especially when the data distribution among the clients exhibits heterogeneity. In this work, to mitigate communication blockages between clients and the central PS, we introduce the concept of knowledge relaying wherein the successfully participating clients collaborate in relaying their neighbors' local updates to a central parameter server (PS) in order to boost the participation of clients with intermittently failing connectivity. We propose a collaborative relaying based semi-decentralized federated edge learning framework where at every communication round each client first computes a local consensus of the updates from its neighboring clients and eventually transmits a weighted average of its own update and those of its neighbors to the PS. We appropriately optimize these averaging weights to reduce the variance of the global update at the PS while ensuring that the global update is unbiased, consequently improving the convergence rate. Finally, by conducting experiments on CIFAR-10 dataset we validate our theoretical results and demonstrate that our proposed scheme is superior to Federated averaging benchmark especially when data distribution among clients is non-iid.* Michal Yemini and Andrea J. Goldsmith are with the Faculty of Electrical and Computer Engineering, Princeton University.

show abstract

“…Distributed ML over wireless networks: Recent literature concerning ML by wireless networks has shifted towards federated learning [23], [24], and is mostly focused on studying the convergence and behavior of federated learning over wireless networks [9], [10], [12], [25], [26], [27], [28], [29], [30], [31], [32], [33]. Conventional federated learning assumes training a single ML model for all the engaged devices.…”

Section: Related Workmentioning

confidence: 99%

“…Constraint (32) ensures that the total energy consumed by each worker UAV j for data processing, parameter transmission and flying is less than E Ba j (s) − E Th j , where E Ba j (s) represents the battery energy at j at the start of the s-th training sequence and E Th j encompasses both (i) surplus idle energy needed for extra hovering time caused by potential asynchronocity due to the heterogeneity of leader UAV to AP travel times, and (ii) the minimum energy threshold for j to reach the nearest recharging station after the conclusion of the training sequence. Constraint (33) imposed on the coordinator UAVs is similar to (32), except that coordinator UAVs only conduct data transmission while flying. Constraint (34) imposed on the leader UAVs guarantees that there is enough energy remaining after parameter broadcasting and flying to reach the nearest recharging station.…”

Section: Joint Energy and Performance Optimizationmentioning

confidence: 99%

UAV-assisted Online Machine Learning over Multi-Tiered Networks: A Hierarchical Nested Personalized Federated Learning Approach

Wang¹,

Hosseinalipour²,

Gorlatova³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

We consider distributed machine learning (ML) through unmanned aerial vehicles (UAVs) for geo-distributed device clusters. We propose five new technologies/techniques: (i) stratified UAV swarms with leader, worker, and coordinator UAVs, (ii) hierarchical nested personalized federated learning (HN-PFL): a holistic distributed ML framework for personalized model training across the worker-leader-core network hierarchy, (iii) cooperative UAV resource pooling for distributed ML using the UAVs' local computational capabilities, (iv) aerial data caching and relaying for efficient data relaying to conduct ML, and (v) concept/model drift, capturing online data variations at the devices. We split the UAV-enabled model training problem as two parts. (a) Network-aware HN-PFL, where we optimize a tradeoff between energy consumption and ML model performance by configuring data offloading among devices-UAVs and UAV-UAVs, UAVs' CPU frequencies, and mini-batch sizes subject to communication/computation network heterogeneity. We tackle this optimization problem via the method of posynomial condensation and propose a distributed algorithm with a performance guarantee. (b) Macro-trajectory and learning duration design, which we formulate as a sequential decision making problem, tackled via deep reinforcement learning. Our simulations demonstrate the superiority of our methodology with regards to the distributed ML performance, the optimization of network resources, and the swarm trajectory efficiency.

show abstract

Semi-Decentralized Federated Learning With Cooperative D2D Local Model Aggregations

Cited by 77 publications

References 42 publications

From Federated to Fog Learning: Distributed Machine Learning over Heterogeneous Wireless Networks

From Federated to Fog Learning: Distributed Machine Learning over Heterogeneous Wireless Networks

Robust Federated Learning with Connectivity Failures: A Semi-Decentralized Framework with Collaborative Relaying

UAV-assisted Online Machine Learning over Multi-Tiered Networks: A Hierarchical Nested Personalized Federated Learning Approach

Contact Info

Product

Resources

About