From Federated to Fog Learning: Distributed Machine Learning over Heterogeneous Wireless Networks

Abstract

Federated learning (FedL) has emerged as a popular technique for distributing model training over a set of wireless devices, via iterative local updates (at devices) and global aggregations (at the server). In this paper, we develop parallel successive learning (PSL), which expands the FedL architecture along three dimensions: (i) Network, allowing decentralized cooperation among the devices via device-to-device (D2D) communications. (ii) Heterogeneity, interpreted at three levels: (ii-a) Learning: PSL considers heterogeneous number of stochastic gradient descent iterations with different mini-batch sizes at the devices; (ii-b) Data: PSL presumes a dynamic environment with data arrival and departure, where the distributions of local datasets evolve over time, captured via a new metric for model/concept drift. (ii-c) Device: PSL considers devices with different computation and communication capabilities. (iii) Proximity, where devices have different distances to each other and the access point. PSL considers the realistic scenario where global aggregations are conducted with idle times in-between them for resource efficiency improvements, and incorporates data dispersion and model dispersion with local model condensation into FedL. Our analysis sheds light on the notion of cold vs. warmed up models, and model inertia in distributed machine learning. We then propose networkaware dynamic model tracking to optimize the model learning vs. resource efficiency tradeoff, which we show is an NP-hard signomial programming problem. We finally solve this problem through proposing a general optimization solver. Our numerical results reveal new findings on the interdependencies between the idle times in-between the global aggregations, model/concept drift, and D2D cooperation configuration.

Reliability Optimization in Narrowband Device-to-Device Communication for 5G and Beyond-5G Networks

“…is considered to have prime role to support fully autonomous B5G and 6G network, which are well described in [30]. Therefore, investigating multi-agent RL (MARL) environment, where multiple devices interact with each-other and share the already learned parameter using federated learning is an open issue, and future research should study MARL scenarios.…”

AI-Aided Integrated Terrestrial and Non-Terrestrial 6G Solutions for Sustainable Maritime Networking

“…As the adoption of AI technologies accelerates, the integration of various monitoring and control systems within a centralized cloud can limit the scalability in such systems. Hence, today's predominantly cloud-centric AI solutions that rely on training and inference in the remote cloud have to be complemented by more energy-efficient, partially distributed, and ultimately fully distributed learning mechanisms where numerous devices collaboratively train a part of a global model [14].…”

“…Conventionally, DNN algorithms are executed in the cloud where training data are preprocessed at the edge before being transferred to the cloud [14]. The edge/fog computing infrastructures are intended to accommodate the needs of multiple DNN models that require locality and persistent training.…”

Resource-Efficient and Delay-Aware Federated Learning Design under Edge Heterogeneity

“…Implementations of FL over the wireless edge are affected by heterogeneity in communication and computation capabilities across the devices [6]. To improve communication efficiency, several works have focused on reducing the number of uplink/downlink communication rounds by performing multiple iterations of local model updates between consecutive global aggregations [7], [8].…”