Low precision decentralized distributed training over IID and non-IID data

Aketi, Sai Aparna; Kodge, Sangamesh; Roy, Kaushik

doi:10.1016/j.neunet.2022.08.032

Cited by 3 publications

(2 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A recent proposition, the Global Update Tracking (GUT) method, addresses data heterogeneity in decentralized learning efficiently, without incurring extra communication costs. Empirical results validate that GUT not only accommodates variances in data distribution across devices but also bolsters the performance of decentralized learning processes [17].…”

Section: B Literature Overviewmentioning

confidence: 60%

On the Convergence of Decentralized Federated Learning Under Imperfect Information Sharing

Chellapandi

Upadhyay

Hashemi

et al. 2023

IEEE Control Syst. Lett.

View full text Add to dashboard Cite

A novel Decentralized Noisy Model Update Tracking Federated Learning algorithm (FedNMUT) is proposed that is tailored to function efficiently in the presence of noisy communication channels that reflect imperfect information exchange. This algorithm uses gradient tracking to minimize the impact of data heterogeneity while minimizing communication overhead. The proposed algorithm incorporates noise into its parameters to mimic the conditions of noisy communication channels, thereby enabling consensus among clients through a communication graph topology in such challenging environments. FedNMUT prioritizes parameter sharing and noise incorporation to increase the resilience of decentralized learning systems against noisy communications. Theoretical results for the smooth non-convex objective function are provided by us, and it is shown that the ϵ−stationary solution is achieved by our algorithm at the rate of O 1 √ T , where T is the total number of communication rounds. Additionally, via empirical validation, we demonstrated that the performance of FedNMUT is superior to the existing state-of-the-art methods and conventional parameter-mixing approaches in dealing with imperfect information sharing. This proves the capability of the proposed algorithm to counteract the negative effects of communication noise in a decentralized learning framework.Imperfect information exchange, such as noisy or quantized communication, has been examined in the context of average consensus algorithms within distributed frameworks. Yet, the ramifications of varying noise levels remain underexplored. Moreover, existing research, primarily focused on consensus issues, does not fully address the complex challenges encountered in contemporary decentralized optimization and learning paradigms [23], [24]. In contrast to Federated Learning (FL), where server assistance is common, Decentralized Federated Learning (DFL) operates without a central server, with each client acting autonomously, processing local Stochastic Gradient Descent (SGD) or its variants on its data and interacting directly with neighboring clients.In our previous paper [25], we performed a comparative study of three proposed algorithms for DFL under imperfect communication conditions, typified by noisy channels. These algorithms-FedNDL1, FedNDL2, and FedNDL3-differ in their handling of noise and parameter sharing, demonstrating varying degrees of resilience to communication noise. In this paper, we propose a novel algorithm that employs the Gradient Tracking method in DFL and compare its performance against the previously mentioned algorithms. C. Paper's ContributionsThis paper introduces a novel algorithm that employs the Gradient Tracking method in DFL, considering the impact of communication noise. Previous studies have evaluated the effectiveness of two-time scale methods in DFL with noisy channels. However, these investigations were limited by inflexible assumptions such as strong convexity in papers such as [26]- [29]. These assumptions are rarely satisfied in practi...

show abstract

Section: B Literature Overviewmentioning

confidence: 60%

On the Convergence of Decentralized Federated Learning Under Imperfect Information Sharing

Chellapandi

Upadhyay

Hashemi

et al. 2023

IEEE Control Syst. Lett.

View full text Add to dashboard Cite

show abstract

“…Data heterogeneity in federated learning is characterized by differences in data distribution and quantity, which can vary significantly due to environmental preferences and usage patterns [22,23]. However, current research mainly focuses on the non-IID problem in single-mode datasets, where each participant's dataset only contains one type of data mode, such as pictures, text, audio, or video [24].…”

Section: Data Heterogeneitymentioning

confidence: 99%

A Hierarchical Federated Learning Algorithm Based on Time Aggregation in Edge Computing Environment

Zhang

Zhao

et al. 2023

Applied Sciences

View full text Add to dashboard Cite

Federated learning is currently a popular distributed machine learning solution that often experiences cumbersome communication processes and challenging model convergence in practical edge deployments due to the training nature of its model information interactions. The paper proposes a hierarchical federated learning algorithm called FedDyn to address these challenges. FedDyn uses dynamic weighting to limit the negative effects of local model parameters with high dispersion and speed-up convergence. Additionally, an efficient aggregation-based hierarchical federated learning algorithm is proposed to improve training efficiency. The waiting time is set at the edge layer, enabling edge aggregation within a specified time, while the central server waits for the arrival of all edge aggregation models before integrating them. Dynamic grouping weighted aggregation is implemented during aggregation based on the average obsolescence of local models in various batches. The proposed algorithm is tested on the MNIST and CIFAR-10 datasets and compared with the FedAVG algorithm. The results show that FedDyn can reduce the negative effects of non-independent and identically distributed (IID) data on the model and shorten the total training time by 30% under the same accuracy rate compared to FedAVG.

show abstract

Towards consensual representation: Model-agnostic knowledge extraction for dual heterogeneous federated fault diagnosis

Wang,

Song,

Zhao

2024

Neural Networks

View full text Add to dashboard Cite

Low precision decentralized distributed training over IID and non-IID data

Cited by 3 publications

References 5 publications

On the Convergence of Decentralized Federated Learning Under Imperfect Information Sharing

On the Convergence of Decentralized Federated Learning Under Imperfect Information Sharing

A Hierarchical Federated Learning Algorithm Based on Time Aggregation in Edge Computing Environment

Towards consensual representation: Model-agnostic knowledge extraction for dual heterogeneous federated fault diagnosis

Contact Info

Product

Resources

About