Gossip Learning: Off the Beaten Path

Abstract

The growing computational demands of model training tasks and the increased privacy awareness of consumers call for the development of new techniques in the area of machine learning. Fully decentralized approaches have been proposed, but are still in early research stages. This study analyses gossip learning, one of these state-of-the-art decentralized machine learning protocols, which promises high scalability and privacy preservation, with the goal of assessing its applicability to realworld scenarios.Previous research on gossip learning presents strong and often unrealistic assumptions on the distribution of the data, the communication speeds of the devices and the connectivity among them. Our results show that lifting these requirements can, in certain scenarios, lead to slow convergence of the protocol or even unfair bias in the produced models. This paper identifies the conditions in which gossip learning can and cannot be applied, and introduces extensions that mitigate some of its limitations.

Distributed and Multi-Task Learning at the Edge for Energy Efficient Radio Access Networks

“…However, this may not be the case in many scenarios, where a single node might have multiple useful data points, such as image classification and our case. To this respect, the study in [44] shows that training on multiple data points provides a clear advantage over the original protocol, as models see more data in the same number of iterations, and thus converge faster. Therefore, we consider the extension proposed in [44], which propose to call multiple times the SGD on different data points, i.e., implementing at each node multiple training steps sequentially every loop cycle.…”

“…To this respect, the study in [44] shows that training on multiple data points provides a clear advantage over the original protocol, as models see more data in the same number of iterations, and thus converge faster. Therefore, we consider the extension proposed in [44], which propose to call multiple times the SGD on different data points, i.e., implementing at each node multiple training steps sequentially every loop cycle. In particular, we consider to perform the local training on the whole dataset generated by one BS in one loop cycle, thus the maximum number of steps of the algorithm will be the number of BSs.…”

Implicit model specialization through dag-based decentralized federated learning

“…Compared to our approach, specialization for gossip learning can be reliant on the availability of related peers in the network which could slow down convergence speed [18]. Moreover, many gossip learning algorithms do not consider robustness against poisoning attacks.…”

PDS2: A user-centered decentralized marketplace for privacy preserving data processing

“…Unfortunately, while gossip learning has been shown to be applicable to many different ML workloads [22][23][24], no study that we are aware of tested it in actual physical environments or evaluated its practical use for large-scale deep learning training. However, recent studies suggest that gossip learning compares favorably to federated learning [25] and that it can be extended to work in constrained and highly heterogeneous environments [26].…”