Straggler-Resilient Distributed Machine Learning with Dynamic Backup Workers

Abstract

With the increasing demand for large-scale training of machine learning models, consensus-based distributed optimization methods have recently been advocated as alternatives to the popular parameter server framework. In this paradigm, each worker maintains a local estimate of the optimal parameter vector, and iteratively updates it by waiting and averaging all estimates obtained from its neighbors, and then corrects it on the basis of its local dataset. However, the synchronization phase can be time consuming due to the need to wait for stragglers, i.e., slower workers. An efficient way to mitigate this effect is to let each worker wait only for updates from the fastest neighbors before updating its local parameter. The remaining neighbors are called backup workers. To minimize the globally training time over the network, we propose a fully distributed algorithm to dynamically determine the number of backup workers for each worker. We show that our algorithm achieves a linear speedup for convergence (i.e., convergence performance increases linearly with respect to the number of workers). We conduct extensive experiments on MNIST and CIFAR-10 to verify our theoretical results. * Equal contribution. Ordering determined by alphabetical order. 2 Each worker needs to receive the aggregate of updates from all other workers to move to the next iteration, where aggregation is performed either by PS or along the ring through multiple rounds.

Seizing Critical Learning Periods in Federated Learning

“…Since the term of federated learning was introduced in the seminal work , there is an explosive growth in federated learning research. For example, a line of works focuses on designing algorithms to achieve higher learning accuracy and analyze their convergence performance, e.g., (Smith et al 2017;Li et al 2020b;Liu et al 2020;Wang et al 2020b;Xiong, Yan, and Li 2021). Another line of works aim to improve the communication efficiency between the central server and clients through compressions or sparsification, (Konečnỳ et al 2016;Suresh et al 2017;Xu et al 2019), communication frequency optimization (Wang and Joshi 2019;Karimireddy et al 2020), client selections (Lai et al 2021;Wang et al 2020a), etc.…”

“…Unlike traditional centralized machine learning, data samples of each client in FL follow a non-identical and independent distribution (non-IID), introducing bias that slows down or even fails the training. A few recent studies have been proposed to address these challenges by model compression (Konečnỳ et al 2016;Suresh et al 2017), communication frequency optimization (Wang and Joshi 2019;Karimireddy et al 2020), and client selections (Lai et al 2021;Wang et al 2020a;Xiong, Yan, and Li 2021).…”

Building and Using Multiple Stacks of Models for the Classification of Learners and Custom Recommending of Quizzes

“…The usage of many trained models has been addressed by [13] in a distributed context. The proposed architectural approach involves setting up a dynamic set of backup workers and a custom-designed consensus algorithm showing linear speedup for experiments' convergence.…”