Near Optimal Coded Data Shuffling for Distributed Learning

Abstract

Data shuffling between distributed cluster of nodes is one of the critical steps in implementing large-scale learning algorithms. Randomly shuffling the data-set among a cluster of workers allows different nodes to obtain fresh data assignments at each learning epoch. This process has been shown to provide improvements in the learning process. However, the statistical benefits of distributed data shuffling come at the cost of extra communication overhead from the master node to worker nodes, and can act as one of the major bottlenecks in the overall time for computation. There has been significant recent interest in devising approaches to minimize this communication overhead. One approach is to provision for extra storage at the computing nodes. The other emerging approach is to leverage coded communication to minimize the overall communication overhead.The focus of this work is to understand the fundamental trade-off between the amount of storage and the communication overhead for distributed data shuffling. In this work, we first present an information theoretic formulation for the data shuffling problem, accounting for the underlying problem parameters (number of workers, K, number of data points, N , and the available storage, S per node). We then present an information theoretic lower bound on the communication overhead for data shuffling as a function of these parameters. We next present a novel coded communication scheme and show that the resulting communication overhead of the proposed scheme is within a multiplicative factor of at most K K−1 from the informationtheoretic lower bound. Furthermore, we present the aligned coded shuffling scheme for some storage values, which achieves the optimal storage vs communication trade-off for K < 5, and further reduces the maximum multiplicative gap down topresence of stragglers was proposed in [20] for synchronous gradient descent, and [21-23] for linear computation tasks, e.g., matrix multiplication. The use of Polynomial codes for high dimensional coded matrix multiplication was proposed in [24]. Coded computation over wireless networks was proposed in [25], where only one worker can transmit at a time. The use of codes to reduce the communication overhead due to data shuffling was considered in [26][27][28][29][30][31][32][33][34][35]. In [26][27][28][29], the authors considered the MapReduce setting, where in order to reduce the communication between the mappers and the reducers, coding opportunities are created with more redundant computations at the mappers, leading to a trade-off between communication and computation. [30,31] provided a unified coding framework for distributed computing, where the communication load due to shuffling can be alleviated by trading the computational complexity in the presence of straggling servers. The information theoretic limits for data shuffling in the wired master-worker setting was considered in [19,32,33]. Coded data shuffling in wireless setting was recently considered in [18,34,35] for both centralized and decentralized...

Performance enhancement for iterative data computing with in‐memory concurrent processing

“…Attia and Tandon presented a theoretic lower bound on the communication overhead for data shuffling as a function of the number of workers, number of data points, and the available storage per node. They proposed a coded communication scheme to show that the communication overhead is within a multiplicative factor according to the theoretic lower bound . Elmahdy and Mohajer considered the data shuffling problem that a master node communicates a set of files to a set of worker nodes through a shared link.…”

“…They proposed a coded communication scheme to show that the communication overhead is within a multiplicative factor according to the theoretic lower bound. 30 Elmahdy and Mohajer considered the data shuffling problem that a master node communicates a set of files to a set of worker nodes through a shared link. They proposed a deterministic and systematic coded shuffling scheme to find out the exact rate of cache files.…”

Fundamental Limits of Decentralized Data Shuffling

“…Inspired by the achievable and converse bounds for the single-bottleneck-link caching problem in [8]-[10], the authors in [11] then proposed a general coded data shuffling scheme, which was shown to be order optimality to within a factor of 2 under the constraint of uncoded storage. Also in [11], the authors improved the performance of the general coded shuffling scheme by introducing an aligned coded delivery, which was shown to be optimal under the constraint of uncoded storageRecently, inspired by the improved data shuffling scheme in [11], the authors in [12] proposed a linear coding scheme based on interference alignment, which achieves the optimal worstcase communication load under the constraint of uncoded storage for all system parameters. In addition, under the constraint of uncoded storage, the proposed coded data shuffling scheme in [12] was shown to be optimal for any shuffles (not just for the worst-case) when q = 1.…”

“…Inspired by the achievable and converse bounds for the single-bottleneck-link caching problem in [8]-[10], the authors in [11] then proposed a general coded data shuffling scheme, which was shown to be order optimality to within a factor of 2 under the constraint of uncoded storage. Also in [11], the authors improved the performance of the general coded shuffling scheme by introducing an aligned coded delivery, which was shown to be optimal under the constraint of uncoded storage…”

“…Recently, inspired by the improved data shuffling scheme in [11], the authors in [12] proposed a linear coding scheme based on interference alignment, which achieves the optimal worstcase communication load under the constraint of uncoded storage for all system parameters. In addition, under the constraint of uncoded storage, the proposed coded data shuffling scheme in [12] was shown to be optimal for any shuffles (not just for the worst-case) when q = 1.…”