Storage, Computation, and Communication: A Fundamental Tradeoff in Distributed Computing

Abstract

Distributed computing has become one of the most important frameworks in dealing with large computation tasks. In this paper, we investigate a MapReduce like distributed computing system. Our main contribution is the characterization of the optimal tradeoff between storage space, computation load, and communication load. To this end, we derive an information-theoretical converse and show that time-and memory-sharing between the operating points achieved by the modified coded distributed computing (M-CDC) scheme proposed by Ezzeldin et al. attains all the points on this tradeoff. Our result thus extends the result by Li et al. on the optimum tradeoff between storage and communication to account also for the computation load. We further show how to obtain a distributed computing scheme from any placement delivery array (PDA) whose ordinary symbols occur at least twice. Previously proposed PDAs to solve the subpacketization problem in coded caching by Yan et al. and Shangguan et al. allow us then to derive optimal distributed computing schemes that require only a small number of files, and thus have reduced complexity.Recently, Li et al.[3] proposed a scheme, termed coded distributed computing (CDC), that in the map phase stores files multiple times across users so as to enable multicast opportunities for the shuffle phase. This approach can significantly reduce the communication load over traditional uncoded schemes as illustrated in Fig. 2, and was proved in [3] to have the smallest communication load among all the distributed computing schemes with same total storage requirements. It is worth mentioning that Li et al. in [3] used the term computation-communication tradeoff, because they assumed that each node calculates all the IVAs that can be obtained from the data stored at that node, irrespective of whether these IVAs are used in the sequel or not. In this sense, the total number of calculated IVAs is actually a measure of the total storage space consumed across the nodes. This is why we would rather refer to it as the storage-communication tradeoff. In this context, it is natural to investigate a more general framework, where each node is allowed to choose for each IVA that it can calculate from its locally stored data, whether or not to perform this calculation. The number of IVAs effectively calculated at all the nodes, normalized by the total number of IVAs, is then used to measure the real computation load. In this sense, we extend the storage-communication tradeoff in [3] to a storage-computation-communication tradeoff. That means, our goal is to identify all storage-computation-communication (SCC) triples, for which the task of computing the K output functions in (1) can be accomplished.August 28, 2018 DRAFT Fig. 2: Comparison of CDC scheme and uncoded scheme for a task of computing K = 10 output functions.In this work, we fully characterize all feasible SCC triples, and as a consequence, the pareto-optimal tradeoff surface. We show that any feasible triple on this optimal tradeoff surface can be attai...

Heterogeneous Coded Distributed Computing: Joint Design of File Allocation and Function Assignment

“…The optimal tradeoff between the computation load in the Map phase and the communication load in the Shuffle phase is derived in [3], which finds that increasing computation load of the Map phase by r can reduce communication load of the Shuffle phase by the same factor r. This idea of coded distributed computing has since been extended widely, e.g., [4]- [9]. In particular, [4], [5] propose new coded distributed computing schemes, [6] studies distributed computing with storage constraints at nodes, [7] studies distributed computing under time-varying excess computing resources, and [8], [9] studies the wireless distributed computing systems.…”

“…, φ Q } with |W k | = W k . Note that, unlike [3]- [6], [8]- [12], W k may vary for different k. Similar to [5], [6], [8]- [12], [14], we assume that W j ∩W k = ∅ for j = k so that each function is assigned to exactly one node. Thus, we have k∈[K] W k = Q.…”

Fundamental Limits of Decentralized Data Shuffling

“…A converse bound was proposed in [14] to show that the proposed coded distributed computing scheme is optimal in terms of communication load. This coded distributed computing framework was extended to the cases such as computing only necessary intermediate values [15], [16], reducing file partitions and number of output functions [16], [17], and considering random network topologies [18], stragglers [19], storage cost [20], and heterogeneous computing power, function assignment and storage space [21], [22].…”

“…We then add (39), (40), and (34) to obtain, 20 Hence, for each integer p ∈ [K], the bound in (42) becomes a linear function in M. When M = qp, from (42) we have R u ≥ N(K−p) (K−1)p . When M = q(p + 1), from (42) we have R u ≥ N(K−p−1) (K−1)(p+1) .…”

“…; the whole memory-load tradeoff curve is achieved by memory-sharing between these points (given in (20)) and the (trivially achievable) points in (21)- (22).…”