Hierarchical coded matrix multiplication

Abstract

In distributed computing systems slow working nodes, known as stragglers, can greatly extend finishing times. Coded computing is a technique that enables straggler-resistant computation. Most coded computing techniques presented to date provide robustness by ensuring that the time to finish depends only on a set of the fastest nodes. However, while stragglers do compute less work than non-stragglers, in real-world commercial cloud computing systems (e.g., Amazon's Elastic Compute Cloud (EC2)) the distinction is often a soft one. In this paper, we develop hierarchical coded computing that exploits the work completed by all nodes, both fast and slow, automatically integrating the potential contribution of each. We first present a conceptual framework to represent the division of work amongst nodes in coded matrix multiplication as a cuboid partitioning problem. This framework allows us to unify existing methods and motivates new techniques. We then develop three methods of hierarchical coded computing that we term bit-interleaved coded computation (BICC), multilevel coded computation (MLCC), and hybrid hierarchical coded computation (HHCC). In this paradigm, each worker is tasked with completing a sequence (a hierarchy) of ordered subtasks. The sequence of subtasks, and the complexity of each, is designed so that partial work completed by stragglers can be used in, rather than ignored. We note that our methods can be used in conjunction with any coded computing method. We illustrate this showing how we can use our methods to accelerate all previously developed coded computing technique by enabling them to exploit stragglers. Under a widely studied statistical model of completion time, our approach realizes a 66% improvement in expected finishing time. On Amazon EC2, the gain was 28% when stragglers are simulated.

Hierarchical Coded Elastic Computing

“…Worker n works on only 4 subtasks, denoted by A n,m B, where m ≡ (n + i − 1) mod 8 and i ∈ [4]. The cyclic allocation of CEC allows only 4 workers to contribute to the completion of each set {Â n,m } n∈[N ] , m ∈ [8]. In Figures 1b and 1c, it is shown how to continue the computation when workers are gradually preempted reducing the number of workers from N = 8 to N = 6 and to N = 4.…”

“…While coded computing goes beyond simple and traditional replication that can result extremely high redundancy, the computation overhead of many of the initial coded computing designs was not optimal. To reduce computation overhead, hierarchical coded computing was proposed in [5,6,7,8,9] that enables both fast and slow nodes to contribute to the output recovery. In hierarchical coding, the computation completed by stragglers is exploited rather than being ignored.…”

Bivariate Polynomial Coding for Straggler Exploitation with Heterogeneous Workers

“…However, due to unpredictable delays in their service time, some workers, called stragglers, may complete their assigned tasks much slower than the others, leading to serious delays. Mitigating the negative impact of stragglers on the completion time of the distributed matrix multiplication has recently been a very active research area [1]- [13].…”

“…In [8], a hierarchical coding framework for straggler exploitation problem is proposed concerning decoding time, which is the time spent to recover the main computation task from partial computations, in addition to the computation time. The work in [8] is extended to matrix-vector and matrix-matrix multiplications in [13]. For both type of multiplications, they numerically and experimentally show that, while gaining in terms of the decoding time, the computation time of hierarchical coding is only slightly larger than [10] with univariate polynomial coding.…”

“…In this work, we focus on the computation time in distributed matrix-matrix multiplication with an emphasis on the efficient use of storage capacities at workers. Similarly to [8]- [10], [12], [13], our main goal is to exploit partial computations carried out by stragglers. If the partial computations of workers are utilized and the workers are heterogeneous in their computational power and storage, the best we can do is to maximize the number of computations every worker can provide.…”