Rateless codes for near-perfect load balancing in distributed matrix-vector multiplication

Abstract

Large-scale machine learning and data mining applications require computer systems to perform massive matrix-vector and matrix-matrix multiplication operations that need to be parallelized across multiple nodes. The presence of straggling nodes---computing nodes that unpredictably slow down or fail---is a major bottleneck in such distributed computations. Ideal load balancing strategies that dynamically allocate more tasks to faster nodes require knowledge or monitoring of node speeds as well as the ability to quickly move data. Recently proposed fixed-rate erasure coding strategies can handle unpredictable node slowdown, but they ignore partial work done by straggling nodes, thus resulting in a lot of redundant computation. We propose a rateless fountain coding strategy that achieves the best of both worlds---we prove that its latency is asymptotically equal to ideal load balancing, and it performs asymptotically zero redundant computations. Our idea is to create linear combinations of the m rows of the matrix and assign these encoded rows to different worker nodes. The original matrix-vector product can be decoded as soon as slightly more than m row-vector products are collectively finished by the nodes. Evaluation on parallel and distributed computing yields as much as three times speedup over uncoded schemes.

Network Coding Approaches for Distributed Computation over Lossy Wireless Networks

“…Among the first four schemes, LT + IR achieved the best performance for all four scenarios. Note that IR eliminates the packet-loss issue, and this result has also been demonstrated in [ 14 ], where only the straggler issue was considered. This is because LT codes can achieve near-perfect load balance among the worker nodes in the presence of stragglers.…”

“…As in [ 14 ], we consider a computation delay model as follows. The computation delay at each worker node consists of two parts.…”

“…LT code [ 14 ], where the 1000 original sub-matrices were encoded using LT codes, and an infinite number of coded sub-matrices was assigned to each worker node.…”

“…Ferdinand et al [ 12 ] and Kiani et al [ 13 ] proposed approaches to make use of stragglers by allocating more fine-grained computing tasks to each worker. Very recently, Mallick et al [ 14 ] proposed the use of rateless codes such as LT codes [ 15 ] and Raptor codes [ 16 ] and demonstrated that a rateless coding approach can achieve an asymptotically optimal latency. However, all these approaches assumed that the communication between each worker node and the master node is reliable and can only lead to inferior performance in wireless distributed computing.…”

Research on User Review Analysis Based on Text Mining and ISM

“…this shows the adjacency matrix of the model, the initial input matrix, abstracted from the relationships between the elements. According to Table 9, the Adjacency matrix multiplication of the model, which is an intermediate computational procedure to obtain the reachable matrix by successively multiplying the adjacency multiplication matrix until the matrix does not change 22 .…”