Improving Distributed Gradient Descent Using Reed-Solomon Codes

Halbawi, Wael; Azizan, Navid; Salehi, Fariborz; Hassibi, Babak

doi:10.48550/arxiv.1706.05436

Cited by 19 publications

(31 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Assignment phase. The cyclic assignment is used, which was widely used in the existing works on the distributed computing problems [1], [3]- [6], [19], [20]. More precisely, we divide all the K datasets into N non-overlapping and equal-length groups, where the i th group for each 3 We assign all datasets in G i to the servers in…”

Section: Resultsmentioning

confidence: 99%

“…To solve the min-max optimization problem in (19) is highly combinatorial and becomes a part of on-going works. For some specific cases, this optimization problem has been solved in this paper (see Theorems 4 and 7).…”

Section: Resultsmentioning

confidence: 99%

“…For some specific cases, this optimization problem has been solved in this paper (see Theorems 4 and 7). In the following, we provide a generally loosen version of the converse bound in (19).…”

Section: Resultsmentioning

confidence: 99%

“…• Under the constraint of the widely used cyclic assignment [1], [3]- [6], [19], [20], we propose an optimal secure computing scheme in the sense that minimum randomness size is achieved.…”

Section: Contributionsmentioning

confidence: 99%

See 3 more Smart Citations

On Secure Distributed Linearly Separable Computation

Wan¹,

Sun²,

Ji³

et al. 2021

Preprint

View full text Add to dashboard Cite

Distributed linearly separable computation, where a user asks some distributed servers to compute a linearly separable function, was recently formulated by the same authors and aims to alleviate the bottlenecks of stragglers and communication cost in distributed computation. For this purpose, the data center assigns a subset of input datasets to each server, and each server computes some coded packets on the assigned datasets, which are then sent to the user. The user should recover the task function from the answers of a subset of servers, such the effect of stragglers could be tolerated.In this paper, we formulate a novel secure framework for this distributed linearly separable computation, where we aim to let the user only retrieve the desired task function without obtaining any other information about the input datasets, even if it receives the answers of all servers. In order to preserve the security of the input datasets, some common randomness variable independent of the datasets should be introduced into the transmission.We show that any non-secure linear-coding based computing scheme for the original distributed linearly separable computation problem, can be made secure without increasing the communication cost (number of symbols the user should receive). Then we focus on the case where the computation cost of each server (number of datasets assigned to each server) is minimum and aim to minimize the size of the randomness variable (i.e., randomness size) introduced in the system while achieving the optimal communication cost. We first propose an information theoretic converse bound on the randomness size. We then propose secure computing schemes based on two well-known data assignments, namely K. Wan and G. Caire are with the Electrical Engineering and

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Section: Contributionsmentioning

confidence: 99%

See 2 more Smart Citations

On Secure Distributed Linearly Separable Computation

Wan¹,

Sun²,

Ji³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…A branch of works proposed efficient erasure coding techniques, e.g. Reed-Solomon [42], LDPC [43], LDGM codes [44] and diagonal codes [45]. Another branch have studied the approximate versions of the gradient descent problem [42], [46]- [48].…”

Section: Introductionmentioning

confidence: 99%

Efficient Replication for Straggler Mitigation in Distributed Computing

Behrouzi-Far,

Soljanin

2020

Preprint

View full text Add to dashboard Cite

The potential of distributed computing to improve the performance of big data processing engines is contingent on mitigation of several challenges. In particular, by relying on multiple commodity servers, the performance of a distributed computing engine is dictated by the slowest servers, known as stragglers.Redundancy could mitigate stragglers by reducing the dependence of the computing engine on every server. Nevertheless, redundancy could yet be a burden to the system and aggravate stragglers. In this paper, we consider task replication as the redundancy technique and study the optimum redundancy planning to improve the performance of a master-worker distributed computing system. We start with optimum assignment policy of a given set of redundant task to a set of workers. Using the results from majorization theory, we show that if the service time of workers is a stochastically (decreasing and) convex random variable, a balanced assignment of non-overlapping batches of tasks minimizes the average job compute time. With that results, we then study the optimum level of redundancy from the perspective of average job compute time and compute time predictability. We derive the efficient redundancy level as a function of tasks' service time distribution. We observe that, the redundancy level that minimizes average compute time is not necessarily the same as the redundancy level that maximized compute time predictability. Finally, by running experiments on Google cluster traces, we show that a careful planning of redundancy according to the tasks' service time distribution can speed up the computing job by an order of magnitude.

show abstract

On the Effect of Task-to-Worker Assignment in Distributed Computing Systems with Stragglers

Behrouzi-Far

Soljanin

2018

2018 56th Annual Allerton Conference on Communication, Control, and Computing (Allerton)

View full text Add to dashboard Cite

We study the expected completion time of some recently proposed algorithms for distributed computing which redundantly assign computing tasks to multiple machines in order to tolerate a certain number of machine failures. We analytically show that not only the amount of redundancy but also the task-to-machine assignments affect the latency in a distributed system. We study systems with a fixed number of computing tasks that are split in possibly overlapping batches, and independent exponentially distributed machine service times. We show that, for such systems, the uniform replication of non-overlapping (disjoint) batches of computing tasks achieves the minimum expected computing time.

show abstract

Improving Distributed Gradient Descent Using Reed-Solomon Codes

Cited by 19 publications

References 0 publications

On Secure Distributed Linearly Separable Computation

On Secure Distributed Linearly Separable Computation

Efficient Replication for Straggler Mitigation in Distributed Computing

On the Effect of Task-to-Worker Assignment in Distributed Computing Systems with Stragglers

Contact Info

Product

Resources

About