Efficient replication of queued tasks for latency reduction in cloud systems

Abstract

Abstract-In cloud computing systems, assigning a job to multiple servers and waiting for the earliest copy to finish is an effective method to combat the variability in response time of individual servers. Although adding redundant replicas always reduces service time, the total computing time spent per job may be higher, thus increasing waiting time in queue. The total time spent per job is also proportional to the cost of computing resources. We analyze how different redundancy strategies, for eg. number of replicas, and the time when they are issued and canceled, affect the latency and computing cost. We get the insight that the log-concavity of the service time distribution is a key factor in determining whether adding redundancy reduces latency and cost. If the service distribution is log-convex, then adding maximum redundancy reduces both latency and cost. And if it is log-concave, then having fewer replicas and canceling the redundant requests early is more effective.

Cutting Latency Tail: Analyzing and Validating Replication without Canceling

“…In these systems it is not feasible to cancel all replicas of a request upon completion of the first one, which is a mechanism commonly used to limit the additional load introduced by replication. In fact, most existing modeling studies [6]- [11] assume that canceling is performed. This assumption simplifies the analysis, thanks to the implicit synchronization introduced by the canceling mechanism (all replicas of a request finish service at the same time).…”

“…Request replication has been considered in [3], [6]- [11], [14]- [18] as an efficient way to combat the variability in latency. Joshi et.…”

“…al. [11] analyze the impact of replication on the mean response time and the cost of computing resources, and show that this impact depends on the processing-time distribution. Vulimiri et.…”

“…Most existing works [3], [6]- [11], [17], [18] consider the adoption of canceling to limit the additional load introduced by the replicas. In fact, it is common to assume that redundant replicas can be canceled at no cost.…”

“…In fact, canceling plays a key role in obtaining the scarce analytical solutions available for request replication [10], [11]. For instance, canceling enables [11] to reduce the problem to a single M/G/1 queue in the early-canceling case or to an approximate M/G/C queue in the late-canceling one. Similarly, canceling is key in obtaining the solution to the Markov chain model in [10].…”