An Alternate Load Distribution Scheme in DHTs

Goel, Pulkit; Rishabh, Kumar; Varma, Vasudeva

doi:10.1109/cloudcom.2017.49

Cited by 7 publications

(3 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…3 The updates in JCH are limited because the updates only allowed at the tail node. 4 The updates in SACH are limited because the total number of nodes cannot exceed the initial maximum size. Similar to Maglev, m in SACH denotes the size of the lookup table which is much larger than n. There are two update complexity corresponding to the two update schemes in SACH.…”

Section: Related Work and Motivationmentioning

confidence: 99%

“…When the concept of consistent hashing (CH) was first proposed [5], it quickly became popular and played a essential role as a data router and a load balancer in various fields, such as disrtibuted database [10], [7], [1], cloud infrastructure [9], [13] and pear-to-pear network [11], [4], etc. In these scenarios, CH evenly maps the keys of the load to the backends (or nodes) with consistency.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

DxHash: A Scalable Consistent Hash Based on the Pseudo-Random Sequence

Dong¹,

Wang²

2021

Preprint

View full text Add to dashboard Cite

Consistent hasing has played a fundamental role as a data router and a load balancer in various fields, such as distributed database, cloud infrastructure, and peer-to-peer network. However, the existing consistent hashing schemes can't meet the requirements simultaneously, including full consistency, scalability, small memory footprint, low update time and low query complexity. Thus, We propose DxHash, a scalable consistent hashing algorithm based on the pseudo-random sequence. For the scenario of distributed storage, there are two optimizations based on DXHash are proposed. First, the Weighted DxHash can adjust the workloads on arbitrary nodes. Second, the Asymmetric Replica Strategy (ARS) is combining the replica strategy in distributed storage with the scaleup process to improve the availability of the system and reduce the remapping rate. The evaluation indicates that compared with the state-of-art works, DxHash achieves significant improvements on the 5 requirements. Even with 50% failure ratio, DxHash still can complete 16.5 million queries per second. What's more, the two optimizations both achieve their own results.

show abstract

Section: Related Work and Motivationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

DxHash: A Scalable Consistent Hash Based on the Pseudo-Random Sequence

Dong¹,

Wang²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Their approach has less "migration cost" than that of [33]. Goel et al [24] empirically study several consistent hashing strategies in DHTs.…”

Section: Related Workmentioning

confidence: 99%

Cache-aware load balancing of data center applications

et al. 2019

View full text Add to dashboard Cite

Our deployment of cache-aware load balancing in the Google web search backend reduced cache misses by ∼0.5x, contributing to a double-digit percentage increase in the throughput of our serving clusters by relieving a bottleneck. This innovation has benefited all production workloads since 2015, serving billions of queries daily. A load balancer forwards each query to one of several identical serving replicas. The replica pulls each term's postings list into RAM from flash, either locally or over the network. Flash bandwidth is a critical bottleneck, motivating an application-directed RAM cache on each replica. Sending the same term reliably to the same replica would increase the chance it hits cache, and avoid polluting the other replicas' caches. However, most queries contain multiple terms and we have to send the whole query to one replica, so it is not possible to achieve a perfect partitioning of terms to replicas. We solve this via a voting scheme, whereby the load balancer conducts a weighted vote by the terms in each query, and sends the query to the winning replica. We develop a multi-stage scalable algorithm to learn these weights. We first construct a large-scale term-query graph from logs and apply a distributed balanced graph partitioning algorithm to cluster each term to a preferred replica. This yields a good but simplistic initial voting table, which we then iteratively refine via cache simulation to capture feedback effects.

show abstract