A robust partitioning scheme for ad-hoc query workloads

Shanbhag, Anil; Jindal, Alekh; Madden, Samuel; Quiane, Jorge; Elmore, Aaron J.

doi:10.1145/3127479.3131613

Cited by 30 publications

(17 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is motivated by the fact that modern data analytics utilizes a large number of ad-hoc queries to do exploratory analysis. For example, in the context of building a robust partitioning scheme for ad-hoc query workloads, Shanbhag et al [38] found that after analyzing the first 80% of real-world workload traces the remaining 20% still contained 57% completely new queries.…”

Section: Efficiency Of Interleavingsmentioning

confidence: 99%

Dynamic interleaving of content and structure for robust indexing of semi-structured hierarchical data

2020

View full text Add to dashboard Cite

We propose a robust index for semi-structured hierarchical data that supports content-and-structure (CAS) queries specified by path and value predicates. At the heart of our approach is a novel dynamic interleaving scheme that merges the path and value dimensions of composite keys in a balanced way. We store these keys in our trie-based Robust Content-And-Structure index, which efficiently supports a wide range of CAS queries, including queries with wildcards and descendant axes. Additionally, we show important properties of our scheme, such as robustness against varying selectivities, and demonstrate improvements of up to two orders of magnitude over existing approaches in our experimental evaluation.

show abstract

Section: Efficiency Of Interleavingsmentioning

confidence: 99%

Dynamic interleaving of content and structure for robust indexing of semi-structured hierarchical data

2020

View full text Add to dashboard Cite

show abstract

“…Some distributed streaming systems use adaptive load-balancing that redistribute the workload based on AQWA's cost model for estimating the workload, e.g., STAR [10], Tornado [16,17], Amoeba [21,22], and PS 2 Stream [11]. However, these techniques are relatively slow when updating the statistics and updating the workload cost model when the statistics change.…”

Section: Related Workmentioning

confidence: 99%

TrioStat

Daghistani

Aref

Ghafoor

2020

Proceedings of the 28th International Conference on Advances in Geographic Information Systems

View full text Add to dashboard Cite

The wide spread of GPS-enabled devices and the Internet of Things (IoT) has increased the amount of spatial data being generated every second. The current scale of spatial data cannot be handled using centralized systems. This has led to the development of distributed spatial data streaming systems that scale to process in real-time large amounts of streamed spatial data. The performance of distributed streaming systems relies on how even the workload is distributed among their machines. However, it is challenging to estimate the workload of each machine because spatial data and query streams are skewed and rapidly change with time and users' interests. Moreover, a distributed spatial streaming system often does not maintain a global system workload state because it requires high network and processing overheads to be collected from the machines in the system. This paper introduces TrioStat; an online workload estimation technique that relies on a probabilistic model for estimating the workload of partitions and machines in a distributed spatial data streaming system. It is infeasible to collect and exchange statistics with a centralized unit because it requires high network overhead. Instead, TrioStat uses a decentralised technique to collect and maintain the required statistics in real-time locally in each machine. TrioStat enables distributed spatial data streaming systems to compare the workloads of machines as well as the workloads of data partitions. TrioStat requires minimal network and storage overhead. Moreover, the required storage is distributed across the system's machines.

show abstract

“…SWARM can distribute the workload among all executor machines, if necessary. Amoeba [33,32] is an adaptive data partitioning scheme in relational systems. Amoeba does not consider real-time stream processing.…”

Section: Related Workmentioning

confidence: 99%

SWARM: Adaptive Load Balancing in Distributed Streaming Systems for Big Spatial Data

Daghistani¹,

Aref²,

Ghafoor³

et al. 2020

Preprint

View full text Add to dashboard Cite

The proliferation of GPS-enabled devices has led to the development of numerous locationbased services. These services need to process massive amounts of spatial data in real-time. The current scale of spatial data cannot be handled using centralized systems. This has led to the development of distributed spatial streaming systems. Existing systems are using static spatial partitioning to distribute the workload. In contrast, the real-time streamed spatial data follows non-uniform spatial distributions that are continuously changing over time. Distributed spatial streaming systems need to react to the changes in the distribution of spatial data and queries. This paper introduces SWARM, a light-weight adaptivity protocol that continuously monitors the data and query workloads across the distributed processes of the spatial data streaming system, and redistribute and rebalance the workloads soon as performance bottlenecks get detected. SWARM is able to handle multiple query-execution and data-persistence models. A distributed streaming system can directly use SWARM to adaptively rebalance the system's workload among its machines with minimal changes to the original code of the underlying spatial application. Extensive experimental evaluation using real and synthetic datasets illustrate that, on average, SWARM achieves 200% improvement over a static grid partitioning that is determined based on observing a limited history of the data and query workloads. Moreover, SWARM reduces execution latency on average 4x compared with the other technique.

show abstract

A robust partitioning scheme for ad-hoc query workloads

Cited by 30 publications

References 24 publications

Dynamic interleaving of content and structure for robust indexing of semi-structured hierarchical data

Dynamic interleaving of content and structure for robust indexing of semi-structured hierarchical data

TrioStat

SWARM: Adaptive Load Balancing in Distributed Streaming Systems for Big Spatial Data

Contact Info

Product

Resources

About