Resource bricolage for parallel database systems

Abstract

Running parallel database systems in an environment with heterogeneous resources has become increasingly common, due to cluster evolution and increasing interest in moving applications into public clouds. For database systems running in a heterogeneous cluster, the default uniform data partitioning strategy may overload some of the slow machines while at the same time it may under-utilize the more powerful machines. Since the processing time of a parallel query is determined by the slowest machine, such an allocation strategy may result in a significant query performance degradation.We take a first step to address this problem by introducing a technique we call resource bricolage that improves database performance in heterogeneous environments. Our approach quantifies the performance differences among machines with various resources as they process workloads with diverse resource requirements. We formalize the problem of minimizing workload execution time and view it as an optimization problem, and then we employ linear programming to obtain a recommended data partitioning scheme. We verify the effectiveness of our technique with an extensive experimental study on a commercial database system.

Performance Models of Data Parallel DAG Workflows for Large Scale Data Analytics

“…However, the analytical model for MapReduce is different due to different task execution frameworks. Resource Bricolage is proposed for parallel query optimization in a heterogeneous cluster [24]. This approach quantifies the performance differences among machines with various resources by profiling workloads.…”

Spark-based Cloud Data Analytics using Multi-Objective Optimization

“…When all Service-Level Objectives (SLOs) cannot be satisfied, it guarantees max-min fairness over SLO satisfactions; otherwise, it degrades to WS for recommending a single solution. Recent optimization for cluster and cloud computing [24,37] focus on running time of SQL queries, but not dataflow problems or multiple objectives. Morpheus [19] addresses the tradeoff between cluster utilization and job's performance predictability by codifying implicit user expectations as explicit SLOs and enforces SLOs using scheduling techniques.…”

Efficient Join Query Processing Algorithm CHMJ Based on Hadoop

“…在并行查询处理中,数据的分布策略是一个关键问题,直接影响查询处理的执行效率.目前,围绕在数据分 布 策 略 方 面 已 有 大 量 的 研 究 工 作 , 提 出 了 许 多 有 效 的 并 行 数 据 分 布 方 法 , 例 如 Round-Robin,Hash,Range-Partition [17] ,CMD [18] 以及基于聚类和一致 Hash 的数据布局 [19] 等数据分布方法. 一致性哈希算法基于一致性哈希算法,一致性哈希算法 [20] 的思想主要是为了解决网络中的热点问题.鉴于一致 性哈希算法只考虑单个数据没有考虑多个副本的问题,多副本一致性哈希算法修正与扩充了一致性哈希算法, 其支持多个副本的存放方式,从而达到多个副本的相关数据同样达到聚集的效果.多副本一致性哈希算法同时 也支持 Dynamo 中提出的虚拟节点的概念 [21] ,一个实节点可以影射为多个虚拟节点.在多副本一致性哈希算法 中需要构建一个哈希环,哈希环配置如图 2 所示.…”