Halt or Continue: Estimating Progress of Queries in the Cloud

Abstract

Abstract. With cloud-based data management gaining more ground by day, the problem of estimating the progress of MapReduce queries in the cloud is of paramount importance. This problem is challenging to solve for two reasons: i) cloud is typically a large-scale heterogeneous environment, which requires progress estimation to tailor to non-uniform hardware characteristics, and ii) cloud is often built with cheap and commodity hardware that is prone to fail, so our estimation should be able to dynamically adjust. These two challenges were largely unaddressed in previous work. In this paper, we propose PEQC, a Progress Estimator of Queries composed of MapReduce jobs in the Cloud. Our work is able to apply to a heterogeneous setting and provides a dynamically update mechanism to repair the network when failure occurs. We experimentally validate our techniques on a heterogeneous cluster and results show that PEQC outperforms the state of the art.

COLA: A cloud-based system for online aggregation

“…The secondphase sampler allows reducers perform sampling from mappers' output during shuffle phase of join_job, and produces distributed stratified random samples from multiple tables respectively. You can find more details in our paper [8].…”

“…Then the component computes the most critical path of the PERT network, which can represent the execution of the whole query. Our paper [8] has the detailed discussion about how to make an estimate of the query progress according to the critical path.…”