OptEx: A Deadline-Aware Cost Optimization Model for Spark

Abstract

We present OptEx, a closed-form model of job execution on Apache Spark, a popular parallel processing engine. To the best of our knowledge, OptEx is the first work that analytically models job completion time on Spark. The model can be used to estimate the completion time of a given Spark job on a cloud, with respect to the size of the input dataset, the number of iterations, the number of nodes comprising the underlying cluster. Experimental results demonstrate that OptEx yields a mean relative error of 6% in estimating the job completion time. Furthermore, the model can be applied for estimating the cost optimal cluster composition for running a given Spark job on a cloud under a completion deadline specified in the SLO (i.e., Service Level Objective). We show experimentally that OptEx is able to correctly estimate the cost optimal cluster composition for running a given Spark job under an SLO deadline with an accuracy of 98%.Comment: 10 pages, IEEE CCGrid 201

A survey and taxonomy of resource optimisation for executing bag-of-task applications on public clouds

“…Some researchers have chosen to exclude workload allocation. Since Sidhanta et al [42] built their OptEx framework on the Spark framework, they relied on the existing built-in mechanisms for workload allocation. On the other hand, Mao et al [36] assumed prior knowledge regarding the fixed distribution of task allocations to each VM type.…”

“…The majority of the existing work considers both constraints and objectives whilst making a scheduling decision. For instance, some researchers have focused on optimising cloud usage by minimising the monetary cost while satisfying the deadline constraint [18,20,21,26,27,29,33,37,39,40,41,42], which aims to achieve the desired performance with the minimal cost. Researchers [24,32,46] have addressed the problem of performance maximisation with a budget constraint with the objective of obtaining with the maximum performance within a budgetary constraint.…”

“…For instance, Oprescu et al [31], Thai et al [40] and Hoseinyfarahabady et al [16] performed a sampling execution, in which a portion of a job was executed on VMs of all available types in order to estimate the task execution time. On the other hand, Sidhanta et al [42] estimated the job execution time using profiling techniques.…”

“…Integer Linear Programming is another choice which requires all decision variable to be integers [13,33,36,40,41], or Binary Integer Programming in which the decision variables are binary values (either one or zero) [37]. Other exact algorithms that are reported in the literature include Non-Linear Convex Optimisation Problem [42] and Integer Quadratic Programming [41].…”

“…This is because they can produce timely results although there is a trade-off against the optimality of solutions obtained. 14 X X X X X [19] X X X X X [15] X X X X X X [18] X X X X X X [13] X X X X X X [20] X X X X X X [21] X X X X X X [22] X X X X X X [36] X X X X X X [23] X X X X X X [34] X X X X X X X [27] X X X X X X X X X X [33] X X X X X [28] X X X X X X X [38] X X X X X X X [24] X X X X X X X [46] X X X X X X X [31,32] X X X X X X X X [26] X X X X X X X X [37] X X X X X X X X [39] X X X X X X X [42] X X X X X X [30] X X X X X X X X [25] X X X X X X X [29] X X X X X X X X [35] X X X X X X [40,41] X X X X X X X X X X [16] X X X X X X X X…”