Spotgres - parallel data analytics on Spot Instances

Binnig, Carsten; Salama, Abdallah; Zamanian, Erfan; El-Hindi, Muhammad; Feil, Sebastian; Ziegler, Tobias

doi:10.1109/icdew.2015.7129538

Cited by 10 publications

(2 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…work [10] that exploits the cost benefits ranges from running batch [20,28] to interactive [25,27]. The resource intensive nature of big data processing [3,24,31] and machine learning workloads [14,19,20,25,30] also make them good candidates for running on transient servers. To mitigate the revocation impacts of transient servers, fault-tolerant techniques such as checkpointing [25,30,32] and server selection policies [26,27] have been explored extensively.…”

Section: Transient Server Lifetimes and Cost Analysismentioning

confidence: 99%

CloudCoaster: Transient-aware Bursty Datacenter Workload Scheduling

Ogden¹,

Tian²

2019

Preprint

View full text Add to dashboard Cite

Today's clusters often have to divide resources among a diverse set of jobs. These jobs are heterogeneous both in execution time and in their rate of arrival. Execution time heterogeneity has lead to the development of hybrid schedulers that can schedule both short and long jobs to ensure good task placement. However, arrival rate heterogeneity, or burstiness, remains a problem in existing schedulers. These hybrid schedulers manage resources on statically provisioned cluster, which can quickly be overwhelmed by bursts in the number of arriving jobs.In this paper we propose CloudCoaster, a hybrid scheduler that dynamically resizes the cluster by leveraging cheap transient servers. CloudCoaster schedules jobs in an intelligent way that increases job performance while reducing overall resource cost. We evaluate the effectiveness of CloudCoaster through simulations on realworld traces and compare it against a state-of-art hybrid scheduler. CloudCoaster improves the average queueing delay time of short jobs by 4.8X while maintaining long job performance. In addition, CloudCoaster reduces the short partition budget by over 29.5%.

show abstract

Section: Transient Server Lifetimes and Cost Analysismentioning

confidence: 99%

CloudCoaster: Transient-aware Bursty Datacenter Workload Scheduling

Ogden¹,

Tian²

2019

Preprint

View full text Add to dashboard Cite

show abstract

“…However, Hadoop lacks the built-in checkpointing and recomputation mechanisms that Flint leverages in Spark. Prior work has also explored running a distributed database on spot instance [9,24]. This work addresses the problem of deciding serialization points for database operations, which differs from Flint's focus on defining checkpointing and server selection policies.…”

Section: Related Workmentioning

confidence: 99%

Flint

Sharma

Guo

et al. 2016

Proceedings of the Eleventh European Conference on Computer Systems

View full text Add to dashboard Cite

Cloud providers now offer transient servers, which they may revoke at anytime, for significantly lower prices than ondemand servers, which they cannot revoke. The low price of transient servers is particularly attractive for executing an emerging class of workload, which we call Batch-Interactive Data-Intensive (BIDI), that is becoming increasingly important for data analytics. BIDI workloads require large sets of servers to cache massive datasets in memory to enable low latency operation. In this paper, we illustrate the challenges of executing BIDI workloads on transient servers, where revocations (akin to failures) are the common case. To address these challenges, we design Flint, which is based on Spark and includes automated checkpointing and server selection policies that i) support batch and interactive applications and ii) dynamically adapt to application characteristics. We evaluate a prototype of Flint using EC2 spot instances, and show that it yields cost savings of up to 90% compared to using on-demand servers, while increasing running time by < 2%.

show abstract