A SLA-based Spark cluster scaling method in cloud environment

Oh, Yoori; Choi, Jieun; Song, Eunjung; Kim, Moonji; Kim, Yoonhee

doi:10.1109/apnoms.2016.7737242

Cited by 2 publications

(1 citation statement)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Among the more recent results is a performance study of autoscaling Spark atop Kubernetes [11] which conrms dynamic executor allocation on worker nodes (as pods) but also that the worker nodes themselves remain static (setting spark.dynamicAllocation.enabled). Another approach for scaling Spark on top of OpenStack is based on SLAs with speci ed deadlines [8]. All of these approaches are bound to the runtime environment, such as Spark hosted on IaaS, CaaS or HPC.…”

Section: Related Workmentioning

confidence: 99%

Demo: SLASH: Serverless Apache Spark Hub

Spillner

2023

Proceedings of the 17th ACM International Conference on Distributed and Event-Based Systems

View full text Add to dashboard Cite

Application needs for big data processing are shifting from planned batch processing to emergent scenarios involving high elasticity. Consequently, for many organisations managing private or public cloud resources it is no longer wise to pre-provision big data frameworks over large xed-size clusters. Instead, they are looking forward to on-demand provisioning of those frameworks in the same way that the underlying compute resources such as virtual machines or containers can already be instantiated on demand today. Yet many big data frameworks, including the widely used Apache Spark, do not sandwich well in between underlying resource managers and user requests. With SLASH, we introduce a light-weight serverless provisioning model for worker nodes in standalone Spark clusters that help organisations slashing operating costs while providing greater exibility and comfort to their users and more sustainable operations based on a unique triple scaling method.

show abstract

Section: Related Workmentioning

confidence: 99%