Stela: Enabling Stream Processing Systems to Scale-in and Scale-out On-demand

Xu, Ling; Peng, Boyang; Gupta, Indranil

doi:10.1109/ic2e.2016.38

Cited by 64 publications

(41 citation statements)

References 17 publications

(28 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Other works() use more complex policies to determine the scaling decisions. Lohrmann et al propose a strategy that enforces latency constraints by relying on a predictive latency model based on queueing theory; nevertheless, their solution manages only stateless DSP applications.…”

Section: Related Workmentioning

confidence: 99%

Optimal operator deployment and replication for elastic distributed data stream processing

Cardellini

Presti

Nardelli

et al. 2017

Concurrency and Computation

View full text Add to dashboard Cite

Summary Processing data in a timely manner, data stream processing (DSP) applications are receiving an increasing interest for building new pervasive services. Due to the unpredictability of data sources, these applications often operate in dynamic environments; therefore, they require the ability to elastically scale in response to workload variations. In this paper, we deal with a key problem for the effective runtime management of a DSP application in geo‐distributed environments: We investigate the placement and replication decisions while considering the application and resource heterogeneity and the migration overhead, so to select the optimal adaptation strategy that can minimize migration costs while satisfying the application quality of service (QoS) requirements. We present elastic DSP replication and placement (EDRP), a unified framework for the QoS‐aware initial deployment and runtime elasticity management of DSP applications. In EDRP, the deployment and runtime decisions are driven by the solution of a suitable integer linear programming problem, whose objective function captures the relative importance between QoS goals and reconfiguration costs. We also present the implementation of EDRP and the related mechanisms on Apache Storm. We conduct a thorough experimental evaluation, both numerical and prototype‐based, that shows the benefits achieved by EDRP on the application performance.

show abstract

Section: Related Workmentioning

confidence: 99%

Optimal operator deployment and replication for elastic distributed data stream processing

Cardellini

Presti

Nardelli

et al. 2017

Concurrency and Computation

View full text Add to dashboard Cite

show abstract

“…Although any replication of a specific operator provides additional processing capabilities, it needs to be noted that any reconfiguration of the topology enactment has a negative impact on the processing performance. To minimize these reconfiguration aspects, Stela (Xu, Peng & Gupta, 2016), introduces new performance indicators to focus on the actual throughput of the SPE and to reduce any reconfiguration aspects.…”

Section: Related Workmentioning

confidence: 99%

“…Based on the Cloud Computing paradigm (Armbrust et al, 2010), a more promising provisioning approach, namely elastic provisioning for stream processing systems, emerged in recent years (Satzger et al, 2011;Gedik et al, 2014;Heinze et al, 2015;Lohrmann, Janacik & Kao, 2015;Xu, Peng & Gupta, 2016). This approach allows the SPE to lease computational resources on-demand whenever they are required.…”

Section: Introductionmentioning

confidence: 99%

Cost-efficient enactment of stream processing topologies

Hochreiner

Vögler

Schulte

et al. 2017

PeerJ Computer Science

View full text Add to dashboard Cite

The continuous increase of unbound streaming data poses several challenges to established data stream processing engines. One of the most important challenges is the cost-efficient enactment of stream processing topologies under changing data volume. These data volume pose different loads to stream processing systems whose resource provisioning needs to be continuously updated at runtime. First approaches already allow for resource provisioning on the level of virtual machines (VMs), but this only allows for coarse resource provisioning strategies. Based on current advances and benefits for containerized software systems, we have designed a cost-efficient resource provisioning approach and integrated it into the runtime of the Vienna ecosystem for elastic stream processing. Our resource provisioning approach aims to maximize the resource usage for VMs obtained from cloud providers. This strategy only releases processing capabilities at the end of the VMs minimal leasing duration instead of releasing them eagerly as soon as possible as it is the case for threshold-based approaches. This strategy allows us to improve the service level agreement compliance by up to 25% and a reduction for the operational cost of up to 36%. Subjects

show abstract

“…Related applications include location tracking, energy saving, transportation, safety control, and so on. However, the rapid growth of the IoT also increases the rate at which data is generated and results in big data; in the meantime, time-critical data streaming [6,7] applications have the time-limit requirement. Therefore, the cloud computing system with sophisticated resource management is a good candidate to handle these time-critical applications.…”

Section: Related Workmentioning

confidence: 99%

A Framework for Proactive Resource Provisioning in IaaS Clouds

Lee

Huang

et al. 2017

Applied Sciences

View full text Add to dashboard Cite

Abstract:Cloud computing is an emerging technology for rapidly provisioning and releasing resources on-demand from a shared resource pool. When big data is analyzed/mined on the cloud platform, the efficiency of resource provisioning would affect the system performance. This work proposes a framework for proactive resource provisioning in IaaS (Infrastructure as a Service) clouds to improve system performance. The proposed framework consists of the virtual cluster computing system, the profiling system, the resource management system, and the monitoring system. In this framework, the over-commit mechanism is applied to improve resource utilization. Furthermore, a proactive task scheduling approach is also present to prevent the postponement of tasks in critical stages, especially when the amount of aggregated resources requested by virtual machines exceeds that of available resources on the over-committed physical machines. Experimental results show that the over-commit approach indeed improves the resource utilization. However, when the degree of applying the over-commit approach increases, the burden of this proposed approach also conceivably increases. Therefore, the proposed framework further applies the proactive task scheduling approach to execute the time-critical tasks earlier to shorten the processing time. A small-scale cloud system including 3 servers is built for experiments. Preliminary experimental results show the performance improvement of our proposed framework in IaaS clouds.

show abstract

Stela: Enabling Stream Processing Systems to Scale-in and Scale-out On-demand

Cited by 64 publications

References 17 publications

Optimal operator deployment and replication for elastic distributed data stream processing

Optimal operator deployment and replication for elastic distributed data stream processing

Cost-efficient enactment of stream processing topologies

A Framework for Proactive Resource Provisioning in IaaS Clouds

Contact Info

Product

Resources

About