Decentralized self-adaptation for elastic Data Stream Processing

Cardellini, Valeria; Presti, Francesco Lo; Nardelli, Matteo; Russo, Gabriele Russo

doi:10.1016/j.future.2018.05.025

Cited by 61 publications

(47 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [35], Cardellini et al propose a hierarchical architecture for DSP systems for geo-distributed environment. In such architecture, adaptation is done by employing Reinforced Learning techniques.…”

Section: Autoscaling Distributed Stream Processing Systemsmentioning

confidence: 99%

PASCAL: An architecture for proactive auto-scaling of distributed services

Lombardi

Muti

Aniello

et al. 2019

Future Generation Computer Systems

View full text Add to dashboard Cite

One of the main characteristics that today makes cloud services so popular is their ability to be elastic, i.e., they can adapt their provisioning to variable workloads, thus increasing resource utilization and reducing operating costs. At the core of any elastic service lies an automatic scaling mechanism that drives provisioning on the basis of a given strategy. In this paper we propose PASCAL, an architecture for Proactive Auto-SCALing of generic distributed services. PASCAL combines a proactive approach, to forecast incoming workloads, with a profiling system, to estimate required provision. Scale-in/out operations are decided according to an application-specific strategy, which aims at provisioning the minimum number of resources needed to sustain the foreseen workload. The main novelties introduced with PASCAL architecture are: (i) a strategy to proactively auto-scale a distributed stream processing system (namely, Apache Storm) with the aim of load balancing operators through an accurate system performance estimation model, and (ii) a strategy to proactively auto-scale a distributed datastore (namely, Apache Cassandra), focussed on how to choose when executing scaling actions on the basis of the time needed for the activation/deactivation of storage nodes so as to have the configuration ready when needed. We provide a prototype implementation of PASCAL for both use cases and, through an experimental evaluation conducted on a private cloud, we validate our approach and demonstrate the effectiveness of the proposed strategies in terms of saved resources and response time.

show abstract

Section: Autoscaling Distributed Stream Processing Systemsmentioning

confidence: 99%

PASCAL: An architecture for proactive auto-scaling of distributed services

Lombardi

Muti

Aniello

et al. 2019

Future Generation Computer Systems

View full text Add to dashboard Cite

show abstract

“…While the authors focus on rather powerful edge devices, this is nevertheless the approach that comes closest to the work at hand. Finally, Cardellini et al [5] present a fine-grained approach to allocate computational resources for stream processing in the fog in an elastic manner.…”

Section: Current State Of the Artmentioning

confidence: 99%

Towards a Framework for Data Stream Processing in the Fog

Hießl

Hochreiner

Schulte

2019

Informatik Spektrum

View full text Add to dashboard Cite

“…Moreover, aggregating information from the whole infrastructure to build a global view of the system is a challenging problem on its own. To improve scalability, decentralized management solutions have been proposed in the DSP context (e.g., [8][9][10][11]). However, devising a decentralized self-adaptation policy that only relies on a local system view is, in general, not trivial.…”

Section: Introductionmentioning

confidence: 99%

“…Aiming to exploit the strengths of centralized and decentralized solutions, in our previous work [8], we proposed a hierarchical distributed architecture for controlling the elasticity of DSP applications. Building on this solution, we then proposed Multi-Level Elastic and Distributed DSP Framework (E2DF) [13] to include mechanisms for realizing the infrastructure elasticity.…”

Section: Introductionmentioning

confidence: 99%

“…Specifically, to design the ICS elasticity policy, we resort to a model-based solution that instills knowledge about the system dynamics within the learning agent [14]. Differently from the RL-based application-level elasticity policy we proposed in [8], the model-based solution we present in this paper estimates the need of computing resources. In particular, considering active and idle resources, it estimates the ability to satisfy the future application demand.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Multi-Level Elasticity for Wide-Area Data Streaming Systems: A Reinforcement Learning Approach

et al. 2018

Self Cite

View full text Add to dashboard Cite

The capability of efficiently processing the data streams emitted by nowadays ubiquitous sensing devices enables the development of new intelligent services. Data Stream Processing (DSP) applications allow for processing huge volumes of data in near real-time. To keep up with the high volume and velocity of data, these applications can elastically scale their execution on multiple computing resources to process the incoming data flow in parallel. Being that data sources and consumers are usually located at the network edges, nowadays the presence of geo-distributed computing resources represents an attractive environment for DSP. However, controlling the applications and the processing infrastructure in such wide-area environments represents a significant challenge. In this paper, we present a hierarchical solution for the autonomous control of elastic DSP applications and infrastructures. It consists of a two-layered hierarchical solution, where centralized components coordinate subordinated distributed managers, which, in turn, locally control the elastic adaptation of the application components and deployment regions. Exploiting this framework, we design several self-adaptation policies, including reinforcement learning based solutions. We show the benefits of the presented self-adaptation policies with respect to static provisioning solutions, and discuss the strengths of reinforcement learning based approaches, which learn from experience how to optimize the application performance and resource allocation.

show abstract

Decentralized self-adaptation for elastic Data Stream Processing

Cited by 61 publications

References 33 publications

PASCAL: An architecture for proactive auto-scaling of distributed services

PASCAL: An architecture for proactive auto-scaling of distributed services

Towards a Framework for Data Stream Processing in the Fog

Multi-Level Elasticity for Wide-Area Data Streaming Systems: A Reinforcement Learning Approach

Contact Info

Product

Resources

About