A Preventive Auto-Parallelization Approach for Elastic Stream Processing

Kombi, Roland Kotto; Lumineau, Nicolas; Lamarre, Philippe

doi:10.1109/icdcs.2017.253

Cited by 20 publications

(11 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Kombi et al [79] published the AUTOSCALE approach that centrally adapts the parallelization degrees of the operators in an operator graph based on predictive input values. eir goal is to proactively avoid congestion within the operator graph but be resource optimal and avoid too frequent recon gurations.…”

Section: Centralized Elasticitymentioning

confidence: 99%

A Comprehensive Survey on Parallelization and Elasticity in Stream Processing

Röger

Mayer

2019

ACM Comput. Surv.

View full text Add to dashboard Cite

Stream Processing (SP) has evolved as the leading paradigm to process and gain value from the high volume of streaming data produced e.g. in the domain of the Internet of ings. An SP system is a middleware that deploys a network of operators between data sources, such as sensors, and the consuming applications. SP systems typically face intense and highly dynamic data streams. Parallelization and elasticity enables SP systems to process these streams with continuously high quality of service. e current research landscape provides a broad spectrum of methods for parallelization and elasticity in SP. Each method makes speci c assumptions and focuses on particular aspects of the problem. However, the literature lacks a comprehensive overview and categorization of the state of the art in SP parallelization and elasticity, which is necessary to consolidate the state of the research and to plan future research directions on this basis. erefore, in this survey, we study the literature and develop a classi cation of current methods for both parallelization and elasticity in SP systems. or even di erent processing nodes in a shared-nothing cloud-based infrastructure. Frequent state synchronization must not hamper parallel processing, while the processing results have to remain consistent. Research proposes di erent approaches for parallel, stateless and stateful SP. ey di er in assumptions about the operator functions and state externalization mechanisms an SP system supports.is led to the development of a broad range of parallelization approaches tackling di erent problem cases.Second, how to continuously adapt the level of parallelization when the conditions of the SP operators, e.g. the workload or resources available, change at runtime. On the one hand, an SP system always needs enough resources to process the input data streams with a satisfying quality of service (QoS), e.g. latency or throughput. On the other hand, continuous provisioning of computing resources for peak workloads wastes resources at o -peak hours. us, an elastic SP system scales its resources according to the current need. Cloud computing provides on-demand resources to realize such elasticity [9]. e pay-as-you-go business model of cloud computing allows to cut costs by dynamically adapting the resource reservations to the needs of the SP system. It is challenging to strive the right balance between resource over-provisioning-which is costly, but is robust to workload uctuations-and on-demand scaling-which is cheap, but is vulnerable to sudden workload peaks. To this end, academia and industry developed elasticity methods. Again, they di er in their optimization objectives and assumptions about the operator parallelization model employed, the target system architecture, state management as well as timing and methodology.While there are many works that propose methods and solutions for speci c parallelization and elasticity problems in SP systems, there is a severe lack of overview, comparison, and classi cation of these methods. When we investigate...

show abstract

Section: Centralized Elasticitymentioning

confidence: 99%

A Comprehensive Survey on Parallelization and Elasticity in Stream Processing

Röger

Mayer

2019

ACM Comput. Surv.

View full text Add to dashboard Cite

show abstract

“…In addition, this website includes a comparison between Autoscale and Autoscale+. It gives an overview of the gap between the performance of the auto-parallelization strategy presented in [24] and results presented below. In the remainder of this section, the results described correspond to average values over 5 iterations for each configuration, thus lessening the the impact of punctual anomalies during tests.…”

Section: Experiments and Resultsmentioning

confidence: 99%

“…In [24] we defined a proactive approach, named Autoscale, to manage dynamically and automatically the parallelism degree of operators using indicators monitored on streams and operators. Our proposed algorithm decides which operators have to be reconfigured (scale-out or scale-in) and what their new parallelism degrees are.…”

Section: Parallelism Management With Autoscale+mentioning

confidence: 99%

“…Just like its predecessor Autoscale [24], Autoscale+ anticipates potential congestion of operators through stream and operator monitoring 3 . For each operator, input volumes in the near future are estimated according to time series analysis [10].…”

Section: Principlementioning

confidence: 99%

“…Furthermore, we expect as general a solution as possible, which does not depend on users whether for obtaining information from their experience or for triggering system adaptations. To reach this goal, we propose to build over two already published solutions, Autoscale [24] and OSG [29][30][31]. Autoscale proposes a method to fix the parallelism degree of each operator of a workflow with an original data-driven approach, which considers the complete graph of operators and data streams in the workflow to avoid inconsistent local decisions that lead to rapid revisions and therefore significant system instability.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

DABS-Storm: A Data-Aware Approach for Elastic Stream Processing

Kotto-Kombi

Lumineau

Lamarre

et al. 2019

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

In the last decade, stream processing has become a very active research domain motivated by the growing number of stream-based applications. These applications make use of continuous queries, which are processed by a stream processing engine (SPE) to generate timely results given the ephemeral input data. Variations of input data streams, in terms of both volume and distribution of values, have a large impact on computational resource requirements. Dynamic and Automatic Balanced Scaling for Storm (DABS-Storm) is an original solution for handling dynamic adaptation of continuous queries processing according to evolution of input stream properties, while controlling the system stability. Both fluctuations in data volume and distribution of values within data streams are handled by DABS-Storm to adjust the resources usage that best meets processing needs. To achieve this goal, the DABS-Storm holistic approach combines a proactive auto-parallelization algorithm with a latency-aware load balancing strategy.

show abstract

Self‐adaptation on parallel stream processing: A systematic review

Vogel

Griebler

Danelutto

et al. 2021

Concurrency and Computation

View full text Add to dashboard Cite

A recurrent challenge in real-world applications is autonomous management of the executions at run-time. In this vein, stream processing is a class of applications that compute data flowing in the form of streams (e.g., video feeds, images, and data analytics), where parallel computing can help accelerate the executions. On the one hand, stream processing applications are becoming more complex, dynamic, and long-running. On the other hand, it is unfeasible for humans to monitor and manually change the executions continuously. Hence, self-adaptation can reduce costs and human efforts by providing a higher-level abstraction with an autonomic/seamless management of executions. In this work, we aim at providing a literature review regarding self-adaptation applied to the parallel stream processing domain. We present a comprehensive revision using a systematic literature review method. Moreover, we propose a taxonomy to categorize and classify the existing self-adaptive approaches.Finally, applying the taxonomy made it possible to characterize the state-of-the-art, identify trends, and discuss open research challenges and future opportunities.

show abstract

A Preventive Auto-Parallelization Approach for Elastic Stream Processing

Cited by 20 publications

References 11 publications

A Comprehensive Survey on Parallelization and Elasticity in Stream Processing

A Comprehensive Survey on Parallelization and Elasticity in Stream Processing

DABS-Storm: A Data-Aware Approach for Elastic Stream Processing

Self‐adaptation on parallel stream processing: A systematic review

Contact Info

Product

Resources

About