Accommodating bursts in distributed stream processing systems

Drougas, Yannis; Kalogeraki, Vana

doi:10.1109/ipdps.2009.5161015

Cited by 12 publications

(7 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Xing et al [148] have developed a placement algorithm that provides resilient placements that can withstand workload changes without operator migrations. Similarly, Drougas and Kalogeraki [40] propose a method for operator placement and parallelization that is resilient to sudden bursts in the input streams.…”

Section: Operator Placementmentioning

confidence: 99%

A Comprehensive Survey on Parallelization and Elasticity in Stream Processing

Röger

Mayer

2019

ACM Comput. Surv.

View full text Add to dashboard Cite

Stream Processing (SP) has evolved as the leading paradigm to process and gain value from the high volume of streaming data produced e.g. in the domain of the Internet of ings. An SP system is a middleware that deploys a network of operators between data sources, such as sensors, and the consuming applications. SP systems typically face intense and highly dynamic data streams. Parallelization and elasticity enables SP systems to process these streams with continuously high quality of service. e current research landscape provides a broad spectrum of methods for parallelization and elasticity in SP. Each method makes speci c assumptions and focuses on particular aspects of the problem. However, the literature lacks a comprehensive overview and categorization of the state of the art in SP parallelization and elasticity, which is necessary to consolidate the state of the research and to plan future research directions on this basis. erefore, in this survey, we study the literature and develop a classi cation of current methods for both parallelization and elasticity in SP systems. or even di erent processing nodes in a shared-nothing cloud-based infrastructure. Frequent state synchronization must not hamper parallel processing, while the processing results have to remain consistent. Research proposes di erent approaches for parallel, stateless and stateful SP. ey di er in assumptions about the operator functions and state externalization mechanisms an SP system supports.is led to the development of a broad range of parallelization approaches tackling di erent problem cases.Second, how to continuously adapt the level of parallelization when the conditions of the SP operators, e.g. the workload or resources available, change at runtime. On the one hand, an SP system always needs enough resources to process the input data streams with a satisfying quality of service (QoS), e.g. latency or throughput. On the other hand, continuous provisioning of computing resources for peak workloads wastes resources at o -peak hours. us, an elastic SP system scales its resources according to the current need. Cloud computing provides on-demand resources to realize such elasticity [9]. e pay-as-you-go business model of cloud computing allows to cut costs by dynamically adapting the resource reservations to the needs of the SP system. It is challenging to strive the right balance between resource over-provisioning-which is costly, but is robust to workload uctuations-and on-demand scaling-which is cheap, but is vulnerable to sudden workload peaks. To this end, academia and industry developed elasticity methods. Again, they di er in their optimization objectives and assumptions about the operator parallelization model employed, the target system architecture, state management as well as timing and methodology.While there are many works that propose methods and solutions for speci c parallelization and elasticity problems in SP systems, there is a severe lack of overview, comparison, and classi cation of these methods. When we investigate...

show abstract

Section: Operator Placementmentioning

confidence: 99%

A Comprehensive Survey on Parallelization and Elasticity in Stream Processing

Röger

Mayer

2019

ACM Comput. Surv.

View full text Add to dashboard Cite

show abstract

“…If it is too short the system may not be fast enough to exploit the new parallelism level. In prior stream processing systems the used intervals are in the order of tens of seconds like in [4,25]. Such granularity may be inappropriate to react to burstiness at fast time-scales like the one shown in the last two plots in the left hand side of Fig.…”

Section: Workload Variabilities and Solution Overviewmentioning

confidence: 99%

Elastic-PPQ: A two-level autonomic system for spatial preference query processing over dynamic data streams

Mencagli

Torquati

Danelutto

2018

Future Generation Computer Systems

View full text Add to dashboard Cite

“…Recent efforts have studied the problem of balancing the overloads in a DSPS. In our previous work [4], [5], we have proposed the BARRE and RADAR algorithms for accommodating unpredictable bursts of the data streams in DSPS. BARRE proactively computes data stream allocations to identify all feasible allocations and uses them at runtime upon the onset of a burst.…”

Section: Related Workmentioning

confidence: 99%

“…In our previous work we have proposed a distributed solution that uses measurements of elapsed times, application projected latencies and measurements of resource loads to dynamically determine a new rate allocation adjustment for distributed real-time stream processing applications, to react to bursty situations [4]. We have also proposed BARRE [5] that relies on an offline phase to identify all feasible allocations that we can choose at runtime. All the above approaches are either reactive or use offline information, and they do not consider the effect for reacting to the burst, especially when bursts are transient and may occur only for short periods of times.…”

Section: Introductionmentioning

confidence: 99%

Resource management using pattern-based prediction to address bursty data streams

Boutsis

Kalogeraki

2013

16th IEEE International Symposium on Object/Component/Service-Oriented Real-Time Distributed Computing (ISORC 2013)

Self Cite

View full text Add to dashboard Cite

In the recent years we have witnessed a proliferation of distributed stream processing systems that need to operate efficiently, even when data bursts occur. Examples include road traffic networks, processing of financial feeds, network monitoring and real-time sensor data analysis systems. An important challenge in managing these systems is effective resource management and meeting the QoS demands of the stream processing applications under different workload conditions, even under bursts. In this paper we present our approach that aims to predict the execution times of the distributed stream processing applications by taking into account the effects of the bursts and what is the typical workload of the stream processing system. Our approach builds application data rate patterns at run-time and predicts the effect of the burst on the performance of the applications, to identify whether there is a need to react on the onset of a burst. Our detailed experimental results over our Synergy middleware illustrate that our approach is practical, depicts good performance and has low resource overhead.

show abstract

Accommodating bursts in distributed stream processing systems

Abstract: Abstract

Cited by 12 publications

References 18 publications

A Comprehensive Survey on Parallelization and Elasticity in Stream Processing

A Comprehensive Survey on Parallelization and Elasticity in Stream Processing

Elastic-PPQ: A two-level autonomic system for spatial preference query processing over dynamic data streams

Resource management using pattern-based prediction to address bursty data streams

Contact Info

Product

Resources

About