Virtualizing Stream Processing

Duller, Michael; Rellermeyer, Jan S.; Alonso, Gustavo; Tatbul, Nesime

doi:10.1007/978-3-642-25821-3_14

Cited by 13 publications

(8 citation statements)

References 22 publications

(28 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An SPS without dynamic scale out support would have to be provisioned to sustain the peak rate. To generate a sufficiently high input stream rate, we precompute the input stream for L=1 in memory and replicate it for multiple express-ways [10].…”

Section: Dynamic Scale Outmentioning

confidence: 99%

Integrating scale out and fault tolerance in stream processing using operator state management

Fernandez

Migliavacca

Kalyvianaki

et al. 2013

Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data

263

255

View full text Add to dashboard Cite

Section: Dynamic Scale Outmentioning

confidence: 99%

Integrating scale out and fault tolerance in stream processing using operator state management

Fernandez

Migliavacca

Kalyvianaki

et al. 2013

Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data

263

255

View full text Add to dashboard Cite

“…The approaches in the works of Lim and Babu and Duller et al elevate the stream data processing optimization to a multiengine level. Both approaches can create query execution plans to include heterogeneous operators belonging to different engines.…”

Section: Related Researchmentioning

confidence: 99%

Optimizing data stream processing for large‐scale applications

2018

View full text Add to dashboard Cite

Summary Stream processing systems are designed to analyze data arriving in real time and using continuous queries and respond when a specific event or sequence of events are detected. An important aspect of these systems is Streaming Analytics, which facilitates statistical calculations on continuous data within the stream. These systems must be designed to handle high volumes of data, be scalable, and accommodate a multitude of long‐lived concurrently running analytics. The challenges involved in the development of stream processing include on‐the‐fly transformation of data streams to match the query needs of users and the ability to model stream transformations to detect overlaps and possibilities for optimizations and to specify a methodology to deliver optimizations. In particular, this work focuses on exposing data stream application internals in order to detect reusable parts and then consolidate applications to optimize computational resource usage. The Streaming Data Analytics Model presented in this paper adopts a declarative approach that enables processing and manipulation of data streams in a simple manner while facilitating powerful optimizations necessary for managing high volumes of streaming data in real time. An evaluation is provided to demonstrate in both theoretical and quantitative aspects the high performance offered by our approach.

show abstract

“…The past decade has seen substantial interest in softwarebased streaming computation, starting with hardware architectures [Kapasi et al 2003] and growing to include new parallel languages [Chakraborty and Thiele 2005;Gordon et al 2006] and middleware support focused on portability and interoperability [Cooper and Schwan 2005;Jain et al 2006;Neumeyer et al 2010;Duller et al 2011].…”

Section: Related Workmentioning

confidence: 99%

Energy Analysis of Hardware and Software Range Partitioning

Polychroniou

Barker

et al. 2014

ACM Trans. Comput. Syst.

View full text Add to dashboard Cite

Data partitioning is a critical operation for manipulating large datasets because it subdivides tasks into pieces that are more amenable to efficient processing. It is often the limiting factor in database performance and represents a significant fraction of the overall runtime of large data queries. This article measures the performance and energy of state-of-the-art software partitioners, and describes and evaluates a hardware range partitioner that further improves efficiency.The software implementation is broken into two phases, allowing separate analysis of the partition function computation and data shuffling costs. Although range partitioning is commonly thought to be more expensive than simpler strategies such as hash partitioning, our measurements indicate that careful data movement and optimization of the partition function can allow it to approach the throughput and energy consumption of hash or radix partitioning.For further acceleration, we describe a hardware range partitioner, or HARP, a streaming framework that offers a seamless execution environment for this and other streaming accelerators, and a detailed analysis of a 32nm physical design that matches the throughput of four to eight software threads while consuming just 6.9% of the area and 4.3% of the power of a Xeon core in the same technology generation.

show abstract

Virtualizing Stream Processing

Cited by 13 publications

References 22 publications

Integrating scale out and fault tolerance in stream processing using operator state management

Integrating scale out and fault tolerance in stream processing using operator state management

Optimizing data stream processing for large‐scale applications

Energy Analysis of Hardware and Software Range Partitioning

Contact Info

Product

Resources

About