Twitter Heron: Towards Extensible Streaming Engines

Fu, Maosong; Agrawal, Ashvin; Floratou, Avrilia; Graham, Bill; Jorgensen, Andrew; Li, Mark; Lu, Neng; Ramasamy, Karthik; Rao, Sriram; Wang, Cong

doi:10.1109/icde.2017.161

Cited by 21 publications

(16 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similar to Storm, a Heron application is referred to as a topology that consists of spouts and bolts. 13 Heron cannot provide exactly-once guarantee for stateless topologies and only provides at-most-once or at-least-once semantics. For stateful topologies, Heron provides exactly-once guarantee only if the stateful operations are idempotent.…”

Section: Stream Processing Systemsmentioning

confidence: 99%

See 1 more Smart Citation

Optimizing checkpoint‐based fault‐tolerance in distributed stream processing systems: Theory to practice

2021

View full text Add to dashboard Cite

Fault-tolerance is an essential part of a stream processing system that guarantees data analysis could continue even after failures. State-of-the-art distributed stream processing systems use checkpointing to support fault-tolerance for stateful computations where the state of the computations is periodically persisted. However, the frequency of performing checkpoints impacts the performance (utilization, latency, and throughput) of the system as the checkpointing process consumes resources and time that can be used for actual computations. In practice, systems are often configured to perform checkpoints based on crude values ignoring factors such as checkpoint and restart costs, leading to suboptimal performance. In our previous work, we proposed a theoretical optimal checkpoint interval that maximizes the system utilization for stream processing systems to minimize the impact of checkpointing on system performance.In this article, we investigate the practical benefits of our proposed theoretical optimal by conducting experiments in a real-world cloud setting using different streaming applications; we use Apache Flink, a well-known stream processing system for our experiments. The experiment results demonstrate that an optimal interval can achieve better utilization, confirming the practicality of the theoretical model when applied to real-world applications. We observed utilization improvements from 10% to 200% for a range of failure rates from 0.3 failures per hour to 0.075 failures per minute. Moreover, we explore how performance measures: latency and throughput are affected by the optimal interval.Our observations demonstrate that significant improvements can be achieved using the optimal interval for both latency and throughput.

show abstract

Section: Stream Processing Systemsmentioning

confidence: 99%

“…However, due to dividing the data stream into batches, Spark streaming is not well-suited for applications with latency needs below a few hundred milliseconds. 13 Apex is another stream processing system that supports fault-tolerant stateful computations. Apex uses periodic checkpoints to provide fault-tolerance.…”

Section: Stream Processing Systemsmentioning

confidence: 99%

Optimizing checkpoint‐based fault‐tolerance in distributed stream processing systems: Theory to practice

2021

View full text Add to dashboard Cite

show abstract

“…This information is useful to better understand our demonstration scenarios. An extensive description of Heron can be found in [8,9].…”

Section: Heron Backgroundmentioning

confidence: 99%

Dhalion in action

Agrawal

Floratou

2018

Proc. VLDB Endow.

Self Cite

View full text Add to dashboard Cite

In a world where organizations are being inundated with data from various sources, analyzing data and gaining actionable insights in real-time has become a key service differentiator. Over the last few years, several stream processing frameworks have been developed to address the need for large-scale, real-time analytics. A crucial challenge in these environments is the complexity of configuring, managing and deploying long-running streaming applications. Operators must carefully tune these systems to balance competing objectives such as resource utilization and performance. At the same time, they must also account for external shocks such as unexpected load variations and service degradations. In this demonstration, we show how operators can maintain healthy streaming applications without manual intervention while still meeting their performance objectives. We use Dhalion, an open-source library that sits on top of the streaming application, observes its behavior and automatically takes actions to keep the application in a healthy state. In particular, through various Dhalion policies that are configured by the attendees, we demonstrate how a streaming application can meet its performance objective by automatically configuring the amount of resources needed at the various application stages. We also demonstrate Dhalion's modularity and extensibility that greatly simplifies the process of developing new policies which address different application requirements.

show abstract

“…Furthermore, the scheme should also be computationally efficient so that we can trade little decision-making overheads for a significant improvement in the system performance. Moreover, inspired by the recent wide adoption of predictive scheduling in various different systems, 1 some natural questions come along: 1) If tuple arrivals can be predicted ahead of a short time window, then what are the fundamental benefits of such information to tuple scheduling? 2) Considering that such predictive scheduling, if wrongly decided, may consume extra system resources, then what is the impact of mis-prediction on the scheduling?…”

Section: Introductionmentioning

confidence: 99%

Dynamic Tuple Scheduling with Prediction for Data Stream Processing Systems

Huang

Shao

Yang

2019

2019 IEEE Global Communications Conference (GLOBECOM)

View full text Add to dashboard Cite

Most online service providers deploy their own data stream processing systems in the cloud to conduct large-scale and real-time data analytics. However, such systems, e.g., Apache Heron, often adopt naive scheduling schemes to distribute data streams (in the units of tuples) among processing instances, which may result in workload imbalance and system disruption. Hence, there still exists a mismatch between the temporal variations of data streams and such inflexible scheduling scheme designs. Besides, the fundamental benefits of predictive scheduling to data stream processing systems also remain unexplored. In this paper, we focus on the problem of tuple scheduling with predictive service in Apache Heron. With a careful choice in the granularity of system modeling and decision making, we formulate the problem as a stochastic network optimization problem and propose POTUS, an online predictive scheduling scheme that aims to minimize the response time of data stream processing by steering data streams in a distributed fashion. Theoretical analysis and simulation results show that POTUS achieves an ultra-low response time with stability guarantee. Moreover, POTUS only requires mild-value of future information to effectively reduce the response time, even with mis-prediction.

show abstract

Twitter Heron: Towards Extensible Streaming Engines

Cited by 21 publications

References 4 publications

Optimizing checkpoint‐based fault‐tolerance in distributed stream processing systems: Theory to practice

Optimizing checkpoint‐based fault‐tolerance in distributed stream processing systems: Theory to practice

Dhalion in action

Dynamic Tuple Scheduling with Prediction for Data Stream Processing Systems

Contact Info

Product

Resources

About