Research on Optimal Checkpointing-Interval for Flink Stream Processing Applications

Zhang, Zhan; Li, Wenhao; Qing, Xiao; Liu, Xian; Liu, Hongwei

doi:10.1007/s11036-020-01729-7

Cited by 6 publications

(3 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Improving checkpointing is also a topic in academic research. Some of this work is centered around optimizing the checkpoint interval [70], [71]. As we described in our results, the checkpoint interval offers a trade-off between recovery speed and checkpointing overhead.…”

Section: ) Fault Tolerancementioning

confidence: 83%

A Performance Analysis of Fault Recovery in Stream Processing Frameworks

Dongen

Poel

2021

IEEE Access

View full text Add to dashboard Cite

Distributed stream processing frameworks have gained widespread adoption in the last decade because they abstract away the complexity of parallel processing. One of their key features is built-in fault tolerance. In this work, we dive deeper into the implementation, performance, and efficiency of this critical feature for four state-of-the-art frameworks. We include the established Spark Streaming and Flink frameworks and the more novel Spark Structured Streaming and Kafka Streams frameworks. We test the behavior under different types of faults and settings: master failure with and without high-availability setups, driver failures for Spark frameworks, worker failure with or without exactly-once semantics, application and task failures. We highlight differences in behavior during these failures on several aspects, e.g., whether there is an outage, downtime, recovery time, data loss, duplicate processing, accuracy, and the cost and behavior of different message delivery guarantees. Our results highlight the impact of framework design on the speed of fault recovery and explain how different use cases may benefit from different approaches. Due to their task-based scheduling approach, the Spark frameworks can recover quickly and in most cases without necessitating an application restart. Kafka Streams has the shortest downtime, while Flink can offer end-to-end exactly-once semantics at a low cost.

show abstract

Section: ) Fault Tolerancementioning

confidence: 83%

A Performance Analysis of Fault Recovery in Stream Processing Frameworks

Dongen

Poel

2021

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Based on stream computing technology, the data statistics module uses Apache Flink to realize statistics by jurisdiction and date, and the calculation results are pushed to MySQL database and Redis cache. Apache Flink is a distributed processing framework with low latency and high reliability [4]. As shown in Figure 2, the data storage module adopts a three-level data storage architecture.…”

Section: System Architecturementioning

confidence: 99%

Design and Implementation of Wastewater Pollutant Data Analysis and Visualization System for Vessels

Tang

Zou

2022

J. Phys.: Conf. Ser.

View full text Add to dashboard Cite

To monitor the discharge of water pollutants from vessels, a monitoring system for the discharge of water pollutants from vessels has been established on the Yangtze River. The system generates tens of thousands of ship wastewater pollutant data every day. The analysis and visualization of wastewater pollutant discharge data can help the monitoring department to grasp the discharge situation and improve the accuracy of supervision. This system extracts the data to the data warehouse in real-time based on the data extraction technology, improves the query efficiency based on the three-level storage architecture, makes real-time pollutant data statistics based on the stream computing technology, and realizes the visual display of the data based on the front-end and back end-separation technology. The system has been applied in the maritime supervision department to provide a data basis for vessel wastewater pollutant supervision and decision-making.

show abstract

“…Doris is used to store the web statistics results and provide them to the data visualization module for real-time query. Redis is an opensource and high-performance non-relational memory database, and it supports rich data structures, enables rapid addition, deletion, modification and query [10]. It caches intermediate calculation results in this platform.…”

Section: Introductionmentioning

confidence: 99%

A web log real-time analysis platform based on stream computing

Zhang

Tang³

et al. 2023

Third International Conference on Computer Vision and Data Mining (ICCVDM 2022)

View full text Add to dashboard Cite

With the rapid development of Internet technology, web applications have been widely used. How to ensure the reliability and high performance of web applications has become the focus of web site management. When a web application provides services, huge web logs are generated. These logs contain a great deal of information about users access to this web application. Real-time analysis of web logs can obtain system performance indicators and bottlenecks. In order to improve the reliability and performance of web applications, a real-time web log analysis platform based on stream computing is designed and implemented. The platform collects web log data by Flume, realizes data flow by Kafka message queue, analyzes web log by the Flink stream computing platform, stores the computing results in Redis for real-time query, and in Doris for historical data query. The availability of the platform is proved by running in a real environment, and it can improve the performance and reliability of web applications.

show abstract

Research on Optimal Checkpointing-Interval for Flink Stream Processing Applications

Cited by 6 publications

References 18 publications

A Performance Analysis of Fault Recovery in Stream Processing Frameworks

A Performance Analysis of Fault Recovery in Stream Processing Frameworks

Design and Implementation of Wastewater Pollutant Data Analysis and Visualization System for Vessels

A web log real-time analysis platform based on stream computing

Contact Info

Product

Resources

About