Applying Formal Methods to Modeling and Analysis of Real-time Data Streams

Abstract

Achieving situation awareness is especially challenging for real-time data stream applications because they i) operate on continuous unbounded streams of data, and ii) have inherent realtime requirements. In this paper we showed how formal data stream modeling and analysis can be used to better understand stream behavior, evaluate query costs, and improve application performance. We used MEDAL, a formal specification language based on Petri nets, to model the data stream queries and the quality-of-service management mechanisms of RT-STREAM, a prototype system for data stream management. MEDAL's ability to combine query logic and data admission control in one model allows us to design a single comprehensive model of the system. This model can be used to perform a large set of analyses to help improve the application's performance and quality of service.

Modeling and Simulation of Spark Streaming

“…The authors extend standard SQL to allow the specification of sliding windows. Later, Kapitanova et al [19] have proposed a formal specification language called MEDAL to model data stream queries and data admission control. This language is based on Petri nets and focuses on modeling different streamprocessing features such as collaborative decision-making and temporal and spatial data dependencies.…”

Adaptive load shedding via fuzzy control in data stream management systems

“…The decisions are aggregated and cascaded back toward the sources in an incremental manner [16]. Notably, existing DSMS, including [1], [2], [3], [5], [4], [6], [18], [19], [13], [14], [15], [8], [16], do not aim to support the desired per-stream queue length even in the presence of dynamic workloads. In the future, we will investigate how to further reduce per-stream queue length overshoots.…”

“…It has also been used to support real-time periodic queries [18], [19], [23] implemented in STREAM [5]. However, a PI controller may largely fail to support the desired performance, if the model of the controlled system, e.g., a DSMS, derived offline using representative workloads becomes inaccurate due to dynamic workloads and system behaviors [10].…”

“…For performance 2 Note that network congestion control is not directly applicable to load shedding in DSMS [7], since it simply drops packets as the buffer in a network router builds up without simultaneously considering the aforementioned four issues and CQ selectivity important in DSMS. comparisons between our approach and Borealis, we use highspeed Internet traces [17] different from most existing research on DSMS (including [5], [4], [6], [18], [19], [13], [14], [15], [8], [16]), which uses synthetic workloads or outdated traces such as [20], [21]. The performance evaluation results show that our approach closely supports the desired per-stream queue length set-point.…”

A configurable and executable model of Spark Streaming on Apache YARN

“…The authors extend standard SQL to allow the specification of sliding windows. Later,Kapitanova et al (2011) have proposed a formal specification language called MEDAL to model data stream queries and data admission control. This language is based on Petri nets and focuses on modeling different streamprocessing features such as collaborative decision-making and temporal and spatial data dependencies.…”