Defining the execution semantics of stream processing engines

Abstract

IntroductionSeveral modern data-intensive applications need to process large volumes of data on the fly as they are produced. Examples range from credit card fraud detection systems, which analyze massive streams of credit card transactions to identify suspicious patterns, to environmental monitoring applications that continuously analyze sensor data, to click stream analysis of Web sites that identify frequent patterns of interactions. More in general, stream processing is a central requirement in today's information systems.This state of facts pushed the development of several stream processing engines (SPEs) that continuously analyze streams of data to produce new results as new elements enter the streams. Unfortunately, existing SPEs adopt different processing models and standardized execution semantics have not yet emerged. This severely hampers the usability AbstractThe ability to process large volumes of data on the fly, as soon as they become available, is a fundamental requirement in today's information systems. Modern distributed stream processing engines (SPEs) address this requirement and provide low-latency and high-throughput data stream processing in cluster platforms, offering high-level programming interfaces that abstract from low-level details such as data distribution and hardware failures. The last decade saw a rapid increase in the number of available SPEs. However, each SPE defines its own processing model and standardized execution semantics have not emerged yet. This paper tackles this problem and analyzes the execution semantics of some widely adopted modern SPEs, namely Flink, Storm, Spark Streaming, Google Dataflow, and Azure Stream Analytics. We specifically target the notions of windowing and time, traditionally considered the key distinguishing factors that characterize the behavior of SPEs. We rely on the SECRET model, introduced in 2010 to analyze the windowing semantics for the SPEs available at that time. We show that SECRET models well some aspects of the behavior of modern SPEs, and we shed light on the evolution of SPEs after the introduction of SECRET by analyzing the elements that SECRET cannot fully capture. In this way, the paper contributes to the research in the area of stream processing by: (1) contrasting and comparing some widely used modern SPEs based on a formal model of their execution semantics; (2) discussing the evolution of SPEs since the introduction of the SECRET model; (3) suggesting promising research directions to direct further modeling efforts. Affetti et al. J Big Data (2017) et al. J Big Data (2017) 4:12 and interoperability of SPEs, since a user needs to understand system-specific aspects to confront various alternatives and select the ones that better suite her needs. SURVEY PAPERPage 2 of 24 AffettiThe main factors that differentiate the behaviors of SPEs are the models of windows and time they adopt [1]. Windows enable computations that would be otherwise unfeasible on unbounded datasets such as streams. For instance, counting the number o...

Semantic Foundations for Deterministic Dataflow and Stream Processing

“…The denotational semantics of CQL [17] can be reconstructed and greatly simplified within our framework using the notion of stream described in Example 7 (finite time-varying multisets). There are several works that deal with the semantics of specific language constructs (e.g., windows), notions of time, punctuations and disordered streams, but do not give a mathematical description of the overall streaming computation [5,7,25,44,67,75,76,109].…”

Managing Measurement and Occurrence Uncertainty in Complex Event Processing Systems

“…In general, the behavior of time windows, and of the fol-lowedBy operator (− >), depends on the notion of time used by the CEP engine. Different CEP systems implement distinct time semantics, which can be classified into event time, ingestion time or processing time semantics [32], [33]. In event time semantics, each event is timestamped by the source that produces it.…”

“…Any source timestamp is ignored. Even when this is not a deterministic model, it is common in distributed processing systems under the assumption that the clock skew between physical machines processing the events is negligible, and that the CEP patterns processing time does not introduce significant delays [33].…”

Experimental Analysis on Processing of Unbounded Data

“…In a Fixed window based approach, the whole dataset is distributed and organized into fixed size window however sliding window has fixed size and span that slides across the data where size indicates the length of each window, and a slide determines the interval between two consecutive windows [15]. Although there are some novel windowing approach has been introduced which is used based on the application such as count Windows define the size and the slide based on number of elements [15]. If the span is less than the size, then the window overlap [1].…”

“…For instance, consider the fixed windows of ten minutes, the system will integrate data for ten minutes of processing time, and then it will consider the collected data in those ten minutes as a window and process them [1]. Processing time semantics are always non-derivable, as the performance of the framework relies upon the speed at which data enter the system and on the speed at which the data flow among different operators within the engine [15].…”