The 8 requirements of real-time stream processing

Abstract

Applications that require real-time processing of high-volume data steams are pushing the limits of traditional data processing infrastructures. These stream-based applications include market feed processing and electronic trading on Wall Street, network and infrastructure monitoring, fraud detection, and command and control in military environments. Furthermore, as the "sea change" caused by cheap micro-sensor technology takes hold, we expect to see everything of material significance on the planet get "sensor-tagged" and report its state or location in real time. This sensorization of the real world will lead to a "green field" of novel monitoring and control applications with high-volume and low-latency processing requirements.Recently, several technologies have emerged-including off-theshelf stream processing engines-specifically to address the challenges of processing high-volume, real-time data without requiring the use of custom code. At the same time, some existing software technologies, such as main memory DBMSs and rule engines, are also being "repurposed" by marketing departments to address these applications.In this paper, we outline eight requirements that a system software should meet to excel at a variety of real-time stream processing applications. Our goal is to provide high-level guidance to information technologists so that they will know what to look for when evaluation alternative stream processing solutions. As such, this paper serves a purpose comparable to the requirements papers in relational DBMSs and on-line analytical processing. We also briefly review alternative system software technologies in the context of our requirements.The paper attempts to be vendor neutral, so no specific commercial products are mentioned.

Patterns for Distributed Real-Time Stream Processing

“…In the specific case of stream computing [28], [29], [31], [32], there is a set of techniques that may have an impact on the application utilization. In [22] a catalog of ten patterns is proposed.…”

Window Update Patterns in Stream Operators

“…As a rule in data stream processing, we adhere to online in-memory computation, excluding the use of hard disk for performance reasons. Input tuples were always received in timestamp order, so stream imperfections such as delayed or out-of-order items cannot occur [17].…”

“…Stream processing must keep up with the fluctuating arrival rate of high-volume transient items, otherwise dropping unprocessed tuples is inevitable [17]. Therefore, it cannot be expected that fast in-memory computation could be performed over the entire stream, lest that available system resources would get rapidly exhausted.…”

“…Typically, a sliding window is repetitively applied over the stream and returns its most recent items at any given time, e.g., items received in the past hour or the latest 1000 tuples. Several system prototypes have been implemented, such as AURORA [1], Borealis [2], Gigascope [12], STREAM [6], and TelegraphCQ [7], setting a foundation for Stream Processing Engines [17], also suggesting certain window types for specifying suitable stream subsets.…”

Segmentation and Appearance Features Index for Digital Video Data

“…Data management of IP digital video is becoming more important according to the number of digital video cameras are installed widely [2][3][4]. The large volume of video data makes it hard work to store and browse them by just traditional way.…”