Semantics of Data Streams and Operators

Abstract. In this paper state-of-the-art hardware and software technologies for stream data processing are reviewed. IBM InfoSphere Streams and Apache Spark are among of the most popular software products that alleviates burden of distributed program development for data analysis tasks. Capabilities of these systems are considered in application to the time series analysis. IBM InfoSphere Streams turns to be more suitable for online processing, whereas Apache Spark time series library focuses on a bulk processing of big collections of the time series.Keywords: data stream processing, high performance computing, time series analysis.Citation: Protsenko VI, Seraphimovich PG, Popov SB, Kazanskiy NL. Firmware and hardware infrastructure for data stream processing. CEUR Workshop Proceedings, 2016; 1638 : 782-787. DOI: 10.18287/1613 -0073-2016 -1638 1 IntroductionIn the recent years business and scientific organizations faced the problem of development of analytic pipelines that could process large amounts of data in real-time and be able to seamlessly incorporate new data sources and new queries. Previous approach, that suggest to use conventional data bases, was not suited for real-time data analysis because of the need to store data before the processing. Priority in ACID principle in databases also constrains its ability to scale well on clusters of tenths to thousands of nodes. As a consequence, it is hard or impossible to process massive amounts of data in a fixed time. New approaches were proposed: Hadoop framework for bulk processing and dataflow graph based stream processing systems. Still a lot of research results in the data base field is now used in data stream processing systems: random sampling [1], aggregations [2,3], join techniques [4,5], query plan optimizers [6] and schedulers [7,8,9].

show abstract

“…Syntax and semantics of stream query language is still a topic of the research [14][15][16][17][18].…”

Section: Stream Modelmentioning

confidence: 99%

Software and Hardware Infrastructure for Data Stream Processing

Protsenko

Seraphimovich²,

Popov

et al. 2016

Collection of Selected Papers of the II International Conference on Information Technology and Nanotechnology

View full text Add to dashboard Cite

show abstract

“…As an aside, note that associated with any application time t, there is a collection C(t) of input events (ordered by LE) that are live at t, i.e., have a lifetime that is stabbed by t. Our operator has the desirable changing relation property [18] that an output event is live at t if and only if C(t) contains the corresponding event subsequence that M accepts.…”

Section: M Accepts S) and [Le Re) Is The Intersection Of All Eventmentioning

confidence: 99%

“…If the inserted event e is out-of-order (with a timestamp t), we first call RemoveInvalidatedSequences (Lines [10][11][12][13][14][15][16][17][18][19] to delete the pmatch nodes that are no longer valid. We start at the rbtree entry with next largest timestamp after t, and begin deleting pmatch nodes from the linked list until we reach a pmatch with StartLE > t. If a deleted pmatch corresponds to a final state, we output a retraction event to compensate for the invalid prior output.…”

Section: Afa Algorithms and Runtime Complexitymentioning

confidence: 99%

High-performance dynamic pattern matching over disordered streams

2010

Self Cite

View full text Add to dashboard Cite

Current pattern-detection proposals for streaming data recognize the need to move beyond a simple regular-expression model over strictly ordered input. We continue in this direction, relaxing restrictions present in some models, removing the requirement for ordered input, and permitting stream revisions (modification of prior events). Further, recognizing that patterns of interest in modern applications may change frequently over the lifetime of a query, we support updating of a pattern specification without blocking input or restarting the operator. Our new pattern operator (called AFA) is a streaming adaptation of a non-deterministic finite automaton (NFA) where additional schema-based user-defined information, called a register, is accessible to NFA transitions during execution. AFAs support dynamic patterns, where the pattern itself can change over time. We propose clean order-agnostic patterndetection semantics for AFAs, with new algorithms that allow a very efficient implementation, while retaining significant expressiveness and supporting native handling of out-of-order input, stream revisions, dynamic patterns, and several optimizations. Experiments on Microsoft StreamInsight show that we achieve event rates of more than 200K events/sec (up to 5× better than simpler schemes). Our dynamic patterns give up to orders-of-magnitude better throughput than solutions such as operator restart, and our other optimizations are very effective, incurring low memory and latency.

show abstract

“…What we require is that for stream R(R), any prefix P of R can be reconstituted [11] into a linear sequence r 1 , r 2 , …, r m of snapshots over R. Each snapshot is just a finite relation over R. It is useful to consider how each additional event modifies the reconstitution. For example, with Alternative A1 above, we can treat an event <sn, p> as adding a new snapshot to the list that adds p to the previous snapshot.…”

Section: A2 One Control Parameter That Indicates What the Event Reprmentioning

confidence: 99%

On-the-fly progress detection in iterative stream queries

2009

Self Cite

View full text Add to dashboard Cite

Multiple researchers have proposed cyclic query plans for evaluating iterative queries over streams or rapidly changing input. The Declarative Networking community uses cyclic plans to evaluate Datalog programs that track reachability and other graph traversals on networks. Cyclic query plans can also evaluate pattern-matching and other queries based on event sequences.An issue with cyclic queries over dynamic inputs is knowing when the query result has progressed to a certain point in the input, since the number of iterations is data dependent. One option is a -strictly staged‖ computation, where the query plan quiesces between inputs. This option introduces significant latency, and may also -underload‖ inter-operator buffers. An alternative is to settle for soft guarantees, such as -eventual consistency‖. Such imprecision can make it difficult, for example, to know when to purge state from stateful operators.We propose a third option in which cyclic queries run continuously, but detect progress -on the fly‖ by means of a Flying Fixed-Point (FFP) operator. FFP sits on the cyclic loop and circulates speculative predictions on forward progress, which it then validates. FFP is always able to track progress for a class of queries we term strongly convergent. A key advantage of FFP is that it works with existing algebra operators, thereby inheriting their capabilities, such as windowing and dealing with out-oforder input. Also, for stream systems that explicitly model inputevent lifetimes, we know exactly which values are in the query result at each point in time.A key implementation decision is the method for speculating. Using the high-water mark of data events minimizes the number of speculative punctuations. Probing operators on the cyclic loop to determine their external progress circulates many more speculative messages, but tracks actual output progress more closely. We show how a hybrid approach limits predictions while coming close the progress-tracking ability of Probing.

show abstract

Semantics of Data Streams and Operators

Cited by 49 publications

References 18 publications

Software and Hardware Infrastructure for Data Stream Processing

Software and Hardware Infrastructure for Data Stream Processing

High-performance dynamic pattern matching over disordered streams

On-the-fly progress detection in iterative stream queries

Contact Info

Product

Resources

About