Exploiting punctuation semantics in continuous data streams

Abstract

As most current query processing architectures are already pipelined, it seems logical to apply them to data streams. However, two classes of query operators are impractical for processing long or infinite data streams. Unbounded stateful operators maintain state with no upper bound in size and, so, run out of memory. Blocking operators read an entire input before emitting a single output and, so, might never produce a result. We believe that a priori knowledge of a data stream can permit the use of such operators in some cases. We discuss a kind of stream semantics called punctuated streams. Punctuations in a stream mark the end of substreams allowing us to view an infinite stream as a mixture of finite streams. We introduce three kinds of invariants to specify the proper behavior of operators in the presence of punctuation. Pass invariants define when results can be passed on. Keep invariants define what must be kept in local state to continue successful operation. Propagation invariants define when punctuation can be passed on. We report on our initial implementation and show a strategy for proving implementations of these invariants are faithful to their relational counterparts.

Window Update Patterns in Stream Operators

“…Some blocking operators, like aggregation or sorting, are unable to produce any results unless they consume the entire input. Besides, stateful operators, like join or intersection, should maintain tuples from their input streams, in order to guarantee that a fresh item from either stream would still be able to match an older tuple from the other one [19]. To remedy such intricacies, windows have been devised as a means of providing bounded datasets to query operators.…”

New Data Types and Operations to Support Geo-streams

“…Furthermore, queries on data streams are typically continuous queries, which require an intuitive, and semantically simple and clear query language/interface to specify queries and incorporate time window semantics. New processing paradigms and methods have been proposed and implemented in several stream processing systems [5,19,4,3,2] to achieve similar objectives. However, they can only handle streaming point locations naively [14] and do not have adequate support for evolving spatio-temporal extents.…”

“…Modeling data streams collected by sensors in real time as a database system has been proven to be successful over the past years [5,19,4,3,2]. Several data stream management systems (DSMS) have been developed with some of them leading to startup companies [18].…”

Processing Sliding Window Multi-Joins in Continuous Queries over Data Streams

“…stream items arrive in some known order, or duplicate items arrive in one contiguous batch) may be exploited to lower the memory usage in continuous query processing. Moreover, assertions, referred to as punctuations in [25,26], could be inserted into a stream to specify a restriction on subsequently arriving items. For instance, a punctuation may arrive stating that all future items shall have the A attribute larger than ten.…”