From a calculus to an execution environment for stream processing

Abstract

At one level, this paper is about River, a virtual execution environment for stream processing. Stream processing is a paradigm well-suited for many modern data processing systems that ingest high-volume data streams from the real world, such as audio/video streaming, high-frequency trading, and security monitoring. One attractive property of stream processing is that it lends itself to parallelization on multicores, and even to distribution on clusters when extreme scale is required. Stream processing has been co-evolved by several communities, leading to diverse languages with similar core concepts. Providing a common execution environment reduces language development effort and increases portability. We designed River as a practical realization of Brooklet, a calculus for stream processing. So at another level, this paper is about a journey from theory (the calculus) to practice (the execution environment). The challenge is that, by definition, a calculus abstracts away all but the most central concepts. Hence, there are several research questions in concretizing the missing parts, not to mention a significant engineering effort in implementing them. But the effort is well worth it, because using a calculus as a foundation yields clear semantics and proven correctness results. Copyright © 2012 ACM

Testing properties of dataflow program operators

“…If none of these properties hold, the entire program is non-deterministic. Furthermore, blocking operators may cause deadlocks [15], [21].…”

River: an intermediate language for stream processing

“…The finishing point for this paper is the River execution environment . Extending a calculus into an execution environment is challenging.…”

“…This journal paper combines and extends our conference papers on the Brooklet calculus [6] and the River execution environment [7]. It goes beyond the conference papers by providing more details on CQL (source syntax, source semantics, and type system) and StreamIt (complete translation rules).…”

“…This is a significant advance over prior work, where source languages, optimizations, and target platforms are tightly coupled. For instance, because River's distributed target platform, System S, runs on a shared-nothing cluster, our implementation of the CQL front-end on River yields the first distributed implementation of the CQL.This journal paper combines and extends our conference papers on the Brooklet calculus [6] and the River execution environment [7]. It goes beyond the conference papers by providing more details on CQL (source syntax, source semantics, and type system) and StreamIt (complete translation rules).…”