River: an intermediate language for stream processing

Soulé, Robert; Hirzel, Martin; Gedik, Buğra; Grimm, Robert

doi:10.1002/spe.2338

Cited by 10 publications

(5 citation statements)

References 63 publications

(109 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Chandramouli et al presented TiMR, which implemented temporal queries over the MapReduce framework [27]. And finally, Soulé et al [77] presented a type system and smallstep operational semantics for CQL via translation to the Brooklet stream-processing calculus [78]. Synchronous dataflow (SDF) languages were introduced to ease the design of real-time embedded systems.…”

Section: Stream Processing Languagesmentioning

confidence: 99%

See 1 more Smart Citation

Stream Processing Languages in the Big Data Era

Hirzel¹,

Baudart²,

Bonifati

et al. 2018

SIGMOD Rec.

Self Cite

View full text Add to dashboard Cite

This paper is a survey of recent stream processing languages, which are programming languages for writing applications that analyze data streams. Data streams, or continuous data flows, have been around for decades. But with the advent of the big-data era, the size of data streams has increased dramatically. Analyzing big data streams yields immense advantages across all sectors of our society. To analyze streams, one needs to write a stream processing application. This paper showcases several languages designed for this purpose, articulates underlying principles, and outlines open challenges.

show abstract

Section: Stream Processing Languagesmentioning

confidence: 99%

“…Operators can be further configured upon invocation, for example, with windows or output assignments. To facilitate distribution, SPL's semantics are defined to require minimal synchronization between operators [77].…”

Section: Synchronous Dataflowmentioning

confidence: 99%

Stream Processing Languages in the Big Data Era

Hirzel¹,

Baudart²,

Bonifati

et al. 2018

SIGMOD Rec.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Several existing approaches in database and ML optimization have considered the use of IRs for sharing execution strategies and avoiding materialization, such as Halide [16] for ML and image processing, River [17] for Dataflow programming, and more recently Weld [13] which targets the broader spectrum of batch data analytics. Weld is the most general approach so far to express more than a single data programming frontend.…”

Section: Introductionmentioning

confidence: 99%

“…Despite the ongoing standardization and unification efforts in data streaming, no existing intermediate representation to date provides support for these semantics. IBM's River [17] is the only known data streaming IR, however, it falls short of supporting out-oforder processing or any other frontend whatsoever.…”

Section: Introductionmentioning

confidence: 99%

Arc: an IR for batch and stream programming

Kroll

Segeljakt

Carbone

et al. 2019

Proceedings of the 17th ACM SIGPLAN International Symposium on Database Programming Languages

View full text Add to dashboard Cite

In big data analytics, there is currently a large number of data programming models and their respective frontends such as relational tables, graphs, tensors, and streams. This has lead to a plethora of runtimes that typically focus on the efficient execution of just a single frontend. This fragmentation manifests itself today by highly complex pipelines that bundle multiple runtimes to support the necessary models. Hence, joint optimization and execution of such pipelines across these frontend-bound runtimes is infeasible. We propose Arc as the first unified Intermediate Representation (IR) for data analytics that incorporates stream semantics based on a modern specification of streams, windows and stream aggregation, to combine batch and stream computation models. Arc extends Weld, an IR for batch computation and adds support for partitioned, out-of-order stream and window operators which are the most fundamental building blocks in contemporary data streaming. CCS Concepts • Software and its engineering → Context specific languages; Compilers; • Computing methodologies → Distributed computing methodologies.

show abstract

“…A number of approaches also exist for speeding up extract-transform-load process (e.g., [6], [17]), but they are not designed to handle full analytics pipeline. Formal language was introduced [14] to optimise processing of streaming data, however the optimisation happens at the microservice/function level.…”

Section: Introductionmentioning

confidence: 99%

Online and Offline Analysis of Streaming Data

Hoque¹,

Miranskyy²

2018

2018 IEEE International Conference on Software Architecture Companion (ICSA-C)

View full text Add to dashboard Cite

Online and offline analytics have been traditionally treated separately in software architecture design, and there is no existing general architecture that can support both. Our objective is to go beyond and introduce a scalable and maintainable architecture for performing online as well as offline analysis of streaming data.In this paper, we propose a 7-layered architecture utilising microservices, publish-subscribe pattern, and persistent storage. The architecture ensures high cohesion, low coupling, and asynchronous communication between the layers, thus yielding a scalable and maintainable solution.This design can help practitioners to engage their online and offline use cases in one single architecture, and also is of interest to academics, as it is a building block for a general architecture supporting data analysis.

show abstract

River: an intermediate language for stream processing

Cited by 10 publications

References 63 publications

Stream Processing Languages in the Big Data Era

Stream Processing Languages in the Big Data Era

Arc: an IR for batch and stream programming

Online and Offline Analysis of Streaming Data

Contact Info

Product

Resources

About