Arc: an IR for batch and stream programming

Abstract

In big data analytics, there is currently a large number of data programming models and their respective frontends such as relational tables, graphs, tensors, and streams. This has lead to a plethora of runtimes that typically focus on the efficient execution of just a single frontend. This fragmentation manifests itself today by highly complex pipelines that bundle multiple runtimes to support the necessary models. Hence, joint optimization and execution of such pipelines across these frontend-bound runtimes is infeasible. We propose Arc as the first unified Intermediate Representation (IR) for data analytics that incorporates stream semantics based on a modern specification of streams, windows and stream aggregation, to combine batch and stream computation models. Arc extends Weld, an IR for batch computation and adds support for partitioned, out-of-order stream and window operators which are the most fundamental building blocks in contemporary data streaming. CCS Concepts • Software and its engineering → Context specific languages; Compilers; • Computing methodologies → Distributed computing methodologies.

The NebulaStream Platform: Data and Application Management for the Internet of Things

“…In NES, we allow users to choose their preferred programming environments and models and without taking system-internals and performance implications into account. To enable this diversity, we build on top of existing frameworks, such as Weld [14], Arc [8], Emma [1], and LARA [9] to represent diverse queries in a unified intermediate representation, the Nebular-IR. The Nebular-IR allows us to perform optimizations across operators, processing models, and language boundaries.…”