SPL

Abstract

Big data is revolutionizing how all sectors of our economy do business, including telecommunication, transportation, medical, and finance. Big data comes in two flavors: data at rest and data in motion. Processing data in motion is stream processing. Stream processing for big data analytics often requires scale that can only be delivered by a distributed system, exploiting parallelism on many hosts and many cores. One such distributed stream processing system is IBM Streams. Early customer experience with IBM Streams uncovered that another core requirement is extensibility, since customers want to build high-performance domain-specific operators for use in their streaming applications. Based on these two core requirements of distribution and extensibility, we designed and implemented the Streams Processing Language (SPL). This article describes SPL with an emphasis on the language design, distributed runtime, and extensibility mechanism. SPL is now the gateway for the IBM Streams platform, used by our customers for stream processing in a broad range of application domains. CCS Concepts: r Software and its engineering → Data flow languages;

Out-of-Order Sliding-Window Aggregation with Efficient Bulk Evictions and Insertions

“…Our implementation does not directly use SIMD instructions, but the C++ optimizing compiler sometimes uses them automatically. We did not implement partitioning but it is straightforward: when the aggregate is partitioned by key, keep disjoint state, i.e., a separate tree for each key; that would enable fission [12] for parallelization, either user-directed or automatically. Previous work describes an algorithm for range queries [22], and that algorithm also works in the presence of bulk insertion and eviction.…”

“…In theory, we expect that since the data is in-order, bulk insert brings no additional advantage over looping over single inserts. In practice, all algorithms improve in throughput as 𝑚 increases from 2 0 to around 2 12 . This may be because fewer top-level insertions means fewer memory fences, even for algorithms that emulate bulk insert with loops.…”

Distributed data stream processing and edge computing: A survey on resource elasticity and future directions

“…Architectural models [42], SPEs [43,13], and engines for certain application scenarios such as IoT are emerging. Architecture that mixes elements deployed on edge computing resources and the cloud is provided in the literature [43,44,42].…”

“…Stream Processing Language (SPL) offers a language and engine for composing distributed and parallel data-flow graphs and a toolkit for building generic operators [44]. It provides language constructs and compiler optimisations that utilise the performance of the Stream Processing Core (SPC) [68].…”