Auto-parallelizing stateful distributed streaming applications

Abstract

Streaming applications transform possibly infinite streams of data and often have both high throughput and low latency requirements. They are comprised of operator graphs that produce and consume data tuples. The streaming programming model naturally exposes task and pipeline parallelism, enabling it to exploit parallel systems of all kinds, including large clusters. However, it does not naturally expose data parallelism, which must instead be extracted from streaming applications. This paper presents a compiler and runtime system that automatically extract data parallelism for distributed stream processing. Our approach guarantees safety, even in the presence of stateful, selective, and userdefined operators. When constructing parallel regions, the compiler ensures safety by considering an operator's selectivity, state, partitioning, and dependencies on other operators in the graph. The distributed runtime system ensures that tuples always exit parallel regions in the same order they would without data parallelism, using the most e cient strategy as identified by the compiler. Our experiments using 100 cores across 14 machines show linear scalability for standard parallel regions, and near linear scalability when tuples are shu✏ed across parallel regions.

New parallel processing strategies in complex event processing systems with data streams

“…Despite being an effective parallelization framework, their framework is not feasible for pattern matching. Schneider et al 36 implemented intra-operator parallelism through datapartitioning; they introduced a compiler and a run time system that can automatically extract data parallelism from streaming applications. Brenna et al 38 proposed a novel approach to non-deterministic finite automata (NFA)-based distributed event processing where the NFA is decomposed into separate states running on different machines.…”

Adaptive input admission and management for parallel stream processing

“…Very recently, to address the unscalability of partitioning in this work, Zeitler et al [30] propose a parallelized stream splitting operator (parasplit) for massive-volume streams. In addition, once data parallelism is identified, extracting it from queries automatically with a compiler and runtime system constitutes an important step [22].…”

Tutorial

“…Schneider et al explored how to make fission safe in the more general case of partitioned-stateful and selective operators [30]; that work is the topic of the deep-dive in Section 4. Finally, Brito et al propose using transactional memory to make fission safe in the case of arbitrary operator state [7].…”

“…We thank our co-authors from prior work: Kun-Lung Wu worked on the fission optimization with us [30], and Robert Soulé and Robert Grimm worked on the optimization catalog with us [18]. The optimization catalog is currently under submission for a journal article.…”