A systematic mapping of performance in distributed stream processing systems

Vogel, Adriano; Henning, Sören; Ertl, Otmar; Rabiser, Rick

doi:10.1109/seaa60479.2023.00052

Cited by 3 publications

(7 citation statements)

References 56 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A considerable number of studies proposing new benchmarks or reporting on experimental evaluations with available benchmarks have been published over the last years [37]. Recent examples of open-source benchmarks include DSPBench [5], OSPBench [34],…”

Section: Benchmarking Stream Processing Frameworkmentioning

confidence: 99%

“…Table 1 provided an overview of these benchmarks and a comparison with ShuffleBench. For a systematic and comprehensive review of the literature on stream processing benchmarking, we refer to our recent studies [16,37].…”

Section: Benchmarking Stream Processing Frameworkmentioning

confidence: 99%

“…Over the last decade, several works have been published that evaluate the performance of distributed stream processing frameworks or propose new benchmarks, evaluation methods, and auxiliary tools [5,13,18,21,34,37]. Available benchmarks usually contain one or a few task samples that fulfill domain-specific use cases, for example, for analyzing car traffic data [3,34] or aggregating Industrial Internet of Things (IIoT) sensor measurements [13,18].…”

Section: Introductionmentioning

confidence: 99%

“…• We provide open-source implementations 1 of ShuffleBench for different state-of-the-art stream processing frameworks as well as associated tooling to automate running benchmarks in Kubernetes-based cloud environments. • We conduct an experimental evaluation covering throughput and latency that are highly relevant metrics for stream processing [37]. Our evaluation covers the open-source stream processing frameworks Flink, Hazelcast, Kafka Streams, and Spark Structured Streaming due to their industry acceptance and academic relevance [37].…”

Section: Introductionmentioning

confidence: 99%

“…• We conduct an experimental evaluation covering throughput and latency that are highly relevant metrics for stream processing [37]. Our evaluation covers the open-source stream processing frameworks Flink, Hazelcast, Kafka Streams, and Spark Structured Streaming due to their industry acceptance and academic relevance [37]. We provide a replication package and the collected data of our experiments as supplemental material [17], such that other researchers and practitioners may repeat and extend our work.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

ShuffleBench: A Benchmark for Large-Scale Data Shuffling Operations with Distributed Stream Processing Frameworks

Henning,

Vogel,

Leichtfried

et al. 2024

Proceedings of the 15th ACM/SPEC International Conference on Performance Engineering

Self Cite

View full text Add to dashboard Cite

Distributed stream processing frameworks help building scalable and reliable applications that perform transformations and aggregations on continuous data streams. This paper introduces Shuf-fleBench, a novel benchmark to evaluate the performance of modern stream processing frameworks. In contrast to other benchmarks, it focuses on use cases where stream processing frameworks are mainly employed for shuffling (i.e., re-distributing) data records to perform state-local aggregations, while the actual aggregation logic is considered as black-box software components. ShuffleBench is inspired by requirements for near real-time analytics of a large cloud observability platform and takes up benchmarking metrics and methods for latency, throughput, and scalability established in the performance engineering research community. Although inspired by a real-world observability use case, it is highly configurable to allow domain-independent evaluations. ShuffleBench comes as a ready-to-use open-source software utilizing existing Kubernetes tooling and providing implementations for four stateof-the-art frameworks. Therefore, we expect ShuffleBench to be a valuable contribution to both industrial practitioners building stream processing applications and researchers working on new stream processing approaches. We complement this paper with an experimental performance evaluation that employs ShuffleBench with various configurations on Flink, Hazelcast, Kafka Streams, and Spark in a cloud-native environment. Our results show that Flink achieves the highest throughput while Hazelcast processes data streams with the lowest latency.

show abstract

Section: Benchmarking Stream Processing Frameworkmentioning

confidence: 99%

Section: Benchmarking Stream Processing Frameworkmentioning

confidence: 99%