ESPBench: The Enterprise Stream Processing Benchmark

Abstract

Growing data volumes and velocities in fields such as Industry 4.0 or the Internet of Things have led to the increased popularity of data stream processing systems. Enterprises can leverage these developments by enriching their core business data and analyses with up-to-date streaming data. Comparing streaming architectures for these complex use cases is challenging, as existing benchmarks do not cover them. ESPBench is a new enterprise stream processing benchmark that fills this gap. We present its architecture, the benchmarking process, and the query workload. We employ ESPBench on three state-of-the-art stream processing systems, Apache Spark, Apache Flink, and Hazelcast Jet, using provided query implementations developed with Apache Beam. Our results highlight the need for the provided ESPBench toolkit that supports benchmark execution, as it enables query result validation and objective latency measures. CCS CONCEPTS• Software and its engineering → Software performance; • Information systems → Stream management; Database performance evaluation; • Applied computing → Enterprise data management.

Benchmarking Scalability of Stream Processing Frameworks Deployed as Event-Driven Microservices in the Cloud

“…More recently, DSPBench [63], OSPBench [8], and ESP-Bench [9] have been proposed. DSPBench contains 15 benchmarks, which resample typical stream processing applications, derived from reviewing the literature.…”

“…In contrast to most other benchmarks, ESPBench's task samples are implemented with Apache Beam. While Hesse et al [9] only perform evaluations with Spark, Flink, and Hazelcast Jet, we expect that also other Beam runners can be used to run the benchmark.…”

“…While there are several open-source stream processing frameworks promoting scalability as a core feature, there is only little empirical research evaluating and comparing their scalability. However, the need for systematic scalability evaluations has been recognized [8,9]. Benchmarking is a wellestablished method in software engineering to assess and compare the quality of software systems and services [10].…”

Micro-batch and data frequency for stream processing on multi-cores

“…The strategy they used to increase data frequency was to increase the number of data sources rather than increase the rate of item generation. ESPBench [17] is a benchmark for the enterprise stream processing domain. It implements an abstraction layer using Apache Bean, which allows running the benchmark using different DSPSs.…”

Streaming vs. Functions: A Cost Perspective on Cloud Event Processing

“…Previous research comparing different stream processing systems focuses on self-operated, open source systems such as Apache Storm, Apache Flink, and Apache Spark and does not include cloud services for DSP [4,[52][53][54]. Akidau et al [8] present a performance comparison of Apache Flink and Google Cloud Dataflow on GCP.…”