A Comprehensive Performance Analysis of Stream Processing with Kafka in Cloud Native Deployments for IoT Use-cases

Abstract

The constant growth of the number of Internet of Things devices drives a huge increase in data that needs to be analyzed, at times in real time. Multiple platforms are available for delivering such data to analytics engines that can perform various operations on the data with low processing latency. These platforms can find their home in cloud native environments where high availability and scaling to the actual workload can be easily achieved. While the deployment environment is elastic, clusters still need to be adequately dimensioned to accommodate the components of the platforms even under high load.In this paper, we provide an analysis in this regard: we discuss key performance indicators of the popular Kafka message bus and the related Kafka Streams processing engine. Namely, we analyze latency, throughput, CPU and memory resource footprint aspects of these services under varying load and processing tasks that appear in Internet of Things applications. We find subsecond processing latency and linear but heavily task-dependent scaling behavior in the other performance indicators' case.