R-Storm

Abstract

The era of big data has led to the emergence of new systems for real-time distributed stream processing, e.g., Apache Storm is one of the most popular stream processing systems in industry today. However, Storm, like many other stream processing systems lacks an intelligent scheduling mechanism. The default round-robin scheduling currently deployed in Storm disregards resource demands and availability, and can therefore be inefficient at times. We present R-Storm (Resource-Aware Storm), a system that implements resourceaware scheduling within Storm. R-Storm is designed to increase overall throughput by maximizing resource utilization while minimizing network latency. When scheduling tasks, R-Storm can satisfy both soft and hard resource constraints as well as minimizing network distance between components that communicate with each other. We evaluate R-Storm on set of micro-benchmark Storm applications as well as Storm applications used in production at Yahoo! Inc. From our experimental results we conclude that R-Storm achieves 30-47% higher throughput and 69-350% better CPU utilization than default Storm for the micro-benchmarks. For the Yahoo! Storm applications, R-Storm outperforms default Storm by around 50% based on overall throughput. We also demonstrate that R-Storm performs much better when scheduling multiple Storm applications than default Storm.

Distributed data stream processing and edge computing: A survey on resource elasticity and future directions

“…Several approaches for optimising the initial task assignment or scheduling exploit intra-query parallelism by ensuring that certain operators can scale horizontally to support larger numbers of incoming tuples, thus achieving greater throughput. R-Storm [82] handles the problem of task assignment in Apache Storm by providing custom resourceaware scheduling schemes. Under the considered approach, each task in a Storm topology has soft CPU and bandwidth requirements and a hard memory requirement.…”

TurboStream: Towards Low-Latency Data Stream Processing

“…T-Storm [13] requires that each node has only one available worker for each topology to avoid the inter-worker traffic inside one node which is only suitable for lightly loaded topologies. There is a fair number of research work on addressing heterogeneity in terms of resource and input rate in DSP [9,12,33,34]. For instance, Buddhika et al [33] introduce an on-line scheduler to mitigate the impact of interference on the performance of DSP.…”

“…There are two primary yet complementary ways to optimize inter-operator communications: (i) reduce the cost of intra-node IPC and (ii) reduce the inter-node IPC traffic. Prior efforts have focused mainly on reducing the inter-node IPC traffic by scheduling highly communicating operator instances to workers in the same node [9][10][11][12]. However, they may encounter high latency due to the high cost of intranode IPC, resulting from the memory-to-memory copying inside a cluster node.…”

Forum Jeunes Chercheurs à Inforsid 2016

“…L'interrogation de ces flux via des requêtes, dites continues 14 , représente un défi majeur en termes de performance pour le traitement en temps réel : passage à l'échelle pour la gestion de flux massifs et robustesse pour permettre le traitement continu des données. Afin de répondre à ces enjeux, des systèmes de gestion de flux de données (Schneider et al, 2009 ;Peng et al, 2015 ;Xu, Peng, 2016) ont été développés. Dans la suite de cette section, nous nous intéressons aux approches qui permettent l'exécution de requêtes continues de manière parallèle et distribuée.…”