Priority-Based Scheduling Techniques for a Multitenant Stream Processing Platform

Abstract

Apache Storm is a popular distributed stream processing system which has been widely adopted by the key players in the industry including YAHOO and Twitter. An application running in Storm is called a topology that is characterized by a Directed Acyclic Graph.Isolation Scheduler, the default scheduler for a multitenant Storm platform running multiple topologies assigns resources to topologies based on static resource configuration information and does not provide any means to prioritize topologies based on their business significances. One of the problems with this scheduler is that, performance degradation even complete starvation of topologies is possible on a resource constrained cluster. Two priority based resource scheduling strategies are proposed in this thesis to overcome these problems. A performance analysis based on prototyping and measurements is conducted to demonstrate the effectiveness of the proposed techniques. A comprehensive analysis of the results leading to key insights into system behavior and performance is presented.iii Acknowledgements First and foremost, I would like to thank my supervisor, Dr. Shikharesh Majumdar, for his continuous guidance, support, and encouragement during the process of writing this thesis and over the duration of this program. I sincerely appreciate his invaluable feedbacks and patience during every step of this thesis.