Task Scheduling in Big Data Platforms: A Systematic Literature Review

Abstract

Context: Hadoop, Spark, Storm, and Mesos are very well known frameworks in both research and industrial communities that allow expressing and processing distributed computations on massive amounts of data. Multiple scheduling algorithms have been proposed to ensure that short interactive jobs, large batch jobs, and guaranteed-capacity production jobs running on these frameworks can deliver results quickly while maintaining a high throughput. However, only a few works have examined the effectiveness of these algorithms. Objective: The Evidence-based Software Engineering (EBSE) paradigm and its core tool, i.e., the Systematic Literature Review (SLR), have been introduced to the Software Engineering community in 2004 to help researchers systematically and objectively gather and aggregate research evidences about different topics. In this paper, we conduct a SLR of task scheduling algorithms that have been proposed for big data platforms. Method: We analyse the design decisions of different scheduling models proposed in the literature for Hadoop, Spark, Storm, and Mesos over the period between 2005 and 2016. We provide a research taxonomy for succinct classification of these scheduling models. We also compare the algorithms in terms of performance, resources utilization, and failure recovery mechanisms. Results: Our searches identifies 586 studies from journals, conferences and workshops having the highest quality in this field. This SLR reports about different types of scheduling models (dynamic, constrained, and adaptive) and the main motivations behind them (including data locality, workload balancing, resources utilization, and energy efficiency). A discussion of some open issues and future challenges pertaining to improving the current studies is provided.

Manufacturing big data ecosystem: A systematic literature review

“…This paper presents a systematic literature review (SLR) on the current state of research associated with big data technologies in manufacturing [37]. To apply big data technologies in manufacturing successfully, it is essential to systematically review the literature of big data technologies in manufacturing from the following three perspectives: manufacturing data, big data technologies and data applications in manufacturing.…”

Optimized load balancing in high‐performance computing for big data analytics

“…It is a necessity for an efficient load balancing system to use optimized scheduling algorithms 14 . Soualhia et al 15 have done strong research on task scheduling in the Big Data platform. In this research, they mention that multiple jobs and tasks with different characteristics and different resource demands that are received by the scheduler in the big data cloud platforms and cause an imbalanced load in the system.…”

Using Hadoop Technology to Overcome Big Data Problems by Choosing Proposed Cost-efficient Scheduler Algorithm for Heterogeneous Hadoop System (BD3)

“…Reduce phase: In this phase, the reducer job is to process the input data that comes from the mapper by analyzing and merging it to produce the final output which is written to the HDFS in the cluster .Some other programming models such as Spark [44,45] and DataMPI [46] are competing with MapReduce. Table 3 summarizes the big data capabilities and the available primary technologies [5].Since MapReduce is an open source with high performance which is used by many big companies for processing batch jobs [47,48].…”